Over the last few months, I’ve noticed that prices for e-books on my favourite e-Reader have become ‘personalised’; when I am logged into my account with the e-book seller I get one price, but if I browse anonymously, I see another…

Welcome to the world of Big Data – the world where we digitise, store, and analyse more and more aspects of our collective lives to help us make better decisions and personalise goods and services (including charging higher prices to those identified as willing to pay more).

In this new world, business models and sources of competitive advantage are shifting – both for companies we invest in and for active investors themselves. How could investors benefit in this Big Data world?


The Big Promise of Big Data

What is Big Data? It originated as the term to denote the copious volume of data collected by companies. Now Big Data has become synonymous with the broad technological changes brought about by internet connectivity, cheap data storage and algorithmic developments. We produce and maintain far more data than ever before and this is changing our world.

There are many benefits to arming ourselves with rigorous analysis of accurate and complete data. Human perceptions are subjective and we are far less logical than we would like to believe – bound as we are to first impressions, swayed by vivid stories, forgetful of the difference between opinion and fact and prone to bending both in our self-interest. Impartial logic applied to objective data has been the driver of much human progress, where reliance on custom and lore – the sum-total of human judgement – failed.

Big Data technologies promise to apply computers’ impartial logic and objectivity on an unprecedented scale. This could improve medical outcomes, uncover and mitigate a broad range of risk, reduce waste throughout the economy, and even reduce legal frictions through uncovering the most effective ways to write contracts and laws – not to mention the potential advantages for investment analysis.

Are we there yet?

But the state of the art of Big Data is that our ability to process data lags far behind our data collection capabilities, despite what the media lead us to believe. Computers today are extremely powerful, but they are limited. Modern algorithms excel when things are orderly: when data is structured (i.e. arrayed into giant tables or, at the very least, tagged with keywords), the outcomes are clearly defined, there is sufficient history, and the future looks roughly like the past. When any of these conditions are not met – as, for example, is the case with most real-life investing situations – these algorithms fail.

Human minds have serious flaws, but when situations are fluid and ambiguous the human mind remains the best decision-making tool. Our minds are unparalleled when the task at hand requires making linkages between fuzzy concepts, asking the right questions, envisioning the future, and incorporating disjoint, unstructured information.

Big Data technologists recognise this. Google’s search algorithm uses the input of human users to perfect its page rankings: each time you search and select a link to click on, Google tags that link with your search terms. It effectively takes the work you – a human – did selecting the right link and uses it to provide structure to unwieldy online content.

While humans rule the earth

Computers are far from catching up to our general judgment capabilities. If and when they do, humanity will probably no longer rule the earth, since the generally intelligent computers might not want to do our bidding! Fortunately for us, artificial general intelligence is a very difficult problem that we are not close to solving. Until we have artificial general intelligence, there is always a human decision maker behind the computer monitor, turning our entire human and computer power to human interests – public and private.

The result is that Big Data isn’t quite as objective a representation of reality as we might expect – it’s already been filtered through human minds. More than that, it’s been filtered through the gatekeepers controlling that data. Human self-interest can affect the choices of what data we collect and to whom we make it available, what algorithms we use, and how we interpret the results.

This means that Big Data will probably reduce, but not eliminate, all follies of the human mind – and at times may even amplify them. After all, computers are a lever for human minds the same way machines are a lever for human bodies. In relying on faulty underlying assumptions (made by humans), so-called ‘failsafe’ technology turned out not to be, and were largely responsible for a number of market crises: the crash of 1987, the failure of Long Term Capital Management in 1998 and, most recently, the GFC.

Despite all the obstacles, the human race will find a way to benefit from Big Data. As much as our progress may seem surprisingly slow today, what we are able to achieve 20 years from now will astonish us. Such is the nature of compounding growth. Computers will take over most well-defined decision-making tasks, from writing basic press releases to forecasting next year’s earnings. And human minds will be redeployed to situations where fluid intelligence, imagination and asking the right question are required.

Implications for investing

Within the investment management industry, fundamental long-term investing stands out as the area that plays to humans’ strengths: within Allan Gray’s 5-10 year investment horizon, uncertainty is high, critical variables can change, and the future can differ from the past. Under these conditions algorithms struggle and humans shine. Moreover, the inherent limitations and biases of Big Data will continue to present opportunities to the contrarian investor.

This is not to say fundamental investors should write off the advance of Big Data. Even today, before Big Data tools are ready, human experts – including investors – can improve their decisions by adopting Big Data ways of thinking and incorporating rigorous statistical and algorithmic frameworks. This way they can dampen the weaknesses and amplify strengths of the human mind – for the time being, the best mind on the planet.