A contradiction lies at the heart of modern research: although predicting things remains extremely difficult, modern technology and big data has led to the frequent expectation that it should be easy to solve the most complex of problems.
What increasing amounts of data has done is allow us to better explain what has already happened. While nimble-minded researchers can use this information to guide their forward-looking analysis, we need to bear in mind that there is a great deal of difference between post-rationalisation and prediction.
That’s why the combination of big data, artificial intelligence, and market research is so powerful – it gives us access to the full spectrum of human voices and opinions, which in turn allows us to generate insights that are truly predictive (and not just cherry-picked).
This powerful combo makes us all better ‘foxes’ – but what do I mean by that?
The Fox and the Hedgehog
Nate Silver in The Signal and the Noise identified two types of researchers: foxes and hedgehogs. Foxes take into account all available information, and rationally assemble their arguments. This makes their predictions more powerful and, ultimately, more likely to come true.
Hedgehogs, on the other hand, have pre-existing opinions of what is right and wrong (often expressed in a way designed to gain attention). They follow their gut regardless of the information at their disposal.
In an ideal world, having more data at one’s disposal should make everyone more fox-like. But the actuality of the situation is that more data means it is easier to cherry-pick facts and statistics that fit pre-existing prejudice (ie like the hedgehogs Silver describes). It’s little wonder that experts have become less trusted when the information they use as proof (and the conclusions they draw) rarely reflects the reality of peoples’ lives.
Tapping into reality (and big data) with Signify
So how can we most fruitfully use Big Data to better tap into real lives? That’s the question we sat down to answer when we met with data science company Signify to talk about how we could create a product that integrated their powerful AI offering with our own in-depth experience in qualitative research.
A little context to this meeting:
Before the infamous 2016 American election, Nate Silver was rightly held up as a Delphic figure in the worlds of research and predictive analysis. His website, FiveThirtyEight , was a shining example of a predictive website that worked because it was run by a man who knew what he was talking about. He harvested/ harvests information – aggregating opinion polls and adding demographic data to them.
Up until Trump/Clinton, Silver made powerful conjectures by taking on the mantle of a fox – gathering, assessing and analysing information at his disposal. However, he was very wrong with his predictions of this election. At the time the polls closed he gave Clinton a 71.4% chance of victory.
So what went wrong? In short, Silver discounted the wealth of information available to him. Signify, in contrast to Silver, called all the swing states right. They used a combination of clever, bespoke scripts and publicly available search and social data to find out what millions of voters cared about. By understanding what mattered to them Signify were able to forecast how they would vote.
When we heard about Signify’s success in the American election, we just knew that a partnership had the potential to create a game-changing new product for our research clients…
Artificial Intelligence and Market Research – Making us all better foxes
Together Hook Research and Signify have created ‘Social Intelligence’ – a powerful new process that lets us shed light on: what an audience is saying online, why they are saying it and how it makes them feel.
Distinct from traditional social listening, Social Intelligence ensures that we, as researchers, remain fox-like – combining artificial intelligence and market research to unlock the immense power of the internet and give us access to the full range of human opinion and behavior.
You can learn more about Social Intelligence on our blog, or just drop us an email at enquiries@hookresearch.co.uk. We’d love to chat more about this exciting new process (and how it can help power up your next piece of research) over a coffee.
