If you were trying to predict where a hurricane would land, would you rely on weather patterns from satellites, measurements like wind speed and rainfall from the "eye" of the storm, and supercomputer simulations, or your own “hunch?” I hope you would choose data.
If you were making a business decision, would you rely on internal customer data, advertising attribution metrics, projections from analytics or your own “hunch?” I hope you would choose data - but i have a feeling it would be a hard decision.
If you said data, you may be fooling yourself
This post was inspired by a conversation I had with a CEO of a company that uses AI/ML to make better marketing decisions. One of the problems the company faces is getting companies to “trust” their insights. Despite having incredibly large amounts of data and world class machine learning to make insights, if these insights run counter to people’s “hunches”, no matter how much data you have, they won't believe you.
This is a problem everywhere
Quite simply, people don’t trust big data. Across multiple surveys, around the globe (KPMG survey,University Survey), only about 30 - 40% of managers, marketers, executives - who ever is being surveyed, have a high level of trust in their data. The reasons for this tend to be either data quality, poor insights or algorithm skepticism. Often, things aren't tagged right, databases don't update - there are always plenty of reasons to dismiss data.
But the reason no one gives: themselves
Humans struggle with making decisions in the face of new data and suffer from an overwhelming large number of psychological biases which prevent them from making good decisions. Study after study shows when humans are are confronted with new information about their beliefs, humans tend to stick to their original opinion.
Why are we so afraid to change our opinion?
The status quo bias, which, just like it sounds, wants us to maintain the status quo, is a hard bias to get around. Our logic is if we keep the status quo going, and everything turns out fine, we have done a good job. If we try and change the status quo, with something like big data, but it turns out against us, it’s a catastrophe.
This situation was detailed very well on an A16Z podcast, which talks about investing in startups where the consensus is undecided. If you invest in something that people think is crazy and it works out, you are a genius. Many investors passed over AirBnb because they didn’t think the market was big enough or didn’t invest in the travel industry. Those who did, get put on lists like “Most Creative People.”
If you invest in something that no one thinks will work, and it doesn’t, you get labeled an idiot. Pets.com often gets labeled as one of the greatest follies of the internet bubble. Who would ever order their dog food online...
The issue with big data
Big data constantly gives us new information and insights and forces us to rethink our original assumption. Most of the time we just stick with what works, ignoring the signals or potential gold mine of insights. We don't want to mix things up for a fear of failure, so we don't end up "trusting" our data. We say things like "Our data is tagged properly" or come up with other crazy stipulations on why our data is wrong. Of course, you could have mistagged everything in your database, but those excuses only work a handful of times.
So, how do we get out of the way of ourselves?
The easiest way to overcome these biases is with A/B testing. If data shows counter intuitive insights, it should be worth testing on a small scale. A/B testing is so important to companies like Google, LinkedIn and Facebook, they have all built their own internal A/B testing platforms. At Netflix, every major product change is put through rigorous A/B testing. You’ll need people or software to do this, but experiments should allow for better decision making.
It’s easy to say, just change your culture to one rooted in A/B testing and experimentation, but easier said than done. It takes discipline sticking to results and not placing blame if results don't work out as expected.
But then again, no one ever got fired for buying IBM