Cognitive psychology tells us that the unaided human mind is vulnerable to many fallacies and illusions because of its reliance on its memory for vivid anecdotes rather than systematic statistics.
– Steven Pinker
Inference demands evidence; Hard, unaltered, unabused, and unbiased evidence. Not assumptions. More frequently than not, when we look at our websites analytics, key performance indicators, and overall data we will start to impart a story, or see a pattern, where one realistically does not exist. The Clustering Illusion is our mind’s ability and tendency to erroneously consider the inevitable “streaks” or “clusters” arising in small samples from random distributions to be statistically significant.
In other words, we create patterns when they are not overtly evident. To make this fallacy more problematic, even when we don’t see a pattern, our minds are such that we will make one up. Have you ever wondered why you can see images in clouds? How about forms in star constellations? There is no man wearing Orion’s Belt nor is there an actual Dipper in the sky, however, with the help of our imagination we can often times very vividly see the things we perceive. We are also very good at sharing these images with other people (who are also good at seeing something in nothing themselves).
Think of when you’re reading a book. Are you 100% of the time aware you are reading or is your mind creating the scenery of what is being read? How about those few moments in our day when we can actually day dream and let our thoughts wonder? Are you aware you’re staring blankly into the carpet or are you imagining breathing in the ocean breeze on that holiday vacation you’ve been planning to Fiji? The most important part to all of this is that when you’re being imaginary often times you don’t realize it. It just happens and it’s very natural. When attempting to protect ourselves against wrong thinking the hardest urge is to go against nature.
Let’s give an example here. I am going to give you a 3 part list, each comprised of 1’s and 0’s. Determine whether or not there is a pattern.
• 01010100 01101000 01101001 01110011
• 01101001 01110011
• 01000010 01101001 01101110 01100001 01110010 01111001
Now what do you see? Is there a pattern? Does this mean something or is it just random? For a few readers, and those that understand computers, you’re probably a step ahead. But these three lines are Binary Code and translate into, “This is Binary.” So yes, there is certainly a pattern here but unless you knew about Binary Code there would be no way to understand it. In fact, aside from Binary Code, there is no other pattern to be found here that is meaningful (or at least that I intended); except that which you are able to create and impart onto it. To be honest a mathemagician could likely come up with something here, but even then it wouldn’t mean anything; of course, he could likely argue otherwise. Oh the wisdom in numbers?
The human brain seeks patterns and rules to help govern our otherwise chaotic world. As such, people are often overly sensitive to pattern recognition. Things have to make sense and when they don’t we make sense of them; even if that means believing in nonsense. A clap of thunder was once believed to be the almighty Zeus raging about something or another atop his cloud castle. Raining bolts of lightening down on poor mortal souls. I mean really, if a story like that could be made up for something we so easily understand today as static build up in our atmosphere? imagine all the stories and beliefs that come about from seeing patterns in data where patterns don’t exist? In fact, in the light of our ancestors’ inference it’s completely expected that we would do the same.
This is all to say, you would do well to be a healthy skeptic of data and analytics you review. Question everything. Especially since so much data is available nowadays, and those studying that data often have little background or training in data analysis. Don’t search for patterns unless you are trained, and well trained at that, even when the data seems to be logical look at every aspect surrounding it. Try to break your own story and prove it wrong. If it can pass your own scrutiny, and if you trust your own ability to accurately scrutinize, then move forward. Otherwise, defer your analysis to someone who is trained and experienced. Let’s give an example of how analyzing data inaccurately can lead to big problems:
Let’s say you’re looking at Google Analytics for a website selling pretty widgets. You’ve recently created a bunch of new content pieces different from those before to help attract and engage better with your users. You’ve updated your sites design, refined the user interface, made it responsive (meaning the display automatically adjusts for various screen sizes), ramped up social media efforts, and added some new widget products with different patterns. You are happy to see the increase in total traffic, time on site, and total number of page. You also notice that bounce rates have reduced. However, sales have not dramatically improved. As you dig deeper you notice that your new content pieces received a lot of attention on social media where traffic arrived, and those that landed on your site otherwise (organically, direct, paid ads) ended up visiting your content pieces as well.
Now what the Clustering Illusion bias may cause us to think is that:
“Well we created better content that was exposed to more people and our site was more attractive as well as easier to navigate. Thus more people stayed on our site longer and viewed more pages. The lingering question remains why didn’t we convert more? Perhaps our visitors just don’t like the new widgets enough to buy them?”
“Obviously our content attracts users (or leads), but it doesn?t attract customers (or quality leads). Our content plan moving forward should be to create more promotional pieces, rather than the informational, educational, or humorous ones of the past.”
We can come up with any number of scenarios for how this business’ data can be interpreted and you would likely even have your own method for doing so. This example was given because you can easily misunderstand and misinterpret data and, worse yet, once the story you create around the data is believed you likely won’t look any further for another explanation. You have your proof. But what if the story you create is wrong? You can make game changing decisions based off of data you inaccurately analyzed.
This was a real life account of something I’ve witnessed. The only reason that traffic, time on site, and number of pages viewed increased while bounce rates decreased was because their website had become responsive. It wasn’t the new content, it wasn’t the site’s new aesthetic design (though navigability and user experience does count), and it wasn’t even really their attempt at increasing their attention to social media. They never looked at what devices their users browsed their site with; a lot of which was actually untapped mobile users. Once the site became mobile friendly, more people visited more often, they were able to use the website and browse for longer since they weren’t being discouraged. However, a bug on their site caused an error for mobile users making a purchase. Everything increased and improved but their sales because all of their newfound users were unable to buy anything.
Now, most savvy ecommerce operators would be mindful enough to watch the numbers revolving around their users preferred device, operating system, and browser? but, this business was not. They went on to create a new content marketing plan for a few months that was seemingly purely promotional. The content they created was nothing but information about their products. Their users didn’t appreciate this new content (as who really enjoys an ad?), their numbers fell dramatically around traffic and especially from social media. It took them a long while to realize something in their story was not right before they contacted the marketing company where I was working at the time. Once we heard their story, listened to their timeline of events, it didn’t take us long to realize that their success initially was solely due to their new mobile friendly site. It was a mistake to change their content strategy because obviously their users didn’t enjoy it; views were at an all time low.
The point to this story is hopefully pretty apparent. Unless you are certain you know how to accurately analyze data don’t jump to conclusions or make assumptions; because, unless you are confident in your own abilities to not miss any of the intricacies or subtleties of data analysis, especially for ecommerce, you should likely look to a consultant for advice and help. The next time you see a trend, notice a tendency, or stumble upon a game-changing pattern, do your best to disprove your story. Attempt to find all its faults. Having a second professional opinion on important matters is crucial? even if you earnestly believe yourself to be correct.