In today’s business environment, data rules. Because it is used to determine every strategic move at each level of an organization, it makes sense that you should collect as much information as possible right? I argue that this is actually not the case! Like most things in life, too much of anything is bad- data included. The two major reasons why I argue this position are the paradox of choice and its causal relationship to analysis paralysis.
If you have found yourself overwhelmed with a data set before and thought, “Where in the world do I even begin?” then this is the article for you. So grab a seat, pour yourself a cup of coffee, and get ready to never look at data the same way again.
The Paradox of Choice
A Psychologist named Barry Schwartz first coined the term “Paradox of Choice”. He identified this theory as the phenomenon that when given too many options to choose from, an individual has a hard time deciding or making a choice at all. An example he provides in his Ted Talk (based off data from Vanguard) is that for every 10 mutual funds an employer offers, the rate of employee participation goes down 2%. I know what you’re thinking. Why in the world would employees pass up free money for their retirement? The answer is that it is simply too hard to choose from all the funds offered! Based off of this example, and many other in his book, Schwartz proves that paralysis is a consequence of having too much choice.
This theory interests me because I believe that as technology advances, the number of choices one has increases exponentially. Consider this: within a single generation, consumers have gone from going to the store and selecting an item from the 10-15 brands offered (if they were lucky) to logging on to the internet and selecting from the thousands of brands offered. As an individual who studies consumer behavior, this fascinates me, but it also makes me think about the consequences of it. The main consequence I have identified? Paralysis.
As Schwartz’s theory suggests, people often put off making a decision when they are overwhelmed by the number of options to choose from. In the Vanguard case, this meant that people were purposely giving up contributions to their retirement account for no reason other than they consistently put off making a decision. This in tandem with studies proving that overthinking kills productivity, makes me believe that this trend does not solely pertain to retirement funds.
In fact a particular study on When High Powered People Fail suggests that, “The idea that pressure specifically targets individuals who have high working memory capacity carries implications for interpreting performance in real-world high-pressure situations.” Or in other words, individuals who work in high pressure jobs and are expected to perform often choke under pressure. Considering data analysts deliver conclusions that top management uses to set strategy, it seems very likely that they are likely to suffer from this form of paralysis.
How does this relate to data science? I believe that with so many data sets to choose from and metrics to measure against, data analysts can easily find themselves drowning in a sea of statistics. This can lead to many issues like inefficient use of time, inaccurate predictions, and a decrease in their employer’s profitability. Worst of all if they face the paradox of choice constantly, the chances of your analysts experiencing employee burnout is very high.
Although it may seem cliche, I really believe the solution to the problem of too much data is quality over quantity. Of course you need to have a significant number of data points to prove a correlation, but those data points can be concentrated. When running certain experiments, be sure that you are collecting data that centers around the demographic or behavioral trend you are interested in. Collecting unnecessary information opens up room for error because your analyst may identify the wrong variables, account for outliers, and etc.
A major factor in the realm of data collection at the moment is the GDPR. With these rules and regulations in place, companies will no longer be able to easily collect data in bulk; thus, forcing them to think outside of the box. All in all, every company needs data and great analysts in order to be successful. But if you want your analysts to perform to the best of their ability, you should think twice before you dump too much data into their laps.
About The Author:
Nikitha Lokareddy is a recent 4.0 graduate of W.P. Carey School of Business and currently works for Markitors, a digital marketing agency. She is also an advisor to Online Data Science Master's Degrees and is an avid playlist maker and coffee drinker.