We Don’t Need More Data, We Need Smart Analytics
Prior to the late 20th century, very little data was captured. And large data sets, even if they existed, required an incredible amount of manual tabulating.
The problem was solved by creating IT systems to capture and store huge amounts of data.Today, a $50 heart rate monitor or fitness watch captures 250 data points per second — 21.6 million a day.
So now with all of this data we can answer all of life’s questions, right?
Not even close.
Having petaflops of data is useless by itself no matter how organized it is.
Simply having data isn’t necessarily helpful. If it is housed on five different servers in three different formats, all jumbled up, it is worse than useless. Having a million times more useless data simply means that making sense of it is a million times harder and more expensive.
The typical solution to this challenge comes in two parts.
First, by saying “big data” over and over and blindly storing more and more data, many hope that God will magically fix the problem and send them to analytics heaven. (Cue the sound of angelic choir singing.)
Second, to hedge their bets, many have been pouring money into organizing the data via “data dictionaries” and better-organized databases.This is a tragically wrong-headed approach.
Having petaflops of data is useless by itself no matter how organized it is.The Library of Congress has an unfathomable amount of information on its shelves, all of which is well organized and catalogued. But the Dewey Decimal System is not analysis.
In order to make sense of data, you must have a coherent approach to analyzing it. Plunking more and more data into Hadoop does not answer any questions other than “how can we spend ourselves into bankruptcy with Big Data projects?”
So, what about those magical algorithms?
By themselves, algorithms are not the answer.They are simply computer code that executes logic. But first you need to know what you want the algorithm to look for.
There is no magic software bullet that will go through data and answer questions. No program, no matter how expensive, is going to provide a shortcut to make sense out of data.
As for Excel spreadsheets, they are equivalent to a dinosaur trying to drive a car.
To get out of this big data dead end, we need a different approach.
The real goal is to answer key business questions. To make sense of data, we need the analytical skills that can spot patterns and provide insight into the future. These are, to name but a few: Bayesian statistics, System dynamics models, Network Theory, Complexity Theory, econometrics and advanced statistics.
Abraham Maslow said, “When all you have is a hammer, every problem begins to look like a nail.”
Sadly, we have been trying to use a hammer to cut boards, drill holes and paint the walls. It is time we went to Home Depot and approached the task with the right tools.