Wednesday, April 24, 2013

Is big data being overhyped?

It is always easy to present one side of a story. Its very true in the political sphere but it is also true in technical community to a fair extent. There are many places to go to on the web to see how big data would change everything that we do. For the better.  But, there are also some very smart people who are not on the bandwagon and with good reasons.

For a great collection of arguments made by various prestigious authors on why big data may turn out to be a lot of hoopla over nothing, I strongly recommend the post by Gil Press. The author has presented some great arguments on being extremely cautious before taking the deep dive into big data. Some of the passages that I found interesting are:

Ranting about how “blogs and articles yammer on with the benefits of ‘big data,’” Cuzzillo correctly observes that they are simply “repeating promises made years ago about the benefits of small data and small analytics. This is old decision support super-sized and warmed over, the ‘new and improved’ that won’t satisfy any better than the original but which costs much, much more.”

n “Data Skepticism,” O’Reilly Radar’s Mike Loukides adds this gem to the discussion: “The idea that there are limitations to data, even very big data, doesn’t contradict Google’s mantra that more data is better than smarter algorithms; it does mean that even when you have unlimited data, you have to be very careful about the conclusions you draw from that data. It is in conflict with the all-too-common idea that, if you have lots and lots of data, correlation is as good as causation.”

It is hard to argue against many of the points mentioned in the blog but having said that it is also impossible to go back to times when there existed no big data and hence no need to try and find nuggets in big data. That window with a view with no big data is shut. For ever. So, lets try and move forward. Caution & luck is recommended.