4 Steps to uncovering dark data

You want to increase efficiency of your organization or company.  You have heard a lot (or little) about big data and dark data and want to see if you can use some of the techniques to do so. You might have done the steps listed below in years gone by but were told that there is no way to extract useful information from emails, social media etc. Well, lot has changed lately. Now is the time to have the conversation again with your IT/data team.

(Dark data is any data that is not being mined, analyzed)


Here are the 4 basic steps on how best to uncover the dark data:

Step 1 - List out all the decisions that are being made with "gut instincts", "experience", "status quo" etc

Use a brainstorming session type format to elicit this list from the middle and upper level managers.

Step 2 - Look for data sources that can help with decisions listed in Step 1

Check with your management team to see which data sources could be used to help with decision making. No data source is out of bound. Videos, Audios, Emails, Twitter feeds and everything else.

Step 3 - Do a quick back-of-the-envelope cost-benefit analysis of the decisions from Step 1

This step makes me a little queasy since this step itself is not supported with data but then again you need to bootstrap. And eventually this decision making itself could be driven by data available to you.

Step 4 - Get your data/IT team together and plan out the next steps

Be ready to deal with a bunch of engineers telling you why things will not work and how insane you are. Deal with it. And tell them to deal with it as well. That is it.

Hint: Take baby steps towards big data