Windows Apps and Hadoop - Interoperability Options

Windows offers the largest and richest set of enterprise-wide end-user business tools.These tools can be used for analysis, visualization and for slicing and dicing the underlying data. As more and more enterprises embrace big data and Hadoop, the need for interoperability between the two is also growing. As you may note that Hadoop is from the Linux stable. And it should not come as a surprise to anyone that it can be daunting to  inter-operate the two together. This post presents some basics of your interoperability options.

The diagram below depicts the relationship between various elements of the technology stacks for Windows, Hadoop and deployment cloud architectures.
Windows Hadoop Interop

Quick takeaways

  • If you have only Windows/.NET skilled developers and administrators, then you should go with HDInsight product using Windows Azure cloud.
  •  If you have Linux admins but Windows/.NET developers then you have the option of either having a private Linux cloud or use a service like Rackspace to host your Hadoop cluster. 
  • There is no one-size-fits-all approach here. You will need to look around your company and decide.
Assumption here is that all of your business analysts and data science folks are familiar with Windows tools.

Source: Microsoft.com

Interoperability Options

APIs and Drivers

  • Microsoft .NET SDK for Hadoop. This kit provides .NET API access to aspects of HDInsight including HDFS, HCatalag, Oozie and Ambari, and also some Powershell scripts for cluster management. There are also libraries for MapReduce and LINQ to Hive. The latter is really interesting as it builds on the established technology for .NET developers to access most data sources to deliver the capabilities of Hadoop.
  • Microsoft Hive ODBC Driver. This driver enables the connection from Hive to Excel.
  • Microsoft .NET Map Reduce API For Hadoop Recently Microsoft released the .NET API to connect with Map/Reduce functionality of Hadoop Streaming.

Data Explorer Plugin for Excel

This is the easiest way to slice, dice and visualize data sitting in a Hadoop cluster. Please see my earlier post on Data Explorer Plugin.

Support for Third Party Tools

Many businesses are finding that they need to use third party software that may not support Hadoop ecosystem natively. This is where you want to use tools such as Talend, Pentaho and others. With these tools you can either import data to databases that are supported by your favorite tool or comma separated value files (CSVs) can always be relied upon.

Popular posts from this blog

Tennis and cool dashboards for the viewer

Image
In the just concluded Wimbledon Tennis Tournament IBM showcased some cool technology. IBM SlamTracker™ - a real time statistics and data visualization platform that leverages IBM's predictive analytics technology. It provides an ‘at a glance’ visual representation of a match using scores and statistics and encourages fans to get more involved by interacting with the data to gain deeper insight into the game. IBM SlamTracker™ analyses over eight years of Grand Slam data (over 41 million data points), to identify patterns in players and their styles. Before each match, IBM analyses historical matches between the players (or between players of similar styles if the players in question have not met before). In the last couple of years IBM did trial runs of SecondSight , which for the first time enabled the viewer to track the direction, speed and distance of players as they moved about the court. Data from SecondSight enables displays such as one below. Hopefully
Read more

NoSQL Database Comparison Chart (only one you need)

Image
Following is a great resource for any one considering different choices for their "NoSQL" style frame work. As always, "one-size-fits-all" approach does * not * work for NoSQL frameworks.  Just as a side note that MongoDB, Cassandra and CouchDB are the top three skills sought out on indeed.com (popular aggregator of job boards) of all the databases in the table below. Having a large pool of talent is always a good thing for your technology choice. Source : International Journal of Database Theory and Application, NoSQL Database: New Era of Databases for Big data Analytics Classification, Characteristics and Comparison, A B M Moniruzzaman and Syed Akhter
Read more