5 steps for big data application testing success

Big data planning/deployment is hard. What is equally hard and some might say even harder is doing big data QA/testing. There are many questions that need to answered before taking the first steps in coming up with a test plan. Shortcomings in testing can result in poor data quality, poor performance and less than stellar user experience. Big data Hadoop implementation deals with writing complex Pig, Hive programs and running these jobs using Hadoop map reduce framework on huge volumes of data across different nodes.

Use the following figure as a quick guide to zero in on points along the data flow path to target for quality assurance. 

Hadoop testing framework

Here are 5 quick steps to get your testing efforts on the right path -
  1. Document the high level cloud test infrastructure design (Disk space, RAM
    required for each node, etc.)
  2. Identify the cloud infrastructure service provider
  3. Document the SLAs, communication plan, maintenance plan, environment refresh plan
  4. Document the data security plan
  5. Document high level test strategy, testing release cycles, testing types, volume of data processed by Hadoop, third party tools
Credits 
  • Infosys Research Papers, 2013

Popular posts from this blog

Tennis and cool dashboards for the viewer

Image
In the just concluded Wimbledon Tennis Tournament IBM showcased some cool technology. IBM SlamTracker™ - a real time statistics and data visualization platform that leverages IBM's predictive analytics technology. It provides an ‘at a glance’ visual representation of a match using scores and statistics and encourages fans to get more involved by interacting with the data to gain deeper insight into the game. IBM SlamTracker™ analyses over eight years of Grand Slam data (over 41 million data points), to identify patterns in players and their styles. Before each match, IBM analyses historical matches between the players (or between players of similar styles if the players in question have not met before). In the last couple of years IBM did trial runs of SecondSight , which for the first time enabled the viewer to track the direction, speed and distance of players as they moved about the court. Data from SecondSight enables displays such as one below. Hopefully
Read more

NoSQL Database Comparison Chart (only one you need)

Image
Following is a great resource for any one considering different choices for their "NoSQL" style frame work. As always, "one-size-fits-all" approach does * not * work for NoSQL frameworks.  Just as a side note that MongoDB, Cassandra and CouchDB are the top three skills sought out on indeed.com (popular aggregator of job boards) of all the databases in the table below. Having a large pool of talent is always a good thing for your technology choice. Source : International Journal of Database Theory and Application, NoSQL Database: New Era of Databases for Big data Analytics Classification, Characteristics and Comparison, A B M Moniruzzaman and Syed Akhter
Read more