Recently met an interesting engineer in a big data related conference. He relayed the following to me which I found to be a unusual way for a company to discover big data.
The technical problem
The application was deployed on multiple servers with the usual Java stack of Apache, JBOSS, MySQL etc. The multi- tenanted application was brand new with a fair share of bugs. The first customers were willing to use the new cloud deployed service for some cool features and because they saw value in the offering. The year was early 2011.
For reasons mentioned above, the customer experience was less than perfect. Sometimes the application worked as designed and sometimes not. The support team was thin and over worked. Many times customer problems would escalate fast and many levels of management would get sucked in against their wishes. This irked customer support team even further. A solution was needed that did not involve a complete re-write of the entire application or extended down times. To be precise, any downtime.
On a whim and with a little hope, the technical team embarked on a project to "Hadoop" the copious application logs on all the servers to look for anomalies that would point to issues about to happen which could be observed by the customer. And is it turned out a few things did turn up that consistently showed up in logs prior to issues. One of the anomaly had to do with overloading of one of the servers in the application cluster that started a "chain reaction" and resulted in loss of service to customers. After a series of false positives and other false starts things began to settle down and improved over time to a point where support team was willing to work it.
The alerts from the solution did make the support team unhappy but when they realized that they knew about the problem sooner than the customer, they agreed to take on the challenge. Very soon, support was calling the customer to let them know of impending issues. Customers were happy. Support team was happy.
Once the capability of the tools that were put in place became clear, some of the existing old school ETL tools were replaced with Hadoop big data ecosystem. This story has a happy ending!