Journey through Enterprise IT Services
In Journey through Enterprise IT Services, Nadhan, HP Distinguished Technologist, explores the IT Services industry, and discusses technology trends in simplified terms.

Navigating the Big Data Traffic Jam - Are you there yet?

What happens when Big Data becomes truly Big, truly fast? was one of the sessions I walked into at HP Discover 2013 conference. Jaap Suermondt – Vice President, Analytics, HP Labs was showing a highway with a landscape that is characteristic of Nevada. And he asked a very simple question: Are we there yet? Straight road, no foreseeable obstacles -- apply some simple arithmetic and you know the answer.


Highway Nevada.png 

Now switch to a photograph that is more characteristic of the urban jungles of the world like Los Angeles, London or Mumbai. Try answering the same question now. Welcome to the world of Big Data and the inherent challenges of factoring in an incredible set of unknowns. How long will it take you to get there?

 Traffic Jam Downtown.png

There are many real-life situations where probabilistic reasoning techniques have to be applied to brontobytes of data to facilitate risk analysis of various alternatives. Dr. Michael Amylon at the Lucile Packard Children's Hospital at Stanford, for instance, started the pediatric bone marrow transplantation program in the 80s. As the program evolved, he wanted to better understand the risks and likely outcomes of re-transplanting kids who had a recurrence of their leukemia after they had undergone the transplant procedure. To that end, the team had to spend a lot of time in the library and on the phone to manually figure out that the risks of very serious side effects are significant. Here are the published results of this study. The definitive answer was published eventually in one of the 2001 editions of the Journal of Clinical Oncology.  Time taken to get there: Years.


Forward two decades to Dr. Jennifer Frankovich, a pediatrician at the same hospital. Frankovich faced critical clinical decisions on topics such as anticoagulation in a patient with systemic lupus erythematosus. Frankovich and the co-authors of this paper put their institution's electronic medical records (read big data) to unusual use to make an informed decision taking full advantage of the entire experience of Stanford in this field. Thank you, Big Data. Time taken to get there: Hours.


However, when it comes to Big Data, it is always rush hour:

  • Volume: Average stored data per company is 14.6 Petabytes1
  • Variety: 100M business events are generated per second2
  • Velocity: 1 Terabyte of machine data is created per hour3
  • Vulnerability: Average annual cost of information theft is $4M4


With such incessant inflow of data, we must get to the answers that matter in “decision time” through the application of innovative analytical techniques like those from HP Labs. Such techniques involve the core operations of summarizing data, parameterizing it, computing similarities, detecting anomalies, assessing and taking action to realizing value. Projected time to get there: minutes.


Doctors like Amylon and Frankovich have a vital need to navigate the Big Data traffic jam with their flashing red and blue lights with only one life-saving chance to make the right decision. It is decision time. Are we there yet? After attending Suermondt’s session, I walked out with the reassurance that we can be there: in seconds.


Connect with Nadhan on: Twitter, Facebook, Linkedin and Journey Blog.






discover13_LasVegas_signature (4).PNG


1 HP Internal Analysis

2 Gartner: Actionable Analytics Will Be Driven by Mobile, Social and Big Data Forces in 2013 and Beyond Published: 25 January 2013 ID: G0024716

3 Gartner: The Information of Things: Why Big Data Will Drive the Value in the Internet of Things Published: 17 April 2013 ID: G00249066.

4 Ponemon: 2012 Cost of Cyber Crime Study October 2012

Showing results for 
Search instead for 
Do you mean 
About the Author

Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.