The Next Big Thing
Posts about next generation technologies and their effect on business.

Bias and Big Data

data frying pan.pngI was looking at a NYT article titled Big Data is Opening Doors, but Maybe Too Many. The article discusses some of the unintended uses that the metadata and derived context information and privacy. In particular it talks about its use by the insurance industry…


While reading it, I had to laugh a bit about the "deep fat fryer" search example in the article.


“But to a data miner, tracking your click stream, this hunt could be read as a telltale signal of an unhealthy habit — a data-based prediction that could make its way to a health insurer or potential employer.”


If the search was taking place in the southern part of the US and there was concerned about people looking up topics like frying, they would be totally overwhelmed by the expectation of frying expertise. That doesn't mean that people fry every day, but knowing how (and possibly even being interested in the topic) is an expectation of life.


Being able to understand what is "normal" vs. aberrant behavior is a society issue. I am pretty sure what goes on in many of the big cities and considered normal behavior is defined as rude in rural areas (and vice versa). If insurance companies start using the information like this, it may turn out they turn into specialists for certain societies. If they do, they do – that’s why there is room for competition. This type of risk analysis is at the core of why insurance exists, to deny that insurance is based on the statistcal analysis of behavior is... interesting.


The use of big data techniques can definitely be used to classify and categorize (put structure upon) sets of “unstructured” information, including things like behavior. Let’s hope data scientists can get beyond coloring the analysis with their personal bias. 

junemanley | ‎04-11-2013 12:07 AM

You make a great point, Charlie. It goes without saying that organizations need to analyze their massive quantities of Big Data, but there’s still a need for human interpretation. On top of technology, organizations
should employ the expertise of data scientists (who, as Nadhan believes,
have the sexiest job) to filter through nuances like regional and social norms.



| ‎06-03-2013 11:35 AM
Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Showing results for 
Search instead for 
Do you mean 
About the Author
Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.