IBM 5 in 5: Big Data & sensemaking engines start feeling like best friends

Edior's note: This IBM 5 in 5 post about Big Data and analytics is by Jeff Jonas, IBM Distinguished Engineer and Software Group's Chief Scientist of Entity Analytics. He blogs here, www.jeffjonas.typepad.com and can be found on Twitter @jeffjonas.

Click through rates for unsolicited advertisements range from near zero to roughly five percent. From the recipients’ point of view, just about every such communication is more time wasting spam.

Imagine a future where some sources of unsolicited advertisement produce such useful and perfectly timed ads, that you would signup. A world where virtually ever text message or email pushed at you is so relevant that this “service” starts feeling like a best friend.

Here at IBM we are working on sensemaking technologies where the data finds the data, and relevance finds you. Drawing on data points you approve (your calendar, routes you drive, etc.), predictions from such technology will seem ingenious. 

Imagine this: You actually sign up for such an unsolicited advertisement service. Three days later, it has suggested nothing. Why? Because there wasn’t anything worth your attention.  But on day four, 10 minutes before you hop in the car to drive to a meeting over coffee, you get this text: “Don’t take the 405, take the 110, then exit 10b, and 3 blocks up there’s a Starbucks.”

You think: Well that is nuts; my coffee shop meeting with my buddy Kenny is at least 15 miles from there. So you text “?” as a reply.

The answer: “Big accident on the 405, will affect Kenny too; already cleared this with him, and this Starbucks is the proposed compromise when considering all the factors.”

Now you could text another “?” to see what these other factors are, but you know it’s pointless. As you pull out of your driveway you are thinking “I love you” as you think about this new best friend in your phone.

This new era of Big Data is going to materially change what is possible in terms of prediction. Much like the difference between a big pile of puzzle pieces versus, the pieces already in some state of assembly – the latter required to reveal pictures. This is information in context, and while some pieces may be missing; some may be duplicates; others have errors; and a few are professionally fabricated lies – nonetheless, what can be known only emerges as the pieces come together (data finding data).

Big Data in context is one of the most significant trends in the information technology field.

This type of technology is going to be real time. Today, smart insight being produced at the end of every week, after a customer left a web site or after a bank already approved a loan only leaves organizations wondering why the answers are so late. Sensemaking systems will deliver sub-200ms insight, fast enough to do something about a transaction, while the transaction is still happening (aka perfect timing).

We at IBM are well down this road towards massively scalable sensemaking analytics.  And whether you will benefit by ingenious advertising services (versus spam) or better health care outcomes, the future will bring higher quality predictions, faster.

Think this topic is the most-likely prediction, or maybe just the most innovative, among the Next 5 in 5? Vote for it by clicking "like" on IBM's smarter planet.  


  1. Jeff, good article on how analytics can help people. Your examples describe how businesses can provide a better service to customers by figuring out what the customer really wants and providing the right information at the right time. This seems like a powerful way to satisfy clients (shoppers, people on way to meetings, etc). I wonder if there is a need for analytics that the customer would use? Some sort of application that can help me decide which company is giving me false or misleading information about their product or service. How do I, as a consumer, know that the timely "junk mail" is not just telling me something I want to hear? There is probably some semi-complicated equation that represents a business transaction. Some of the terms are related to the business, some are gov't regulations, some are customer related, etc. Where are the customer analytics in the equation? BTW, I also like how analytics are used in national security and law enforcement.

  2. Yes advertisers will patiently wait until just the right moment to harass you. That is not how marketers have worked in the past and probably won't be how they behave in the future. Maybe a few will behave at first, but as soon as some low life figures out how to abuse the system, they will.

  3. Wow, that seems... astonishingly annoying.

  4. This application needs info about me to be able to supply direct marketing and it needs a lot of info. To gather the info, i think there are two options:
    1. I have to give in a lot of imputs (route i take to work, where i work, hobbies, ...)
    2. This application will find info about me without me doing anything

    I am not willing to do option 1, why would i give more personal information away to marketeers? Option 2 would be creepy, it would be like somebody spying on me. Maybe future generations will care less, but i doubt it.

  5. No one will opt in to such a model, where so much personal data is disclosed to third parties ... unless of course the service was "irresistible."

  6. First good thought n ideology Jeff, Advertisers may become oppurtunists but there are ways where we are tackling such situations now, may be by ratings or trusted sources. The customer can have the choice of the suggestions to be opted and the service can be provided by a third party provider but not by a advertiser directly.
    But this can change the way customer analytics are moving.