IBM Research: Your personal shopping assistant

Have you ever found yourself in the supermarket staring at a shelf full of different cereal boxes, wishing someone could just point out the one with the best price, lowest sugar content, and the best reviews? New solutions for smarter retail will soon give customers the same type of information they get online when researching or comparing products – delivered inside the store as they shop.

A new augmented reality mobile shopping app being developed by researcher scientists at IBM’s lab in Haifa, Israel, is about to change the way we shop in stores. When shoppers use their smart phone or tablet video camera to pan over products on the shelf, the application will instantly display recommendations and offers based on their specific preferences.

"We're going way beyond simple facial recognition for products to provide superimposed information that points out the products shopper prefer – whether based on previous purchases, price, consumer rating, sodium content, environmentally friendly packaging, or other considerations," said Amnon Ribak, project leader for the augmented shopping advisor.

For example, a shopper looking for a high-quality facial moisturizer can specify important characteristics of the product, such as having a sun protection factor (SPF) of 15, is hypoallergenic, was not tested on animals, and that is on sale.

As the shopper points a smart device camera at the shelf of moisturizers in the pharmacy, the app recognizes the merchandise and displays information on the device’s screen, superimposed on the product images. It also highlights information based on the shopper's stated preferences, and can offer coupons or special discounts that may apply.

"The idea of standing in an aisle in the supermarket and having your mobile device point out the gluten-free cookies you need can be a real time saver," Ribak said. "This has the potential to completely change the shopping experience from one of hunting, reading, and searching to simply picking up those products you prefer."

Product recognition doesn't require bar codes or RFID tags. It is done using a blend of different approaches, including image recognition for the packaging that matches the colors and shapes to a database collection, optical character recognition techniques, as well as on-shelf context and positioning.

To develop this new technology, Ribak leads a team of research scientists whose image processing expertise has already contributed advances in such areas as license plate recognition at night or inclement weather for the Swedish toll road system; identification of malignant tumors in medical images; and optical recognition for ancient texts in libraries across Europe.

The researchers wrote algorithms that combine techniques used in facial recognition, color and shape matching, and associations with surrounding products. The app can take into account the mobile device’s camera angle, and distance from a shelf to help distinguish between products.

"Our first mission was to create our own mini-supermarket in the lab, so we could test the various approaches and challenges involved," Ribak said. "We've already submitted a number of patent applications based on the new techniques we discovered to overcome the challenges in recognizing products with less than ideal lighting, shadows, and reflections."

Aside from assisting shoppers, the new application could also help retail managers organize their stores, instantly point out what's missing on the shelves, or summarize which products were on sale during the week. As more and more shoppers use the application, it will give retailers deeper insight into how their customers shop – which is an analytics goldmine for optimized shelf and store arrangement.

Read more about IBM's e-commerce technology on the Smarter Planet blog.