Having a newfound dependency on the mobile web, internet users are increasingly looking for shortcuts and the easiest way to get things done. For example, our GPS devices automatically know where we are – providing us with driving directions instantly or the location of the nearest hamburger joint quickly – without having to think much about it. And now, IBM Research has taken web automation technologies one step further, with IBM CoScripter Conversational Interface: an Intelligent Assistant for interacting with the Web, nicknamed CoCo for this blog entry.
This new tool originated from CoScripter, a system for recording and automating repetitive activities on the web, such as paying a monthly credit card bill, requesting a vacation hold for postal mail, or checking flight arrival times. With CoCo, you can do all of this from your mobile device.
CoCo brings together web automation technologies - CoScripter and Highlight (http://www.almaden.ibm.com/cs/disciplines/user/#highlight) - making the web available to the user’s command through any simple text messaging interface. With this, users can, for example, send an SMS to CoCo and ask it to do something on the web, without actually being in front of a computer.
How does it work?
From a very high-level, the flow as seen in the picture is the following: - The user sends a message to CoCo via SMS, via e-mail or any other text messaging system
- CoCo takes the message and looks through this user's CoScripter scripts for a match
- When CoCo finds the script, it adds the necessary parameters from the user's command and then ships it for execution in a headless browser, running on Highlight
- Highlight executes the user's CoScript and returns the relevant portions of the Web site navigated
- CoCo sends the result back to the user via SMS, e-mail, etc.
Whenever there is a need for clarification or confirmation, CoCo can dialogue with the user to ask for additional information. In this way, CoCo becomes a communicative assistant with which you can talk to get computing tasks done.
Watch the video demonstration
IBM researchers Tessa Lau and Julian Ariel Cerruti presented this paper, "A Conversational Interface to Web Automation," at the 23rd ACM UIST Symposium in New York City on October 5. Co-authors are Guillermo Manzato, Mateo Bengualid, Jeffrey Bigham, and Jeffrey Nichols.
