New IBMer begins work to make Watson work smarter, having made HBO’s Silicon Valley look smarter
to talk.” Generally not something you want to hear from your PhD advisor. But
when it was followed by, “Have you heard of HBO?”, then-Stanford student, now-IBM
Watson engineer Vinith Misra was intrigued (and as a film enthusiast, a little
amused). His advisor, Dr. Tsachy Weissman, had been contacted by the technical
advisor for HBO’s Silicon Valley in 2013 to help the show’s creators develop
fictionalized compression algorithms, and he wanted to bring Vinith on board.
series, about to begin its second season, follows the story of Richard Hendriks, a
brilliant, young programmer and founder of Silicon Valley
startup Pied Piper. He and his colleagues face a race against time – in the
form of competition from a bigger tech company, reverse-engineering their work
– to land funding from venture capitalists.
“The most memorable
algorithm I’ve heard has to be the solution to the countably-infinite prisoners-with-hats problem. You really have to admire the ludicrousness of
it (and the insanity of its claim).”
“I’m a big fan of the
director Bong Joon-Ho, and his film Memories of Murder is probably my
favorite. There is a new layer every time you watch it.”
Vinith, who studied and now works in information theory and machine learning, the
project held great promise. The challenge: create a lossless compression
algorithm more powerful and efficient than anything that currently exists – or,
rather, to make it look like they had created such an algorithm. They had to
come up with something that isn’t currently possible, but isn’t immediately
identifiable as such.
approached the challenge of writing the algorithm like he would any research
problem. “You have a few things to keep in mind when formulating a problem. It
needs to be impactful to people, it needs the potential to work, and the ideas
should be elegant, compelling, and provocative,” he said.
script in hand, Vinith knew that the algorithms were in essence their own
character in the show, so he needed to make sure his work could play the part. He
combined elements of lossy algorithms for their visual aesthetics on a
whiteboard, with the lossless algorithms the show called for, and even created
the Weissman Score — a fictitious compression benchmark that could fool even
the biggest fanboys.
and Weissman think it’s reasonable to imagine radically innovative compression
algorithms of this sort emerging in the future – which explains why these
algorithms would also survive a cursory glance from highly trained engineers. “If
you gave these ideas to a first-year grad student, they could run with them,”
Vinith said. It would require a more detailed analysis to uncover the
algorithms’ fundamental unfeasibility.
work finishing his doctoral studies in electrical engineering, Vinith didn’t
spend much time on set during filming. Instead, he would conduct what he called
“firefighting calls” with cast and crew, adjusting technical dialogue, finding
material for the whiteboards and sets, and helping producers and actors deal
with unexpected technical elements.
knew that the finale to the first season would involve a drastic alteration to
Pied Piper’s compression algorithm – an alteration that was the product of a
mathematical joke from a pivotal scene. Developing and coding even as his team
pitches the compression during the final episode, Richard comes up with a
groundbreaking development after a humorous debate about how to influence the
potential funders to choose their work. This breakthrough, which set the stage
for the second season, stemmed from the phrase “middle out.” Vinith even
published a 12-page analysis of the scenario that gives rise to the
breakthrough, and while it is highly detailed and mathematically rigorous, it is explicit; reader discretion is advised.
will continue to work on the show, but this month he also began a new chapter
with IBM’s Watson Group. “Watson is transitioning from the conceptual to
working on real problems…and I’m glad to be a part of it,” he said.
will work on Watson out of the IBM Research – Almaden lab, where he will be
developing algorithms to reason about concepts across a variety of domains, including
subjects ranging from food to baseball cards — all in an effort to make Watson
the future of the show, Vinith can’t reveal much. He continues to work on Pied
Piper’s breakthroughs and is looking forward to seeing the finished product of
his work on screen. “I hope people treat the show like science fiction,” he
said, “but the good kind.” He says it has been a fantastic and unique
experience, and one from which he has gained a greater appreciation for the
aesthetic side of what he does. “Algorithms and systems are designed to be used,
but the ideas behind them can often be compelling, even to non-technical people.”
Labels: algorithms, HBO, ibm research almaden, silicon valley, Weissman Score