IBM Research: Mixing media and metadata

Hillel Kolodner, VISION Cloud
principal investigator

Editor's note: This blog entry was contributed by Eliot Salant and Hillel Kolodner, who lead the VISION Cloud project out of IBM Research - Haifa

Traditional media companies are facing serious challenges in today's digital world. Paid circulation newspapers and journals are in decline as the media industry struggles to compete with the timeliness offered by social media. Even traditional television is now under pressure from content streamed over the Internet. And yet even the evolution of the media industry itself to more digital technologies and higher definition formats is creating difficulties, as the industry finds itself now needing to manage vast amounts of digital information and support producing content for a variety of user devices, such as smartphones, tablets and personal computers.

The VISION Cloud project is prototyping a next generation object store system that can tackle this challenge. This three-year research project, led by IBM and sponsored by the European Union’s FP7 initiative, will be demonstrated at the International Broadcasters Conference (IBC) in Amsterdam.

In traditional hierarchical file systems, information is stored in a tree structure – a directory path eventually leads to where files are kept; and within a directory, files are typically located by their name. While this approach more or less works on your local computer, (what was the name of that file again and where did I put it?), it does not scale well to an environment where new information is constantly being stored by multiple sources. (And, in fact, even on your local computer it is very limiting – how can you locate that great sunset photo on the lake from your summer vacation when you’ve named your files lake1.jpg, lake2.jpg, lake3.jpg, etc?)

VISION Cloud stores its information as objects, where objects contain not only the data (e.g. the bits of the photograph), but also user and system defined metadata. Allowing the user to tag objects with metadata allows for powerful searches on the data (“Find all photos where location = lake and time = sunset”), and for relationships to be created between logically linked objects.

Metadata and relationships can then be used in the production process to help locate, amongst the vast amounts of stored data, all the objects relevant to a story being produced. Locating all the information required for media production in a traditional file system would be like trying to buy everything you want for breakfast in a new, large supermarket – you would have to wander around to find the milk, then wander around until you located the breakfast cereal aisle, and then go through all of the types of cereal until you found your favorite etc. Using VISION Cloud would then be analogous to going to the same large and confusing store, but this time handing a list of what you want to someone waiting there, and then being handed a complete bag with all the requested items.

Another major innovation of VISION Cloud is the concept of storlets, units of computation that run close to the storage. Storlets can be used to transparently carry out operations such as transcoding on data to support multiple output formats, or analytics to support Big Data. The implications of storlets are quite powerful – stored data no longer needs to be transferred over the network to a local computer, processed and then put back onto the storage server, incurring both network transfer costs and latencies.

As an example, VISION Cloud can be defined so that whenever a high definition video file is stored, a storlet will automatically transcode it into a number of other formats. These formats could be for mobile devices or low resolution Youtube videos. All this is done without requiring user intervention to transfer the high definition file to a local computer, run a series of conversion programs, and then transfer the new files back to the storage server.

Over the last three years of the project, we have not only gained a lot of insights from the implementation and benchmarking of our research, but have also watched how the industry in general has been evolving, such as with the emergence of OpenStack, an open source cloud computing platform. We are now involved in taking what we’ve learned in VISION Cloud and implementing it on OpenStack’s Swift infrastructure. VISION Cloud’s usefulness in not limited just to media, but is equally relevant to other sectors, such as healthcare, manufacturing, and enterprise.

IBM, as the technical and managerial leader of VISION Cloud, will be accepting a prestigious Special Award at the IBC 2013 event on behalf the project’s technical partners.

You can follow the progress of VISION Cloud and obtain the latest documents from our web site: www.visioncloud.eu. Feel free to contact us if you’d like any additional information.