Editor’s note: This
blog posting was authored by Michael Factor, Distinguished Engineer and expert on
Storage Systems at IBM Research - Haifa
Traditional file systems store our information in a tree structure. Although this works fine for
small collections of data – like those on our local hard drive – they are not
designed for the massive volumes of unstructured content most businesses are
collecting, storing, and accessing on the cloud.
A new method of storing information is called object storage. This approach stores information as objects. Each object contains the data (the
bits and bytes of our documents, movies, images, and so forth), together with
metadata that holds user- and system-defined tags. These smart data objects
include rich information – or metadata – that describes the content of the
data, how the object is related to other objects, how the data should be
handled, replicated, or backed up, and more.
storage can store objects, manage them, protect them, and so on – it doesn’t by
itself dramatically increase the rate at which we can extract value from
objects. But what if we could turn a software-defined object store into a smart storage platform?
“But what if we could turn a software-defined object store into a smart storage platform?”
the computation to the data
A new research
prototype called "storlets" holds the promise of greatly increasing the value
we get out of storage and the speed at which we can access what we need. A new software-defined
mechanism, storlets allow object storage to move the
computation to the data, instead of the system having to move the data to a
server to carry out the computation.
Imagine if every time you wanted to cook a
meal, you had to bring all your ingredients to a central neighborhood depot
where stoves, appliances, and cooking utensils were available to "process" the
food. That’s similar to the current situation with data on the cloud. Storlets
come to remedy this situation by moving the heavy lifting to where it’s
needed – similar to allowing you to cook everything in your kitchen, where all
your raw materials are already located.
The impact of
storlets is substantial. Stored data can be processed locally, and no longer
needs to be transferred over the network to a remote computer, processed, and
then put back onto the storage server – all of which incurs both network
transfer latencies and thus real dollar costs.
Our vision is to reduce
costs, increase flexibility and improve security by turning the object store
into a platform, and allowing the functionality of the object store to be
extended using software.
more value from data – much faster
Aside from saving
bandwidth by eliminating unnecessary data transfers, storlets are an ideal way
to introduce new services. Storlets can analyze
each object and extract its metadata, including size, subject, resolution,
format, and more. Since storlets are
dynamically loaded code, the function is only limited by the developer’s
For example, a
media company could upload a movie to an object store and have it automatically
generate a representative image. A physician doing rounds could have the object
store send only the portion of a patient’s x-ray needed to her wireless
device for immediate viewing. A lawyer could request that the object store
provide a document from a previous court case in a specific format. Or
pathologists could have images analyzed in the object store itself. All this
sophisticated computation is moved into the storage infrastructure via the
software of dynamically loaded storlets, making it faster, more flexible, and
far less expensive.
IBM researchers in
Haifa, Israel have been working on
prototyping storlets for several years in the context of European Union Research projects
such as ENSURE, VISION Cloud, and Forget IT. In short, we’re looking
forward to seeing more value from data in faster and more flexible ways.
Labels: IBM Research - Haifa, object storage, SDS, software-defined storage, storage, storlets, systems storage