|Michael Factor, IBM Research|
Editor’s note: This post was authored by Michael Factor, IBM
I sat on the plane during the return trip from the OpenStack Summit in Tokyo, I couldn’t
help but reflect on the tremendous value of open source. One might ask: Why is
IBM so interested in open source?
not just about OpenStack and its various projects such as Nova,
or Swift. There are so many benefits in the broad world of open source
projects; some of which are being integrated with and used in IBM’s cloud,
including OpenStack and others. Examples of these projects include Spark, Docker,
among others. Each of these projects brings value on its own – not the least of
which is enabling consistency and choice as to where the workload runs since these
projects can all be deployed in local, dedicated, and public clouds. But the
really big value comes from combining projects to address real world problems.
It’s a definite case of the whole is greater than the sum of its parts. In
can look at all of these services as puzzle pieces that when put together solve
real world problems. In an ideal world, the different open source projects we
need would all be available as services on public, private, and dedicated
clouds. And it would be trivial to integrate the services by a point and a
example, we should be able to take a message bus like Kafka, and use a simple configuration
command with no coding to have it archive the messages in an object store
service. Once in that service, another point-and-click could pull the data into
an analytics engine (like Apache Spark).
The best (and worst) of open source
is the best of times for developers because of the wide array of open source puzzle
pieces out there, which can be used to build a solution. It is also the worst
of times since there is still work needed to make sure these pieces snap
together easily, and too often this burdens the developer.
my colleagues at IBM’s Research lab in Haifa and I are working on integrating
several of these projects – so we really can snap together those puzzle pieces!
example, my colleague Gil
Vernik, a cloud storage, security and analytics expert in the lab, is enabling Tachyon to use Swift as its persistent underlying storage system. Tachyon
is a very active, but relatively new, open source project that provides an
in-memory file system with automated tiering. While Tachyon is general purpose,
one of its most known use cases is improving Spark’s performance over
conventional stores. While Tachyon is in-memory, it needs some place to persist
data when it is no longer being used, or if it runs out of memory in higher
Gil first started this work, the main choices for persisting data were HDFS or Amazon
S3 (other options exist today). Both are good solutions but each has its
limitations: HDFS was not designed as a long term, multi-tenant store; Amazon’s
S3 is part of a public cloud. By adding support for OpenStack Swift, there is
now a multi-tenant, object store under Tachyon that can provide long term,
cost-effective persistent store.
Hadash, another colleague on our cloud team, is developing a solution to
aggregate and store messages from the Kafka message bus in OpenStack Swift.
Visual bookmark social media site Pinterest had a small project called Secor which knows how to subscribe to Kafka,
aggregate messages, and then put them in Amazon's S3. These messages are stored
sequentially in the object. The object can later be retrieved for batch
processing of the messages.
extended Secor so it could store messages via the Swift API. This gives Secor
users the choice of using Kafka-Secor-Swift in all three deployment models:
local, dedicated, and public. He then stored the data in Parquet format, not just as a list of messages. Parquet is designed
for objects or files that contain tabular data (like CSV). This data is stored in
a columnar format, enabling efficient compression and the retrieval of only
selected columns from the store. Lastly, Guy annotated the created objects with
metadata, such as the minimum or maximum values of various columns in the
table. We have already implemented prototypes of all three steps. A patch for
the first step is already part of the community Secor code, and we are starting
to work with the community on the second.
there’s what we are doing with our partners in the context of the EU COSMOS project. Paula Ta-Shma,
a cloud security and analytics expert on our team, presented this work at the recent Spark Summit in Amsterdam. The goal of this
work is to help improve the timeliness of Madrid’s city buses by automating their
reactions to changes in traffic.
is available from thousands of static sensors. Using NodeRed (an open source
project for defining data flows), the data is retrieved from the sensors and
put on Kafka. As described above, Secor aggregates, formats, and annotates this
data, before storing it in Swift. Spark is then used to analyze this data. Using
standard Spark machine learning, the solution defines the expected values for
speed and traffic density on different days and times. A complex event
processing engine, which also subscribes to the messages from the sensors,
takes the threshold values and can issue warnings to trigger corrective action,
such as rerouting a bus or changing the traffic light pattern, when needed.
examples clearly show the tremendous value in putting puzzle pieces together from
different open source projects to build something even better. We are
continuing to work on integrating services and look forward to describing some
of our other efforts in future blogs.
Labels: COSMOS, Gil Vernik, Guy Hadash, IBM Research - Haifa, IoT, Kafka, Michael Factor, open stack, Parquet, Paula Ta-shma, Secor, Spark, storage, Swift