Data Interoperability

One might address this data interoperability in a variety of manners, including the following:

• Each system might be extended to read all the formats generated by the other systems. In the limit, this approach is not practical, since one cannot easily anticipate all of the formats new systems might generate.
• A library of data conversion programs could be assembled. This would unfortunately add a processing step, to convert the data between formats, slowing processing pipelines. Note however that many data conversion libraries operate by converting data into and out of a lingua franca format, using a single format as a pivot point.  This hints at a third possibility.
• Enable each system to read and write a common format.  Some systems might use other formats internally for performance, but whenever data is meant to be accessible to other systems a common format is used.

In practice all of these strategies will used to some extent. However the last strategy, a common format, seems to offer the most efficient path both in terms of engineering effort and processing time. This article will focus on the use of Avro’s data file format as such a common format.

Avro

Apache Avro is a data serialization format. Avro shares many features with Google’s Protocol Buffers and Apache Thrift, including:

• Rich data types.
• Fast, compact serialization.
• Support for many programming languages.
• Datatype evolution, also known as versioning.

Avro additionally provides some other features that are especially useful when storing data, namely:

• Avro defines a standard file format. Avro data files are self-describing, containing the full schema for the data in the file. Thus users can exchange Avro data files without also having to separately communicate metadata.  Once an Avro data file is written, one will always be able to read it, with full datatype information, without relying on any external software or metadata repository.  Avro data files also support compression, using Gzip or Snappy codecs.
• Avro’s serialization is more compact. Avro avoids storing a field identifier with each field value. For some datasets this savings can be significant.
• Avro implementations permit one to dynamically define new datatypes and to easily process previously unseen datatypes, without generation and loading of code. This provides natural support for script and query languages.
• Avro datatypes can define their sort-order, facillitating use of Avro data in MapReduce or ordered key/value stores.

Avro as a Common Format

Most of the major ecosystem components already or will soon support reading and writing Avro data files:

• MapReduce: I added support for Java MapReduce programs, included in Avro 1.4 and greater.
• Streaming: Tom White from Cloudera has added support for Hadoop Streaming programs to Avro (AVRO-808 & AVRO-830).
• Flume 0.9.2 and above support collecting data in Avro’s format (FLUME-133), contributed by Jon Hsieh of Cloudera.  Note also that Flume has recently been accepted into the Apache Incubator and will soon be known as Apache Flume.
• Sqoop 1.3 can import data as Avro data files in HDFS from a relational database (SQOOP-207), contributed by Tom White of Cloudera.  Sqoop has also recently been accepted into the Apache Incubator.
• Pig release 0.9 will be able read and write Avro data files (PIG-1748), thanks to Lin Guo and Jakob Homan at LinkedIn.
• Hive support for reading and writing Avro data files has been posted by Jakob Homan of LinkedIn, and should hopefully be included in Hive 0.9 (HIVE-895).
• HCatalog input and output drivers have been contributed by Tom White of Cloudera (HCATALOG-49).
• Thiruvalluvan M. G. from Yahoo! is working on a column-major format for Avro, which would accelerate Hive and Pig queries (AVRO-806).

For folks who are currently using Protocol Buffers or Thrift to store data, some tools for conversion are planned:

• Raghu Angadi from Twitter is working on tools that will let folks read and write their Thrift-defined data structures as Avro format data (AVRO-804).
• We also hope to soon add tools to convert between Protocol Buffers and Avro (AVRO-805).

At Cloudera we’re committed to helping Avro become a common format for the Hadoop ecosystem.  It’s great to see so many other companies and individuals also investing in Avro.

Aside from JavaOne ’10 having a new home as part of the greater Oracle OpenWorld conference, it was business as usual this year. Lots of great sessions, lots of interesting labs, and lots and lots of excited developers. (I think there may have even been more attendees than the conference planners expected.) This year Hadoop joined the ranks of the JavaOne hands-on labs with a lab co-produced by Oracle and Cloudera.

JavaOne Hands-on Lab S314413: Extracting Real Value from Your Data With Apache Hadoop was offered as a two-hour interactive lab designed to introduce attendees to the Hadoop environment, including writing a MapReduce program, writing a custom input reader, running, monitoring, and managing Hadoop jobs, and working with the Hive data warehousing platform. The lab was designed for participants with at least some Java programming experience but not necessarily any prior exposure to Hadoop.

In case you missed the lab at JavaOne, Oracle and Cloudera are both making the lab materials available online. Oracle will post the materials as part of the greater JavaOne presentations posting. Cloudera has already posted the lab materials online in the training section of the website.

When you download the zip file, in it you will find a lab workbook as a PDF in the root directory. At the back of the workbook, you will find an appendix that describes how to set up your own lab environment. I highly recommend that you grab the Cloudera Distribution for Hadoop (v2) to use as an environment for the lab. Cloudera even makes a prebuilt Linux/Hadoop environment available as a virtual machine. The lab was written for Solaris 11 Express and NetBeans, but you should still be able to do the lab on another OS with another IDE.

At JavaOne, the lab was very successful. Turnout was good and the comments were great! I’ve already incorporated lots of great feedback from that session into the set of lab materials that Cloudera is now hosting, but I’m always happy to hear any additional comments and/or feedback. Happy coding!

We are also offering training sessions for newcomers and experienced Hadoop users alike. Whether you are looking for an Introduction to Hadoop, Hadoop Certification, or you want to learn more about related Hadoop projects we have the training you are looking for.

Included with our top-notch Hadoop training you will have full access to Hadoop World free of charge.

Available Training Sessions include:

Oct 11:

Introduction to Hadoop: http://www.eventbrite.com/event/762326138

This one-day course provides a solid foundation for those seeking to understand large scale data processing with MapReduce and Hadoop. This session is designed for developers, analysts or system administrators that are new to Hadoop. This course provides the pre-requisite knowledge for the later classes: Developer Training, Administrator Training or Analyzing Data with Hive and Pig.

Hadoop Essentials For Managers: http://www.eventbrite.com/event/762237874

This one-day course will give decision-makers the information they need to know about Apache Hadoop, answering questions such as:

• When is Hadoop appropriate?
• What are people using Hadoop for?
• How does Hadoop fit into our existing environment?
• What do I need to know about choosing Hadoop?

Cloudera HUE SDK Training: http://www.eventbrite.com/event/764021208

Cloudera Hue provides developers with back end APIs to simplify interacting with Hadoop and front end APIs to deliver rich, web based, graphical user experiences. For this training, developers should have experience building web apps using modern MVC frameworks and Ajax. Experience with Python and Django is a strong plus. In this session we spend half the day covering the following topics, and the other half of the day interactively building applications with the Cloudera Hue team.

Oct 13 & 14:

Developer Training & Certification: http://www.eventbrite.com/event/762320120

In this two-day hands-on session, developers learn the MapReduce framework and how to write programs against its API. In addition to learning how to write individual MapReduce jobs, we discuss design techniques for larger workflows. This course also covers advanced skills for debugging MapReduce programs and optimizing their performance. At the end of the course, attendees have the option to take a certification exam documenting their understanding of the concepts taught during the training session.

Administrator Training & Certification: http://www.eventbrite.com/event/762677188

This two-day hands-on session covers the system administration aspects of Hadoop from installation and configuration to load balancing and tuning including diagnosing and solving problems in your deployment. At the end of the course, attendees have the option of taking a certification exam documenting their understanding of the concepts taught at the training session.

Analyzing Data with Hive and Pig: http://www.eventbrite.com/event/762318114

Cloudera’s two-day hands-on course on Hive and Pig is designed for people who have a basic understanding of how Hadoop works and want to utilize these languages for analysis of their data. Hive makes Hadoop accessible to users who already know SQL; Pig is similar to popular scripting languages. This course teachs you how to process data by using filters, joins, user-defined functions and more.

Oct 15:

HBase Training: http://www.eventbrite.com/event/762317111

This one-day hands-on course gives you the necessary knowledge for using HBase as a distributed data store to achieve low-latency queries and highly scalable throughput. This class covers the HBase architecture, data model, and Java API as well as advanced topics and best practices. This course is for developers who already have a basic understanding of Hadoop (Java experience is recommended).

Cloudera’s HBase course discusses use-cases for HBase, and covers the HBase architecture, schema modeling, access patterns, and performance considerations. During hands-on exercises, students write code to access HBase from Java applications, and use the HBase shell to manipulate data. Introduction to HBase also covers deployment and advanced features.

Our Hive and Pig course is designed for developers who are skilled with SQL or scripting languages, but who are not Java experts. Hive and Pig are two approaches which allow non-Java programmers to access and manipulate massive amounts of data while abstracting away the complexities of MapReduce. Hive offers an SQL-like interface, while Pig’s scripting language, named PigLatin, is very easy for developers learn. This course covers both technologies, and includes multiple hands-on exercises to reinforce key concepts.

Cloudera’s Hadoop for System Administrators course has recently been expanded from one day to two, and covers the important issues for System Administrators charged with looking after Hadoop clusters. Topics include planning and deploying the cluster, managing MapReduce jobs, scheduling jobs using the Fair Scheduler, cluster monitoring and troubleshooting, populating HDFS from existing relational database management systems with Sqoop, and using Flume to import logs and other files into HDFS.

Our most popular course, Hadoop for Developers, is a three-day offering which covers everything from an introduction to HDFS and MapReduce right through to advanced MapReduce APIs and algorithms. Students learn to build MapReduce jobs through a combination of instructor-led training and hands-on exercises; the course includes an exam offering students the chance to earn Cloudera Certified Hadoop Developer credentials.

A complete list of events including upcoming training is available here: http://www.cloudera.com/company/events/

• Thrift: A cross-language RPC framework that powers many of Facebook’s services, include search, ads, and chat. Among other things, Thrift defines a compact binary serialization format that is often used to persist data structures for later analysis.
• Scribe: A Thrift service for distributed logfile collection. Scribe was designed to run as a daemon process on every node in your data center and to forward log files from any process running on that machine back to a central pool of aggregators. Because of its ubiquity, a major design point was to make Scribe consume as little CPU as possible.
• Hive: Once the data has been serialized using Thrift and collected using Scribe, it can be loaded into a Hadoop cluster for analysis. Running Hive above your Hadoop cluster will allow you to query the data using a SQL-like syntax; Hive will also manage the partitioning of logs inside the Hadoop Distributed File System.
• Cassandra: If you’ve got millions of users requesting and updating data, Cassandra can help you scale with your community. Cassandra was designed to power inbox search at Facebook and is now storing an index of around 35 TB. Design points included incremental scalability and low system administration overhead; Cassandra could be useful in many places where a horizontally partitioned (“sharded”) MySQL instance is currently deployed.

I was recently invited by Robert Grossman of Open Data to speak about these projects at the inaugural Cloud Computing and Its Applications conference in Chicago. You can check out the slides from my talk below:

All of these projects have small but growing user communities. I hope you’ll find them useful for your data management projects, and I look forward to seeing a few new users on the mailing lists soon.

– Jeff Hammerbacher, VP Product and Chief Scientist, Cloudera