In Early 2010 at RichRelevance, we were searching for a new way to store our long lived data that was compact, efficient, and maintainable over time. We had been using Hadoop for about a year, and started with the basics – text formats and SequenceFiles. Neither of these were sufficient. Text formats are not compact enough, and can be painful to maintain over time. A basic binary format may be more compact, but it has the same maintenance issues as text. Furthermore, we needed rich data types including lists and nested records.

After analysis similar to Doug Cutting’s blog post, we chose Apache Avro. As a result we were able to eliminate manual version management, reduce joins during data processing, and adopt a new vision for what data belongs in our event logs. On Cyber Monday 2011, we logged 343 million page view events, and nearly 100 million other events into Avro data files.

Avoiding Version Management Baggage

Have you ever seen code for manual serialization version management like the below?

int version = input.readInt();
this.name = input.readName();
this.age = input.readInt();
if (version >= 2) {
  this.favoriteColor = input.readString();
} else {
  this.favoriteColor = "";
}

Manual version management is painful. If you evolve what data you store continuously, it does not take long to end up with dozens of versions. In order to read every version that has been written and stored, your code has to carry a lot of baggage.

With Avro, you can avoid writing code like the above. The concept is simple. Store the schema used to write your data along with your data, and use it to make the written data conform to the schema that the reader expects. If a field is missing, use the default. If has been removed or moved, handle it.

Over the last two years, we have doubled the complexity of our our page view schema across about 15 schema versions. There is not one line of code that deals with version management, and the current code can read any of the data written over that time. What may be a surprise to some is that our old code can read newly written data as well. The data is both forward and backward compatible, within the rules in the Avro Specification.

Leveraging Complex Data Types

Avro supports complex data types such as arrays, maps, enumerations, and nested records. The Avro data model makes it possible to serialize any non-recursive data structure, including trees and heterogeneous lists. We use this property to describe our events using Avro Schemas that map to natural object representations on our front end servers. For example, one of the elements in a page view is an array of product recommendation sets, each set containing a list of products displayed. Another element in a page view is what we call a page context – each type of page on a merchant’s site has a unique context that differs from other page types. A product page context is the product being displayed. A search page context is the search terms in the search query. There are about 30 page context types, and we represent the range of page context possibilities using an Avro union, so that all of these different event variations can be written in the same format and to the same log.

With a simpler data model, one might have had to log each context type separately, making it harder to get a full picture of what happened in a single request during analysis.

A New Vision for an Event Log

With the above properties of Avro, we were able to formulate a new vision for what an event log should be. The new model has the following properties:

• A singe HTTP request creates a single, atomic log event defined by an Avro schema
• The event contains all of the resolved request inputs
• The event contains the result of any decisions made during the request

Together, these imply that it is never necessary to join different sets of data together to reconstruct what happened in an individual request during analysis. This also significantly reduces the value of data contained in raw HTTP logs, since the Avro based logs become the origin for all major processing. Since raw HTTP logs are significantly larger than compressed binary structured data, this significantly reduces the size of data we must keep for long periods of time.

More Avro at RichRelevance

We have built Hive and Pig adapters to map our Avro data into these tools for ad-hoc queries and automated tasks. Additionally, we leverage the same Avro schemas from our log files to store click streams in HBase. We also use Avro to store data compactly in key-value stores.

The log file example is what I call a schema first use case of Avro, where we define a schema for log events that can be used across different systems over a long period of time. An alternative usage style is what I call code first, where you start with code and bind that to a serialization with a less schema-centric view. I feel that the code first usage style is more applicable for data that lives for short or medium time scales, such as with RPC or MapReduce intermediates. We will be deepening our investment in Avro and using it with code first use cases in the future, in the process working with the community to improve the developer experience for those use cases.

Avro is a growing, evolving project that I see as more broad than a serialization framework. At heart Avro is about applying a schema to data, in order to manipulate that data in well defined ways. Serialization, validation, and transformation are only some of the operations you can apply to data that conforms to a known schema. Over time the project will grow to have more and more functionality centered around operations you can apply to data that conforms to an Avro Schema. I look forward to working with the Avro community as the project continues to evolve!

Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. Flume is currently undergoing incubation at The Apache Software Foundation. More information on this project can be found at http://incubator.apache.org/flume. Flume NG is work related to new major revision of Flume and is the subject of this post.

Prior to entering the incubator, Flume saw incremental releases leading up to version 0.9.4. As Flume became adopted it became clear that certain design choices would need to be reworked in order to address problems reported in the field. The work necessary to make this change began a few months ago under the JIRA issue FLUME-728. This work currently resides on a separate branch by the name flume-728, and is informally referred to as Flume NG. At the time of writing this post Flume NG had gone through two internal milestones – NG Alpha 1, and NG Alpha 2 and a formal incubator release of Flume NG is in the works.

At a high-level, Flume NG uses a single-hop message delivery guarantee semantics to provide end-to-end reliability for the system. To accomplish this, certain new concepts have been incorporated into its design, while certain other existing concepts have been either redefined, reused or dropped completely.

In this blog post, I will describe the fundamental concepts incorporated in Flume NG and talk about it’s high-level architecture. This is a first in a series of blog posts by Flume team that will go into further details of it’s design and implementation.

Core Concepts

The purpose of Flume is to provide a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. The architecture of Flume NG is based on a few concepts that together help achieve this objective. Some of these concepts have existed in the past implementation, but have changed drastically. Here is a summary of concepts that Flume NG introduces, redefines, or reuses from earlier implementation:

• Event: A byte payload with optional string headers that represent the unit of data that Flume can transport from it’s point of origination to it’s final destination.
• Flow: Movement of events from the point of origin to their final destination is considered a data flow, or simply flow. This is not a rigorous definition and is used only at a high level for description purposes.
• Client: An interface implementation that operates at the point of origin of events and delivers them to a Flume agent. Clients typically operate in the process space of the application they are consuming data from. For example, Flume Log4j Appender is a client.
• Agent: An independent process that hosts flume components such as sources, channels and sinks, and thus has the ability to receive, store and forward events to their next-hop destination.
• Source: An interface implementation that can consume events delivered to it via a specific mechanism. For example, an Avro source is a source implementation that can be used to receive Avro events from clients or other agents in the flow. When a source receives an event, it hands it over to one or more channels.
• Channel: A transient store for events, where events are delivered to the channel via sources operating within the agent. An event put in a channel stays in that channel until a sink removes it for further transport. An example of channel is the JDBC channel that uses a file-system backed embedded database to persist the events until they are removed by a sink. Channels play an important role in ensuring durability of the flows.
• Sink: An interface implementation that can remove events from a channel and transmit them to the next agent in the flow, or to the event’s final destination. Sinks that transmit the event to it’s final destination are also known as terminal sinks. The Flume HDFS sink is an example of a terminal sink. Whereas the Flume Avro sink is an example of a regular sink that can transmit messages to other agents that are running an Avro source.

These concepts help in simplifying the architecture, implementation, configuration and deployment of Flume.

Flow Pipeline

A flow in Flume NG starts from the client. The client transmits the event to it’s next hop destination. This destination is an agent. More precisely, the destination is a source operating within the agent. The source receiving this event will then deliver it to one or more channels. The channels that receive the event are drained by one or more sinks operating within the same agent. If the sink is a regular sink, it will forward the event to it’s next-hop destination which will be another agent. If instead it is a terminal sink, it will forward the event to it’s final destination. Channels allow the decoupling of sources from sinks using the familiar producer-consumer model of data exchange. This allows sources and sinks to have different performance and runtime characteristics and yet be able to effectively use the physical resources available to the system.

Figure 1 below shows how the various components interact with each other within a flow pipeline.

Figure 1: Schematic showing logical components in a flow. The arrows represent the direction in which events travel across the system. This also illustrates how flows can fan-out by having one source write the event out to multiple channels.

By configuring a source to deliver the event to more than one channel, flows can fan-out to more than one destination. This is illustrated in Figure 1 where the source within the operating Agent writes the event out to two channels – Channel 1 and Channel 2.

Conversely, flows can be converged by having multiple sources operating within the same agent write to the same channel. A example of the physical layout of a converging flow is show in Figure 2 below.

Figure 2: A simple converging flow on Flume NG.

Reliability and Failure Handling

Flume NG uses channel-based transactions to guarantee reliable message delivery. When a message moves from one agent to another, two transactions are started, one on the agent that delivers the event and the other on the agent that receives the event. In order for the sending agent to commit it’s transaction, it must receive success indication from the receiving agent. The receiving agent only returns a success indication if it’s own transaction commits properly first. This ensures guaranteed delivery semantics between the hops that the flow makes. Figure 3 below shows a sequence diagram that illustrates the relative scope and duration of the transactions operating within the two interacting agents.

Figure 3: Transactional exchange of events between agents.

This mechanism also forms the basis for failure handling in Flume NG. When a flow that passes through many different agents encounters a communication failure on any leg of the flow, the affected events start getting buffered at the last unaffected agent in the flow. If the failure is not resolved on time, this may lead to the failure of the last unaffected agent, which then would force the agent before it to start buffering the events. Eventually if the failure occurs when the client transmits the event to its first-hop destination, the failure will be reported back to the client which can then allow the application generating the events to take appropriate action.

On the other hand, if the failure is resolved before the first-hop agent fails, the buffered events in various agents downstream will start draining towards their destination. Eventually the flow will be restored to its original characteristic throughput levels. Figure 4 below illustrates a scenario where a flow comprising of two intermediary agents between the client and the central store go through a transient failure. The failure occurs between agent 2 and the central store, resulting in the events getting buffered at the agent 2 itself. Once the failing link has been restored to normal, the buffered events drain out to the central store and the flow is restored to its original throughput characteristics.

Figure 4: Failure handling in flows. In (a) the flow is normal and events can travel from the client to the central store. In (b) a communication failure occurs between Agent 2 and the event store resulting in events being buffered on Agent 2. In (c) the cause of failure was addressed and the flow was restored and any events buffered in Agent 2 were drained to the store.

Wrapping up

In this post I described the various concepts that are a part of Flume NG and its high-level architecture. This is first of a series of posts from the Flume team that will highlight the design and implementation of this system. In the meantime, if you need anymore information, please feel free to drop an email on the project’s user or developer lists, or alternatively file the appropriate JIRA issues. Your contribution in any form is welcome on the project.

Links:

Project Website: http://incubator.apache.org/flume/

Flume NG Getting Started Guide: https://cwiki.apache.org/confluence/display/FLUME/Getting+Started

Mailing Lists: http://incubator.apache.org/flume/mail-lists.html

Issue Tracking: https://issues.apache.org/jira/browse/FLUME

IRC Channel: #flume on irc.freenode.net

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Abstract: CBS Interactive successfully adopted Hadoop as the web analytics platform, processing one Billion weblogs daily from hundreds of web site properties that CBS Interactive oversees. After introducing Lumberjack—the Extraction, Transformation and Loading framework we built based on python and streaming, which is under review for Open-Source release—Michael will talk about web metrics processing on Hadoop, focusing on weblog harvesting, parsing, dimension look-up, sessionization, and loading into a database. Since migrating processing from a proprietary platform to Hadoop, CBS Interactive achieved robustness, fault-tolerance and scalability, and significant reduction of processing time to reach SLA (over six hours reduction so far).

Gateway: Cluster Virtualization Framework
Konstantin Shvachko, eBay

Abstract: Access to Hadoop clusters through dedicated portal nodes (typically located behind firewalls and performing user authentication and authorization) can have several drawbacks — as shared multitenant resources they can create contention among users and increase the maintenance overhead for cluster administrators. This session will discuss the Gateway system, a cluster virtualization framework that provides multiple benefits: seamless access from users’ workplace computers through corporate firewalls; the ability to failover to active clusters for scheduled or unscheduled downtime, as well as the ability to redirect traffic to other clusters during upgrades; and user access to clusters running different versions of Hadoop.

SHERPASURFING – Open Source Cyber Security Solution
Wayne Wheeles, Novii Design

Abstract: Every day billions of packets, both benign and some malicious, flow in and out of networks. Every day it is an essential task for the modern Defensive Cyber Security Organization to be able to reliably survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform. SHERPASURFING is an open source platform built on the proven Cloudera’s Distribution including Apache Hadoop that enables organizations to perform the Cyber Security mission and at scale at an affordable price point. This session will include an overview of the solution and components, followed by a demonstration of analytics.

Integrating Hadoop with Enterprise RDBMS Using Apache SQOOP and Other Tools
Arvind Prabhakar, Cloudera
Guy Harrison, Quest Software

Abstract: As Hadoop graduates from pilot project to a mission critical component of the enterprise IT infrastructure, integrating information held in Hadoop and in Enterprise RDBMS becomes imperative. We’ll look at key scenarios driving Hadoop and RDBMS integration and review technical options. In particular, we’ll deep dive into the Apache SQOOP project, which expedites data movement between Hadoop and any JDBC database, as well as providing an framework which allows developers and vendors to create connectors optimized for specific targets such as Oracle, Netezza etc.

Next Generation Apache Hadoop MapReduce
Mahadev Konar, Hortonworks

Abstract: The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale, high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. We will be presenting the architecture and design of the next generation of map reduce and will delve into the details of the architecture that makes it much easier to innovate. We will also be presenting large scale and small scale comparisons on some benchmarks with MRV1.”

There are several training classes and certification sessions provided surrounding the Hadoop World conference. Don’t forget to register and become Cloudera Certified in Apache Hadoop.

As Hadoop has spread from web companies to other industries, the variety of data that is stored in HDFS has expanded dramatically. Hadoop clusters are being used to process satellite images, time series data, audio files, and seismograms. These formats are not a natural fit for the data schemas imposed by Pig and Hive, in the same way that structured binary data in a relational database can be a bit awkward to work with. For these use cases, we either end up writing large, custom libraries of user-defined functions in Pig or Hive, or simply give up on our high-level tools and go back to writing MapReduces in Java. Either of these options is a serious drain on developer productivity.

Today, we’re pleased to introduce Crunch, a Java library that aims to make writing, testing, and running MapReduce pipelines easy, efficient, and even fun. Crunch’s design is modeled after Google’s FlumeJava, focusing on a small set of simple primitive operations and lightweight user-defined functions that can be combined to create complex, multi-stage pipelines. At runtime, Crunch compiles the pipeline into a sequence of MapReduce jobs and manages their execution.

Example

Let’s take a look at the classic WordCount MapReduce, written using Crunch:

import com.cloudera.crunch.DoFn;
import com.cloudera.crunch.Emitter;
import com.cloudera.crunch.PCollection;
import com.cloudera.crunch.PTable;
import com.cloudera.crunch.Pipeline;
import com.cloudera.crunch.impl.mr.MRPipeline;
import com.cloudera.crunch.lib.Aggregate;
import com.cloudera.crunch.type.writable.Writables;

public class WordCount {
  public static void main(String[] args) throws Exception {
    // Create an object to coordinate pipeline creation and execution.
    Pipeline pipeline = new MRPipeline(WordCount.class);
    // Reference a given text file as a collection of Strings.
    PCollection<String> lines = pipeline.readTextFile(args[0]);

    // Define a function that splits each line in a PCollection of Strings into a
    // PCollection made up of the individual words in the file.
    PCollection<String> words = lines.parallelDo(new DoFn<String, String>() {
      public void process(String line, Emitter<String> emitter) {
        for (String word : line.split("\\s+")) {
          emitter.emit(word);
        }
      }
    }, Writables.strings()); // Indicates the serialization format

    // The Aggregate.count method applies a series of Crunch primitives and returns
    // a map of the unique words in the input PCollection to their counts.
    // Best of all, the count() function doesn't need to know anything about
    // the kind of data stored in the input PCollection.
    PTable<String, Long> counts = Aggregate.count(words);

    // Instruct the pipeline to write the resulting counts to a text file.
    pipeline.writeTextFile(counts, args[1]);
    // Execute the pipeline as a MapReduce.
    pipeline.done();
  }
}

Advantages

1. It’s just Java. Crunch shares a core philosophical belief with Google’s FlumeJava: novelty is the enemy of adoption. For developers, learning a Java library requires much less up-front investment than learning a new programming language. Crunch provides full access to the power of Java for writing functions, managing pipeline execution, and dynamically constructing new pipelines, obviating the need to switch back and forth between a data flow language and a real programming language.
2. Natural type system. Crunch supports reading and writing data that is stored using Hadoop’s Writable format or Apache Avro records. You do not need to write code that maps data stored in these formats into Crunch’s type system– they are supported natively. You can even mix and match Writable and Avro types within a single MapReduce: changing the Writables.strings() call to Avros.strings() in the WordCount example will run the MapReduce using Avro serialization instead of Writables.
3. A modular library released under the Apache License. Experts in machine learning, text mining, and ETL can craft libraries using Crunch’s data model, and other developers can use those libraries to build custom pipelines that operate on their data. For example, Crunch can be used to create the glue code that converts raw data into the structured input that a machine learning algorithm expects, and Crunch will compile the glue code and the machine learning algorithm into a single MapReduce.

Future Work

We are releasing Crunch as a development project, not a product. We’re eager for developers to play with it and tell us what they like and what they dislike. You can get started with Crunch by downloading it from Cloudera’s github repository here.

We have tested the library on a number of our use cases, but there will be bugs and rough edges that we will work out in the coming months. We gladly welcome contributions from the Hadoop ecosystem to help us improve Crunch as we prepare it for submission to the Apache Incubator, especially around:

• More efficient MapReduce compilation, including cost-based optimization,
• Support for HBase and HCatalog as data sources/targets,
• Tools and examples that build Crunch pipelines in other JVM languages, such as Scala, JRuby, Clojure, and Jython.

Using Hadoop for analytics and data processing requires loading data into clusters and processing it in conjunction with other data that often resides in production databases across the enterprise. Loading bulk data into Hadoop from production systems or accessing it from map reduce applications running on large clusters can be a challenging task. Users must consider details like ensuring consistency of data, the consumption of production system resources, data preparation for provisioning downstream pipeline. Transferring data using scripts is inefficient and time consuming. Directly accessing data residing on external systems from within the map reduce applications complicates applications and exposes the production system to the risk of excessive load originating from cluster nodes.

This is where Apache Sqoop fits in. Apache Sqoop is currently undergoing incubation at Apache Software Foundation. More information on this project can be found at http://incubator.apache.org/sqoop.

Sqoop allows easy import and export of data from structured data stores such as relational databases, enterprise data warehouses, and NoSQL systems. Using Sqoop, you can provision the data from external system on to HDFS, and populate tables in Hive and HBase. Sqoop integrates with Oozie, allowing you to schedule and automate import and export tasks. Sqoop uses a connector based architecture which supports plugins that provide connectivity to new external systems.

What happens underneath the covers when you run Sqoop is very straightforward. The dataset being transferred is sliced up into different partitions and a map-only job is launched with individual mappers responsible for transferring a slice of this dataset. Each record of the data is handled in a type safe manner since Sqoop uses the database metadata to infer the data types.

In the rest of this post we will walk through an example that shows the various ways you can use Sqoop. The goal of this post is to give an overview of Sqoop operation without going into much detail or advanced functionality.

Importing Data

The following command is used to import all data from a table called ORDERS from a MySQL database:

---
$ sqoop import --connect jdbc:mysql://localhost/acmedb \
    --table ORDERS --username test --password ****
---

In this command the various options specified are as follows:

• import: This is the sub-command that instructs Sqoop to initiate an import.
• –connect <connect string>, –username <user name>, –password
  <password>: These are connection parameters that are used to connect with the database. This is no different from the connection parameters that you use when connecting to the database via a JDBC connection.
• –table <table name>: This parameter specifies the table which will be imported.

The import is done in two steps as depicted in Figure 1 below. In the first Step Sqoop introspects the database to gather the necessary metadata for the data being imported. The second step is a map-only Hadoop job that Sqoop submits to the cluster. It is this job that does the actual data transfer using the metadata captured in the previous step.

Figure 1: Sqoop Import Overview

The imported data is saved in a directory on HDFS based on the table being imported. As is the case with most aspects of Sqoop operation, the user can specify any alternative directory where the files should be populated.

By default these files contain comma delimited fields, with new lines separating different records. You can easily override the format in which data is copied over by explicitly specifying the field separator and record terminator characters.

Sqoop also supports different data formats for importing data. For example, you can easily import data in Avro data format by simply specifying the option –as-avrodatafile with the import command.

There are many other options that Sqoop provides which can be used to further tune the import operation to suit your specific requirements.

Importing Data into Hive

In most cases, importing data into Hive is the same as running the import task and then using Hive to create and load a certain table or partition. Doing this manually requires that you know the correct type mapping between the data and other details like the serialization format and delimiters. Sqoop takes care of populating the Hive metastore with the appropriate metadata for the table and also invokes the necessary commands to load the table or partition as the case may be. All of this is done by simply specifying the option –hive-import with the import command.

----
$ sqoop import --connect jdbc:mysql://localhost/acmedb \
      --table ORDERS --username test --password **** --hive-import
----

When you run a Hive import, Sqoop converts the data from the native datatypes within the external datastore into the corresponding types within Hive. Sqoop automatically chooses the native delimiter set used by Hive. If the data being imported has new line or other Hive delimiter characters in it, Sqoop allows you to remove such characters and get the data correctly populated for consumption in Hive.

Once the import is complete, you can see and operate on the table just like any other table in Hive.

Importing Data into HBase

You can use Sqoop to populate data in a particular column family within the HBase table. Much like the Hive import, this can be done by specifying the additional options that relate to the HBase table and column family being populated. All data imported into HBase is converted to their string representation and inserted as UTF-8 bytes.

----
$ sqoop import --connect jdbc:mysql://localhost/acmedb \
        --table ORDERS --username test --password **** \
        --hbase-create-table --hbase-table ORDERS --column-family mysql
----

In this command the various options specified are as follows:

• –hbase-create-table: This option instructs Sqoop to create the HBase table.
• –hbase-table: This option specifies the table name to use.
• –column-family: This option specifies the column family name to use.

The rest of the options are the same as that for regular import operation.

Exporting Data

In some cases data processed by Hadoop pipelines may be needed in production systems to help run additional critical business functions. Sqoop can be used to export such data into external datastores as necessary. Continuing our example from above – if data generated by the pipeline on Hadoop corresponded to the ORDERS table in a database somewhere, you could populate it using the following command:

----
$ sqoop export --connect jdbc:mysql://localhost/acmedb \
        --table ORDERS --username test --password **** \
        --export-dir /user/arvind/ORDERS
----

In this command the various options specified are as follows:

• export: This is the sub-command that instructs Sqoop to initiate an export.
• –connect <connect string>, –username <user name>, –password
  <password>: These are connection parameters that are used to connect with the database. This is no different from the connection parameters that you use when connecting to the database via a JDBC connection.
• –table <table name>: This parameter specifies the table which will be populated.
• –export-dir <directory path>: This is the directory from which data will be exported.

Export is done in two steps as depicted in Figure 2. The first step is to introspect the database for metadata, followed by the second step of transferring the data. Sqoop divides the input dataset into splits and then uses individual map tasks to push the splits to the database. Each map task performs this transfer over many transactions in order to ensure optimal throughput and minimal resource utilization.

RecordBreaker is a project that automatically turns your text-formatted data (logs, sensor readings, etc) into structured data, without any need to write parsers or extractors. In particular, RecordBreaker targets Avro as its output format. The project’s goal is to dramatically reduce the time spent preparing data for analysis, enabling more time for the analysis itself.

Hadoop’s HDFS is often used to store large amounts of text-formatted data: log files, sensor readings, transaction histories, etc. Much of this data is “near-structured”: the data has a format that’s obvious to a human observer, but is not made explicit in the file itself.

Imagine you have a simple file listing, stored in listing.txt:

5 mjc staff 170 Mar 14 2011 14:14 bin
5 mjc staff 170 Mar 12 2011 05:13 build
1 mjc staff 11080 Mar 14 2011 14:14 build.xml

This “near-structured” data has metadata that is obvious to people who are familiar with file listings: a file owner, a file size, a last-modified date, etc. It’s easy for people despite the fact that certain strings, such as the date and time, cannot be parsed with simple whitespace breaks. In order for a user to process such data with MapReduce, Pig, or some similar tool, she must explicitly and laboriously reconstruct the metadata that is simple for anyone who just eyeballs the data.

Performing this reconstruction usually entails writing a parser or extractor, often one based on relatively brittle regular expressions. For some very common data, writing a good parser for it is probably worthwhile. However, there are also album track listings, temperature readings, flight schedules, and many other kinds of data; the number of good parsers we need to write gets large, quickly. Writing all of these straightforward extractors, again and again, is a time-consuming and error-prone pain for everyone. We believe it is a major obstacle to faster and easier data analytics.

The RecordBreaker project aims to automatically generate structure for text-embedded data. It consists of two main components.

LearnStructure

LearnStructure takes a text file as input and derives a parser that breaks lines of the file into typed fields. For example, the above file listing is broken into fields that include the file owner mjc, the group owner staff, etc. It emits all the schemas and code necessary to turn the raw text file into a file full of structured data. For example, we discover a JSON schema for the above file listing that looks like this:

{
   "type" : "record",
   "name" : "record_1",
   "namespace" : "",
   "doc" : "RECORD",
   "fields" : [ {
      "name" : "base_0",
      "type" : "int",
      "doc" : "Example data: '5', '5', '1'"
   }, {
      "name" : "base_2",
      "type" : "string",
      "doc" : "Example data: 'mjc', 'mjc', 'mjc'"
   }, {
      "name" : "base_4",
      "type" : "string",
      "doc" : "Example data: 'staff', 'staff', 'staff'"
   }, {
      "name" : "base_6",
      "type" : "int",
      "doc" : "Example data: '170', '170', '11080'"
   }, {
      "name" : "base_8",
      "type" : {
         "type" : "record",
         "name" : "base_8",
         "doc" : "",
         "fields" : [ {
            "name" : "month",
            "type" : "int",
            "doc" : ""
         }, {
            "name" : "day",
            "type" : "int",
            "doc" : ""
         }, {
            "name" : "year",
            "type" : "int",
            "doc" : ""
         } ]
      },
      "doc" : "Example data: '(14, 3, 2011)', '(12, 3, 2011)', '(14, 3, 2011)'"
   }, {
      "name" : "base_10",
      "type" : {
         "type" : "record",
         "name" : "base_10",
         "doc" : "",
         "fields" : [ {
            "name" : "hrs",
            "type" : "int",
            "doc" : ""
         }, {
            "name" : "mins",
            "type" : "int",
            "doc" : ""
         }, {
            "name" : "secs",
            "type" : "int",
            "doc" : ""
         } ]
      },
      "doc" : "Example data: '(14, 14, 0)', '(5, 13, 0)', '(14, 14, 0)'"
   }, {
      "name" : "base_12",
      "type" : "string",
      "doc" : "Example data: 'bin', 'build', 'build.xml'"
   } ]
}

Of course, the field names here are nonsense. All of the values, except for subfields of the date and timestamp records, have nondescriptive synthetically-generated names. The LearnStructure step attempts to recover the type of each field, but has no way to know its name or role. Obtaining names for these fields is the job of the SchemaDictionary step. For now, we just live with these bad synthetic names.

SchemaDictionary

SchemaDictionary takes data that’s been parsed by LearnStructure and applies topic-specific labels. For example, mjc should ideally be labelled as owner or perhaps user. The parsed staff data should be labelled as group.

The SchemaDictionary tool matches the newly-parsed data against a known database of structures. It finds the closest match, then assigns human-understandable names based on the best-matching previously-observed dataset. For example, with the above data and a small set of known datasets, SchemaDictionary can find that base_10 should actually be timemodified, that base_8 should be datemodified, and so on. Depending on the input data and the known database of structures, this labelling may be more or less accurate.

As mentioned, the target structured data format is Avro. Avro allows efficient cross-platform data serialization, similar to Thrift or Protocol Buffers. Data stored in Avro has many advantages (read Doug Cutting’s recent overview of Avro for more) and many tools either support Avro or will soon: Hadoop MapReduce, Apache Pig, and others.

Related Work

Our work on the LearnStructure component draws inspiration from the PADS research project (http://www.padsproj.org/index.html), in particular the paper From Dirt to Shovels: Fully Automatic Tool Generation from Ad Hoc Data, by Fisher, Walker, Zhu, and White. Published in POPL, 2008.. That paper itself draws on many papers in the area of information extraction and related fields. The authors have released code for their system, written in ML. ML is a great language, but is not well-suited to our needs: it is not supported by Avro, and is unlikely to appeal to many of the developers currently involved with the Hadoop ecosystem.

SchemaDictionary is more generally inspired by database schema mapping systems. (A famous example is described in The Clio Project: Managing Heterogeneity, by Miller, Hernandez, Haas, Yan, Ho, Fagin, and Popa, published in SIGMOD Record 30(1), March 2001, pp.78-83.) Schema mapping systems are usually designed to help database administrators merge existing databases; for example, when company A purchases company B and must then merge the employee lists. These tools are often expensive and expect a lot of administrator attention. In contrast, our SchemaDictionary is for busy data analysts who simply want to check out a novel dataset as quickly as possible. It is fast and simple, but can only handle relatively simple structures (rendering it inappropriate for databases, but on target for the kind of data that is popular in text-based formats).

Project

RecordBreaker works, but is not complete. It is just the start of what we hope will be many interesting applications and research projects. Please take a look at the code and documentation (the repo is at https://github.com/cloudera/RecordBreaker, and the tutorial is at http://cloudera.github.com/RecordBreaker/). Maybe you can pitch in and help.

Data Interoperability

One might address this data interoperability in a variety of manners, including the following:

• Each system might be extended to read all the formats generated by the other systems. In the limit, this approach is not practical, since one cannot easily anticipate all of the formats new systems might generate.
• A library of data conversion programs could be assembled. This would unfortunately add a processing step, to convert the data between formats, slowing processing pipelines. Note however that many data conversion libraries operate by converting data into and out of a lingua franca format, using a single format as a pivot point.  This hints at a third possibility.
• Enable each system to read and write a common format.  Some systems might use other formats internally for performance, but whenever data is meant to be accessible to other systems a common format is used.

In practice all of these strategies will used to some extent. However the last strategy, a common format, seems to offer the most efficient path both in terms of engineering effort and processing time. This article will focus on the use of Avro’s data file format as such a common format.

Avro

Apache Avro is a data serialization format. Avro shares many features with Google’s Protocol Buffers and Apache Thrift, including:

• Rich data types.
• Fast, compact serialization.
• Support for many programming languages.
• Datatype evolution, also known as versioning.

Avro additionally provides some other features that are especially useful when storing data, namely:

• Avro defines a standard file format. Avro data files are self-describing, containing the full schema for the data in the file. Thus users can exchange Avro data files without also having to separately communicate metadata.  Once an Avro data file is written, one will always be able to read it, with full datatype information, without relying on any external software or metadata repository.  Avro data files also support compression, using Gzip or Snappy codecs.
• Avro’s serialization is more compact. Avro avoids storing a field identifier with each field value. For some datasets this savings can be significant.
• Avro implementations permit one to dynamically define new datatypes and to easily process previously unseen datatypes, without generation and loading of code. This provides natural support for script and query languages.
• Avro datatypes can define their sort-order, facillitating use of Avro data in MapReduce or ordered key/value stores.

Avro as a Common Format

Most of the major ecosystem components already or will soon support reading and writing Avro data files:

• MapReduce: I added support for Java MapReduce programs, included in Avro 1.4 and greater.
• Streaming: Tom White from Cloudera has added support for Hadoop Streaming programs to Avro (AVRO-808 & AVRO-830).
• Flume 0.9.2 and above support collecting data in Avro’s format (FLUME-133), contributed by Jon Hsieh of Cloudera.  Note also that Flume has recently been accepted into the Apache Incubator and will soon be known as Apache Flume.
• Sqoop 1.3 can import data as Avro data files in HDFS from a relational database (SQOOP-207), contributed by Tom White of Cloudera.  Sqoop has also recently been accepted into the Apache Incubator.
• Pig release 0.9 will be able read and write Avro data files (PIG-1748), thanks to Lin Guo and Jakob Homan at LinkedIn.
• Hive support for reading and writing Avro data files has been posted by Jakob Homan of LinkedIn, and should hopefully be included in Hive 0.9 (HIVE-895).
• HCatalog input and output drivers have been contributed by Tom White of Cloudera (HCATALOG-49).
• Thiruvalluvan M. G. from Yahoo! is working on a column-major format for Avro, which would accelerate Hive and Pig queries (AVRO-806).

For folks who are currently using Protocol Buffers or Thrift to store data, some tools for conversion are planned:

• Raghu Angadi from Twitter is working on tools that will let folks read and write their Thrift-defined data structures as Avro format data (AVRO-804).
• We also hope to soon add tools to convert between Protocol Buffers and Avro (AVRO-805).

At Cloudera we’re committed to helping Avro become a common format for the Hadoop ecosystem.  It’s great to see so many other companies and individuals also investing in Avro.

I recently evaluated several serialization frameworks including Thrift, Protocol Buffersand Avro for a solution to address our needs as a demand side platform, but also for a protocol framework to use for the OpenRTB marketplace as well. The working draft of OpenRTB 2.0 uses simple JSON encoding, which has many advantages including simplicity and ubiquity of support. Many OpenRTB contributors requested we support at least one binary standard as well, to improve bandwidth usage and CPU processing time for real-time bidding at scale.

After reviewing many candidates, Apache Avro proved to be the best solution.

To demonstrate what differentiates Avro from the other frameworks (the link to my source code is at the end of this post), I put together a quick test of key features. The following are the key advantages of Avro 1.5:

* Schema evolution – Avro requires schemas when data is written or read. Most interesting is that you can use different schemas for serialization and deserialization, and Avro will handle the missing/extra/modified fields.

* Untagged data – Providing a schema with binary data allows each datum be written without overhead. The result is more compact data encoding, and faster data processing.

* Dynamic typing – This refers to serialization and deserialization without code generation. It complements the code generation, which is available in Avro for statically typed languages as an optional optimization.

Schema Evolution

This is the most exciting feature! It allows for building less decoupled and more robust systems. Below, I made significant changes to the schema, and things still work fine. This flexibility is a very interesting feature for rapidly evolving protocols like OpenRTB.

The following example demonstrates how this works. First, I created a new (example) schema. (Avro schemas are defined in JSON):

{
    "type": "record",
    "name": "Employee",
    "fields": [
        {"name": "name", "type": "string"},
        {"name": "age", "type": "int"},
        {"name": "emails", "type": {"type": "array", "items": "string"}},
        {"name": "boss", "type": ["Employee","null"]}
    ]
}

Next, I serialized a few records into a binary file using that schema. After that, I evolved my schema to the following:

{
    "type": "record",
    "name": "Employee",
    "fields": [
        {"name": "name", "type": "string"},
        {"name": "yrs", "type": "int", "aliases": ["age"]},
        {"name": "gender", "type": "string", "default":"unknown"},
        {"name": "emails", "type": {"type": "array", "items": "string"}}
    ]
}

This is a snapshot of the changes I made to the schema:

1) Renamed the field ‘age’ to ‘yrs’. Thanks to the alias feature, I can retrieve the value of ‘age’ by using the field name ‘yrs’.

2) Added a new ‘gender’ field, and defined a default value for it. This can be used to set values during deserialization as this field isn’t present in the original schema records.

3) Removed the ‘boss’ field.

Finally, I deserialized the binary data file with this new schema, and print it out. Success!

Untagged Data

There are two ways to encode data when serializing with Avro: binary or JSON. In the binary file, the schema is included at the beginning of file. I verified that the binary data was serialized untagged, which resulted in a smaller footprint. Another interesting point is that the schema can be defined, and then the data can be encoded/decoded in JSON; allowing you to define a schema for JSON rich data structures. Anyone needing to implement validation for a JSON protocol (like we did for OpenRTB) will appreciate this feature. And switching between binary and JSON encoding is simply a one-line code change. Switching JSON protocol to a binary format in order to achieve better performance is pretty straightforward with Avro.

Dynamic Typing

The key abstraction is GenericData.Record. This is essentially a set of name-value pairs where name is the field name, and value is one of the Avro supported value types. I found the dynamic typing to be very easy to use. When a generic record is instantiated, you have to provide a JSON-encoded schema definition. To access the fields, just use put/get methods like you would with any map. This approach is referred to as “generic” in Avro, in contrast to the “static” code generation approach also supported by Avro. The extra flexibility of the generic data handling has performance implications. But, this excellent benchmark – https://github.com/eishay/jvm-serializers/wiki/ – shows the penalty is minor, and the benefit is a simplified code base.

In conclusion, Avro is a unique serialization framework that works, although it took a bit of experimentation to get the code working. If you are interested in my Java code for an example of how Avro can be used, you can find it here: https://github.com/rfoldes/Avro-Test.

Robert Foldes

Senior Architect, DataXu

In my summer internship project at Cloudera, I added RPC tracing as a first-order feature of Apache Avro. Avro is a platform for data storage and exchange that caters to data-intensive, dynamic applications. My project focused on Avro’s RPC functionality.

It is common knowledge that tracing in distributed systems can be difficult. In user-facing web services, a front-end function may recursively trigger several function calls to mid and back-tier services. In offline processing, data-center storage layers may distribute data across several hosts, querying one or many of them when a client requests a file. In either case, the inter-dependency of components makes it difficult to pinpoint the source of a slowdown or hang-up when they inevitably occur.

It’s a new year and there’s a new Avro: 1.3.0.

Starting with Avro 1.3.0, the Avro team is releasing packages specially tailored to consumers of each language.  For example, Python users can download an egg, Java users can manage jars using Maven and C/C++ users can grab an autotools package ready to `./configure; make`.  Speaking of languages, we’re thrilled to announce that there’s a Ruby implementation for Avro now!

The Avro specification has been updated to include support for Avro RPC over HTTP.  Currently, only Java and Python support this new RPC specification but you can expect other languages to follow.  The Avro team also designed a test framework to ensure interoperability between any mix of Avro RPC clients and servers.

In Avro 1.3.0, there’s a new Avro data file format that is simpler, better suited for compression and provides support for streaming Avro data.  You’ll find support for this new file format in Ruby, Python, Java and C; giving you an array of languages to choose from for reading and writing Avro data.

There have been more features added to Avro than can fit in a single blog post but here are some of the highlights.

Java

• Substantial improvements to Reflection API. Now uses java.lang.String for Avro strings, either Java collections or arrays for Avro arrays, etc.
• New GenAvro tool provides a high-level syntax for schemas and protocols.
• Command-line tools jar for debugging.
• An RPC statistics system.
• Support for compression in data files
• Better Maven support including a mvn-install ant task to publish jar to local Maven repository, plus source and javadoc artifacts.
• Substantial performance improvements.
• Many bug fixes.

Python

• Rewritten to be slightly more Pythonic, simpler, and with greater test coverage
• RPC over HTTP support
• RPC and data file interoperability
• New command-line utility for sending and receiving RPCs
• Python eggs created

C++

The C++ implementation now uses autotools for its build, has a new API for checking schema resolution and provides a new tutorial to make it easier for you to get up and running with Avro in C++.

Ruby

Ruby hackers will be happy to hear that Ruby has been added to Avro 1.3.0 complete with support for the new data file format.

C

The C implementation has been completely rewritten from top to bottom and

• supports reading and writing the new Avro data file format
• adds a contact database example to make it easier for you learn the Avro C API
• provides schema validation, promotion and projection
• allows schema validation to be optional
• removes all dependencies on external libraries (e.g. APR, APR-util)
• embeds jansson for JSON parsing

You can contact the Avro team by visiting the #avro irc channel on irc.freenode.net or through one of the Avro mailing lists.  The Avro team is always open to suggestions about future features and would love to hear about your experiences using Avro 1.3.0.