Question: Tell us about your current role and how you interact with Apache Hadoop?

Todd: I’m a software engineer on Cloudera’s platform engineering team, where I spend most of my time contributing code to open source projects like Apache Hadoop and HBase. Most recently I’ve been implementing the automatic HA failover feature in Hadoop 2.0, but I’ve also spent a lot of time working on understanding and improving performance of the Hadoop stack.

Question: Tell us about your Hadoop Summit presentation?

Todd: At this year’s summit, I will be presenting about the internals of MapReduce and how you can tune your MapReduce jobs for optimal performance. A lot of developers see MapReduce as a black box, but looking inside that box can help you understand where you might have bottlenecks or easy opportunities to improve performance by changing a few configuration parameters.

Question: What do you expect will be the key takeaway for folks attending your session?

Todd: I hope attendees will walk away with a better understanding of each of the phases of MapReduce task execution, and a few key configuration parameters they can play with to get better performance without changing their code.

Question: What other presentations are you most looking forward to attending?

Todd: I’m really looking forward to Josh Wills’ talk on BranchReduce: Distributed Branch-and-Bound on YARN. There are a lot of optimization problems which can be solved by branch-and-bound approaches, and it’s only recently with the introduction of YARN that these types of algorithms can be efficiently built on Hadoop. Not only this a fresh topic, Josh is also an entertaining speaker!

We (the Apache MRUnit team) have just released Apache MRUnit 0.9.0-incubating (tarball, nexus, javadoc). Apache MRUnit is an Apache Incubator project that is a Java library which helps developers unit test Apache Hadoop MapReduce jobs. Unit testing is a technique for improving project quality and reducing overall costs by writing a small amount of code that can automatically verify the software you write performs as intended. This is considered a best practice in software development since it helps identify defects early, before they’re deployed to a production system.

The MRUnit project is quite active, 0.9.0 is our fourth release since entering the incubator and we have added 4 new committers beyond the projects initial charter! We are very interested in having new contributors and committers join the project! Please join our mailing list to find out how you can help!

The MRUnit build process has changed to produce mrunit-0.9.0-hadoop1.jar and mrunit-0.9.0-hadoop2.jar instead of mrunit-0.9.0-hadoop020.jar, mrunit-0.9.0-hadoop100.jar and mrunit-0.9.0-hadoop023.jar. The hadoop1 classifier is for all Apache Hadoop versions based off the 0.20.X line including 1.0.X. The hadoop2 classifier is for all Apache Hadoop versions based off the 0.23.X line including the unreleased 2.0.X.

This release contains 2 new features, 15 improvements and 6 bug fixes. I will highlight a few below:

• Support custom counter checking in MRUNIT-68
• runTest() should optionally ignore output order in MRUNIT-91
• Driver.runTest throws RuntimeException should it throw AssertionError in MRUNIT-54
• o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner in MRUNIT-67
• Better support for other serializations besides Writable: MRUNIT-70, MRUNIT-86, MRUNIT-99, MRUNIT-77
• Better error messages from validate, null checking and forgetting to set mappers and reducers: MRUNIT-74, MRUNIT-66, MRUNIT-65
• add static convenience methods to PipelineMapReduceDriver class in MRUNIT-89
• Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods in MRUNIT-48

Support custom counter checking

It has always been possible to check the counter values like so:

assertEquals(2, mapDriver.getCounters().findCounter(CustomMapper.CustomCounter.NAME).getValue());

but this is quite tedious. As such Jarek Jarcec Cecho (our second newest committer) added this feature directly to the drivers:

.withCounter(CustomMapper.CustomCounter.Name, 2);

runTest() should optionally ignore output order

Previous to this change MRUnit required Mapper/Reducer classes to output key value pairs in the order specified on the test. Well defined output order is common, but strictly not universal. Dave Beech (our newest committer) contributed a patch so you optionally turn this ordered requirement off by using:

.runTest(false)

instead of

.runTest()

Driver.runTest throws RuntimeException should it throw AssertionError

Previous versions of MRUnit threw a RuntimeException when a test failed. This worked well, but it meant that testing frameworks saw the the test as having erred, not failed. We have changed this to AssertionError so that testing frameworks see the tests as failed. The distinction is small but important.

o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner

Previously the MRUnit only supported a combiner in the mapred MapReduceDriver class but now the mapreduce MapReduceDriver also supports a combiner by:

MapReduceDriver.newMapReduceDriver(mapper, reducer, combiner)

or

.withCombiner(combiner) or .setCombiner(combiner)

Better support for other serializations besides Writable

Previous versions of MRUnit did not support JavaSerialization, Avro or other Serialization frameworks well. We improved alternative serialization support by not forcing K2 in MapReduceDriver to be Comparable and supporting serializations that cannot clone into a object or that do not have default constructors.

Better error messages from validate, null checking and forgetting to set mappers and reducers

We have improved checking of parameters passed to MRUnit and the error messages when the parameters are invalid including throwing NullPointerException immediately when receiving a null value and throwing a IllegalStateExcpetion when no mapper or reducer class is provided instead of a NullPointerException.

Add static convenience methods to PipelineMapReduceDriver class

add static convenience constructors similar to those in the other driver classes:

PipelineMapReduceDriver.newPipelineMapReduceDriver()

or

PipelineMapReduceDriver.newPipelineMapReduceDriver(list of Pair)

Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods

The OutputFromString and InputFromString methods are now deprecated because they required Text inputs or outputs with no way to enforce that the inputs or outputs from a mapper or reducer were actually Text. These methods also provided little convenience as a user can just pass the string they intended to new Text(string)

For those unfamiliar with the Apache HBase project, HBase is open source software that allows for real-time random read/write access to your Big Data in Apache Hadoop with very low latency and high scalability. Presentations in the HBaseCon 2012 Operations track will explain the state of HBase today, how to mitigate HBase failures, and best practices in cluster deployment and cluster monitoring.

Operations Track Presentations

Case Study of HBase Operations at Facebook
Ryan Thiessen, Facebook

At Facebook we have demanding HBase installations which are used for important and real-time user activity, so failure in an HBase cluster can be a serious issue requiring immediate attention. This session will discuss a variety of real-world scenarios where we have had failures in our HBase systems, how our Operations and Engineering teams have worked to mitigate many of these issues, and where HBase still needs to improve instead of relying on workarounds. The database should never go down. This talk is aimed at developers and other users of HBase (both current and potential) who are interested in an operational perspective on the state of HBase today.

HBase Backup
Sunil Sitaula, Cloudera
Madhuwanti Vaidya, Facebook

Reliable backup and recovery is one of the main requirements for any enterprise grade applications. HBase has been very well embraced by enterprises needing random, real-time read/write access with huge volumes of data and ease of scalability. As such they are looking for backup solutions that are reliable, easy to use, and can work with existing infrastructure. HBase comes with several backup options but there is a clear need to improve the native export mechanisms. This talk will cover various options that are available out of the box, their drawbacks and what various companies are doing to make backup and recovery efficient. In particular it will cover what Facebook has done to improve performance of backup and recovery process with minimal impact to production cluster.

HBase Security for the Enterprise
Andrew Purtell, Trend Micro

Trend Micro developed the new security features in HBase 0.92 and has the first known deployment of secure HBase in production. We will share our motivations, use cases, experiences, and provide a 10 minute tutorial on how to set up a test secure HBase cluster and a walk through of a simple usage example. The tutorial will be carried out live on an on-demand EC2 cluster, with a video backup in case of network or EC2 unavailability.

Lightning Talk | Developing Real Time Analytics Applications Using HBase in the Cloud
Rick Tucker, Sproxil

As small companies are adapting to handle Big Data, the cloud and HBase enable developers to leverage that data to provide revenue generating real-time applications. When developing a real-time application for an existing system, one must balance incrementing counters in real-time with MapReduce jobs over the same data-set. When maintaining an analytics platform, ensuring data accuracy is essential. At Sproxil, SMS logs are ingested into HBase at a growing rate and we report metrics such as SMS throughput, unique user growth over time, and return SMS user activity in real time. Sproxil provides a versatile analytics application enabling customers to handpick statistics on demand to gain market insights enabling them to react quickly to trends. This talk will identify the most profitable metrics and demonstrate how to calculate them using Map Reduce while continually updating data as it arrives.

Lightning Talk | Unique Sets on HBase and Hadoop
Elliott Clark, StumbleUpon

Determining the number of unique users that have interacted with a web page, game, or application is a very common use case. HBase is becoming an increasingly accepted tool for calculating sets or counts of unique individuals who meet some criteria. Computing these statistics can range in difficulty from very simple to very difficult. This session will explore how different approaches have worked or not worked at scale for counting uniques on HBase with Hadoop.

Lightning Talk | Orchestrating Clusters with Ironfan and Chef
Robert Berger, Runa

This session will discuss how you can represent your complete cluster with one config file and have it deployed to Cloud or Bare Metal. Infochmimps’ Ironfan builds on Opscode Chef to allow you to specify and orchestrate all flavors of your cluster’s deployment, monitoring and growth. Not just the core HBase/HDFS/MapReduce/Hive/Flume, etc. but all the elements including web / app servers, mysql, redis, rabbitmq and whatever other servers needed to implement your service. These same tools can manage variations for development, staging, R&D as well as the target “rendering” to various Clouds, Bare Metal or even Vagrant VMs.

Apache HBase is an open source software project that provides users with the ability to do real-time random read/write access to their data in Apache Hadoop. This means that when you want to use Hadoop for real-time data processing, HBase is the project you are looking for. The HBase developer community includes contributors from many organizations such as StumbleUpon, Facebook, Salesforce.com, TrendMicro, eBay, Explorys, Huawei and Cloudera. In fact, the HBaseCon Program Committee, constructors of the HBaseCon 2012 agenda, are all committers and PMC members of the Apache HBase project.

Presentations in the HBaseCon 2012 Development track will explain how and why HBase is built the way it is and will also cover HBase schema design and HDFS, the file system on which HBase is most commonly deployed. Some of the presentations for this track include the following below.

Development Track Presentations

Learning HBase Internals
Lars Hofhansl, Salesforce.com

The strength of an open source project resides entirely in its developer community; a strong democratic culture of participation and hacking makes for a better piece of software. The key requirement is having developers who are not only willing to contribute, but also knowledgeable about the project’s internal structure and architecture. This session will introduce developers to the core internal architectural concepts of HBase, not just “what” it does from the outside, but “how” it works internally, and “why” it does things a certain way. We’ll walk through key sections of code and discuss key concepts like the MVCC implementation and memstore organization. The goal is to convert serious “HBase Users” into HBase Developer Users,” and give voice to some of the deep knowledge locked in the committers’ heads.

Lessons learned from OpenTSDB
Benoit Sigoure, StumbleUpon

OpenTSDB was built on the belief that, through HBase, a new breed of monitoring systems could be created, one that can store and serve billions of data points forever without the need for destructive downsampling, one that could scale to millions of metrics, and where plotting real-time graphs is easy and fast. In this presentation we’ll review some of the key points of OpenTSDB’s design, some of the mistakes that were made, how they were or will be addressed, and what were some of the lessons learned while writing and running OpenTSDB as well as asynchbase, the asynchronous high-performance thread-safe client for HBase. Specific topics discussed will be around the schema, how it impacts performance and allows concurrent writes without need for coordination in a distributed cluster of OpenTSDB instances.

HBase Schema Design
Ian Varley, Salesforce.com

Most developers are familiar with the topic of “database design.” In the relational world, normalization is the name of the game. How do things change when you’re working with a scalable, distributed, non-SQL database like HBase? This talk will cover the basics of HBase schema design at a high level and give several common patterns and examples of real-world schemas to solve interesting problems. The storage and data access architecture of HBase (row keys, column families, etc.) will be explained, along with the pros and cons of different schema decisions.

HBase and HDFS: Past, Present, and Future
Todd Lipcon, Cloudera

Apache HDFS, the file system on which HBase is most commonly deployed, was originally designed for high-latency high-throughput batch analytic systems like MapReduce. Over the past two to three years, the rising popularity of HBase has driven many enhancements in HDFS to improve its suitability for real-time systems, including durability support for write-ahead logs, high availability, and improved low-latency performance. This talk will give a brief history of some of the enhancements from Hadoop 0.20.2 through 0.23.0, discuss some of the most exciting work currently under way, and explore some of the future enhancements we expect to develop in the coming years. We will include both high-level overviews of the new features as well as practical tips and benchmark results from real deployments.

Lightning Talk | Relaxed Transactions for HBase
Francis Liu, Yahoo!

For Map/Reduce programmers used to HDFS, the mutability of HBase tables poses new challenges: Data can change over the duration of a job, multiple jobs can write concurrently, writes are effective immediately, and it is not trivial to clean up partial writes. Revision Manager introduces atomic commits and point-in-time consistent snapshots over a table, guaranteeing repeatable reads and protection from partial writes. Revision Manager is optimized for a relatively small number of concurrent write jobs, which is typical within Hadoop clusters. This session will discuss the implementation of Revision Manager using ZooKeeper and coprocessors, and paying extra care to ensure security in multi-tenant clusters. Revision Manager is available as part of the HBase storage handler in HCatalog, but can easily be used stand-alone with little coding effort.

Lightning Talk | Living Data: Applying Adaptable Schemas to HBase
Aaron Kimball, WibiData

HBase application developers face a number of challenges: schema management is performed at the application level, decoupled components of a system can break one another in unexpected ways, less-technical users cannot easily access data, and evolving data collection and analysis needs are difficult to plan for. In this talk, we describe a schema management methodology based on Apache Avro that enables users and applications to share data in HBase in a scalable, evolvable fashion. By adopting these practices, engineers independently using the same data have guarantees on how their applications interact. As data collection needs change, applications are resilient to drift in the underlying data representation. This methodology results in a data dictionary that allows less-technical users to understand what data is available to them for analysis and inspect data using general-purpose tools (for example, export it via Sqoop to an RDBMS). And because of Avro’s cross-language capabilities, HBase’s power can reach new domains, like web apps built in Ruby.

HBaseCon 2012 is coming to San Francisco on May 22, less than 2 months away! The conference agenda continues to grow daily with exciting presentation content, which means it’s time to share a few sessions that have been added to the HBaseCon 2012 Applications Track.

Apache HBase is primarily used for real-time random read/write access to Big Data as part of the Hadoop ecosystem. Applications on Apache HBase are typically built to query Big Data with extremely low latency. Sessions in the HBaseCon 2012 Applications Tracks will include explanations of real-world HBase use cases, where HBase fits in an organization’s entire Big Data stack and when HBase is the “right” solution for an organization.

Applications Track Presentations

Building a Large Search Platform on a Shoestring Budget
Jacques Nadeau, CTO at YapMap

YapMap is a new kind of search platform that does multi-quanta search to better understand threaded discussions. This talk will cover how HBase made it possible for two self-funded guys to build a new kind of search platform. The presentation will discuss the YapMap data model and how YapMap uses row based atomicity to manage parallel data integration problems. Also learn where YapMap does not use HBase and instead uses a traditional SQL based infrastructure; the benefits of using MapReduce and HBase for index generation; the YapMap migration of tasks from a message based queue to the Coprocessor framework; and YapMap’s future Coprocessor use cases. Lastly, learn about YapMap’s operational experience with HBase, hardware choices and the challenges YapMap has faced.

Low Latency “OLAP” with HBase
Cosmin Lehene, Computer Scientist at Adobe Systems

Adobe Systems uses “SaasBase Analytics” to incrementally process large heterogeneous data sets into pre-aggregated, indexed views, stored in HBase to be queried in real- time. The goal was to process new data in real- time (currently minutes) and have it ready for a large number of concurrent queries that execute in milliseconds. This set Adobe’s problem apart from what is traditionally solved with Hive or Pig. This talk will describe the design and the strategies (and hacks) used to achieve low latency and scalability, from theoretical model to the entire process of ETL to warehousing and queries.

Growing Your Inbox, HBase at Tumblr
Blake Matheny, Director of Platform Engineering at Tumblr

This talk goes into detail about Tumblr’s experience developing Motherboy, an eventually consistent inbox style storage system built around HBase. The SLA, write concurrency, data volume, and failure modes for this application created a number of challenges in developing a solution. The user homing scheme introduced additional complexity that made capacity planning tricky as Tumblr tried to trade off availability and cost. Performance testing of our workload, and automation to support that testing, also provided a number of valuable lessons. This talk will be most useful to people considering HBase for their application, but will have enough detail to be useful to current HBase users as well.

Be sure to check the agenda in the coming weeks as we are adding more sessions soon. Remember that the Early Bird registration price expires this Friday April 6 so register soon to take advantage of the discount.

• Apache Hadoop 1.0.1
• Apache Zookeeper 3.4.3
• Apache HBase 0.92.0
• Apache Hive 0.8.1
• Apache Pig 0.9.2
• Apache Mahout 0.6.1
• Apache Oozie 3.1.3
• Apache Sqoop 1.4.1
• Apache Flume 1.0.0
• Apache Whirr 0.7.0

The list of supported Linux platforms has expanded to:

• Fedora 15 and 16
• CentOS and Red Hat Enterprise Linux 5 and 6
• SuSE Linux Enterprise 11
• Ubuntu 10.04 LTS
• Mageia 1

This, we hope, will make our user community’s experience running Apache Hadoop the most seamless Bigtop experience to date: just follow our Installation Guide and you will have your first pseudo-distributed Hadoop PI or Hive query running in no time.

If you’re thinking about deploying Bigtop to a fully-distributed cluster you might find our improved Puppet code to be of assistance. There is some brief documentation  on how to run our Puppet recipes in a master-less puppet configuration, but they should work just fine in a typical Puppet master setup as well.

Whatever you do, don’t forget to check us out at Apache and consider getting involved. Bigtop is a community-driven effort and we need your help. Of course, above all we need you to use Bigtop and give us your the feedback.

Happy Big Data discoveries,
Your faithful and tireless Bigtop development team!

Apache Hadoop consists of two primary components: HDFS and MapReduce. HDFS, the Hadoop Distributed File System, is the primary storage system of Hadoop, and is responsible for storing and serving all data stored in Hadoop. MapReduce is a distributed processing framework designed to operate on data stored in HDFS.

HDFS has long been considered a highly reliable file system.  An empirical study done at Yahoo! concluded that across Yahoo!’s 20,000 nodes running Apache Hadoop in 10 different clusters in 2009, HDFS lost only 650 blocks out of 329 million total blocks. The vast majority of these lost blocks were due to a handful of bugs which have long since been fixed.

Despite this very high level of reliability, HDFS has always had a well-known single point of failure which impacts HDFS’s availability: the system relies on a single Name Node to coordinate access to the file system data. In clusters which are used exclusively for ETL or batch-processing workflows, a brief HDFS outage may not have immediate business impact on an organization; however, in the past few years we have seen HDFS begin to be used for more interactive workloads or, in the case of HBase, used to directly serve customer requests in real time. In cases such as this, an HDFS outage will immediately impact the productivity of internal users, and perhaps result in downtime visible to external users. For these reasons, adding high availability (HA) to the HDFS Name Node became one of the top priorities for the HDFS community.

The remainder of this post discusses the implementation of a new feature for HDFS, called the “HA Name Node.” For a detailed discussion of other issues surrounding the availability of Hadoop as a whole, take a look at this excellent blog post by my colleague Eli Collins.

High-level Architecture

The goal of the HA Name Node project is to add support for deploying two Name Nodes in an active/passive configuration. This is a common configuration for highly-available distributed systems, and HDFS’s architecture lends itself well to this design. Even in a non-HA configuration, HDFS already requires both a Name Node and another node with similar hardware specs which performs checkpointing operations for the Name Node. The design of the HA Name Node is such that the passive Name Node is capable of performing this checkpointing role, thus requiring no additional Hadoop server machines beyond what HDFS already requires.

The HDFS Name Node is primarily responsible for serving two types of file system metadata: file system namespace information and block locations. Because of the architecture of HDFS, these must be handled separately.

Namespace Information

All mutations to the file system namespace, such as file renames, permission changes, file creations, block allocations, etc, are written to a persistent write-ahead log by the Name Node before returning success to a client call. In addition to this edit log, periodic checkpoints of the file system, called the fsimage, are also created and stored on-disk on the Name Node. Block locations, on the other hand, are stored only in memory. The locations of all blocks are received via “block reports” sent from the Data Nodes when the Name Node is started.

The goal of the HA Name Node is to provide a hot standby Name Node that can take over serving the role of the active Name Node with no downtime. To provide this capability, it is critical that the standby Name Node has the most complete and up-to-date file system state possible in memory. Empirically, starting a Name Node from cold state can take tens of minutes to load the namespace information (fsimage and edit log) from disk, and up to an hour to receive the necessary block reports from all Data Nodes in a large cluster.

The Name Node has long supported the ability to write its edit logs to multiple, redundant local directories. To address the issue of sharing state between the active and standby Name Nodes, the HA Name Node feature allows for the configuration of a special shared edits directory. This directory should be available via a network file system, and should be read/write accessible from both Name Nodes. This directory is treated as being required by the active Name Node, meaning that success will not be returned to a client call unless the file system change has been written to the edit log in this directory. The standby Name Node polls the shared edits directory frequently, looking for new edits written by the active Name Node, and reads these edits into its own in-memory view of the file system state.

Note that requiring a single shared edits directory does not necessarily imply a new single point of failure. It does, however, mean that the filer providing this shared directory must itself be HA, and that multiple network routes should be configured between the Name Nodes and the service providing this shared directory. Plans to improve this situation are discussed further below.

Block Locations

The other part of keeping the standby Name Node hot is making sure that it has up-to-date block location information. Since block locations aren’t written to the Name Node edit log, reading from the shared edits directory is not sufficient to share this file system metadata between the two Name Nodes. To address this issue, when HA is enabled, all Data Nodes in the cluster are configured with the network addresses of both Name Nodes. Data Nodes send all block reports, block location updates, and heartbeats to both Name Nodes, but Data Nodes will only act on block commands issued by the currently-active Name Node.

With both up-to-date namespace information and block locations in the standby Name Node, the system is able to perform a failover from the active Name Node to the standby with no delay.

Client Failover

Since multiple distinct daemons are now capable of serving as the active Name Node for a single cluster, the HDFS client must be able to determine which Name Node to communicate with at any given time. The HA Name Node feature does not support an active-active configuration, and thus all client calls must go to the active Name Node in order to be served.

To implement this feature, the HDFS client was extended to support the configuration of multiple network addresses, one for each Name Node, which collectively represent the HA name service. The name service is identified by a single logical URI, which is mapped to the two network addresses of the HA Name Nodes via client-side configuration. These addresses are tried in order by the HDFS client. If a client makes a call to the standby Name Node, a special result is returned to the client, indicating that it should retry elsewhere. The configured addresses are tried in order by the client until an active Name Node is found.

In the event that the active Name Node crashes while in the middle of processing a request, the client will be unable to determine whether or not the request was processed. For many operations such as reads (or idempotent writes such as setting permissions, setting modification time, etc), this is not a problem — the client may simply retry after the failover has completed. For others, the error must be bubbled up to the caller to be correctly handled. In the course of the HA project, we extended the Hadoop IPC system to be able to classify each operation’s idempotence using special annotations.

Current Status

Active development work began on the HA Name Node in August 2011, in a branch off of Apache Hadoop trunk. Development was done under the umbrella JIRAs HDFS-1623 and HADOOP-7454. Last Friday, March 2nd 2012 we merged this branch back into Apache Hadoop trunk. We closed over 170 individual JIRAs in the course of implementing this feature. The stated intention of the community is to merge this work from HDFS trunk into the 0.23 branch, where it will be released as an update of the Apache Hadoop 0.23 release line. Much of this work is already available as part of CDH4 beta 1, released on February 13th, 2012.

Once a failover has been initiated, the actual process of stopping the active and starting the standby Name Node takes a matter of seconds or less. This speed allows for little or no detectable service disruption during a failover. I’ve personally run hundreds of MR jobs over a running HA cluster, doing failovers back and forth between two HA Name Nodes, without any job failures.

This first implementation of the HA Name Node supports only manual failover — that is, failure of one of the Name Nodes is not automatically detected by the system, but rather requires intervention by an operator to initiate a failover between the Name Nodes. Though this is an obvious limitation, this version should still be useful to eliminate the need for planned HDFS downtime in many cases, e.g. changing the configuration of the Name Node, scheduled hardware maintenance of a Name Node, or scheduled OS upgrade of a Name Node.

Next Up

The highest priority feature to add to the HA Name Node implementation is support for automatically detecting the failure of the Active Name Node and initiating a failover to the Standby when it is determined that the Active is no longer functional. HDFS-3042 and its sub-tasks are actively being worked on to provide this functionality.

The dependence on an HA filer for HDFS edit logs is a limitation that we’d like to address in the near to medium term as well. Several different options have been discussed to address this:

• BookKeeper – BookKeeper is a highly available write-ahead logging system. Work has already been done to allow the HDFS Name Node to be able to write its edits log to BookKeeper, though this has not yet been tested with the HA Name Node.
• Multiple, non-HA filers – the HA Name Node presently only supports logging to a single shared edits directory. Perhaps the easiest improvement from the current situation would be to allow the Name Node to log to several shared edits directories, and require that all edits be logged to a quorum of shared edits directories. This proposal is being tracked by HDFS-2782.
• Stream edits to remote NNs – in addition to writing edits to a local file system, edit log entries could be sent directly to other Name Nodes over the network. The active Name Node would require a quorum of the involved Name Nodes to acknowledge receipt of the edits before responding with success to the client call.
• Store edit logs in HDFS itself – systems such as HBase already use HDFS to store a write-ahead log of all data mutations. If HDFS were extended to have a modicum of bootstrapping information, it is not inconceivable that HDFS edit logs could be stored in HDFS itself. This proposal is being discussed on HDFS-2601.

In the next few weeks, we will be evaluating all of these options and selecting one to implement.

Currently, deploying HA Name Nodes is somewhat cumbersome, requiring the operator to manually synchronize the on-disk metadata of the two Name Nodes. HDFS-2731 aims to improve the user experience of this deployment process by having the second Name Node automatically synchronize itself with the state of the first Name Node. This feature will make the process faster and less error prone.

Further Reading

Take a look at the CDH4 docs for detailed information on configuring the HA Name Node in CDH4.

Be on the lookout for an upcoming blog post from my colleague Todd Lipcon, which will go into greater detail about some of the specific challenges encountered while implementing the HA Name Node feature, and how these issues were overcome.

Acknowledgments

This work has been a community effort from the start, and represents the work of many contributors. Both the architecture and implementation were the collaborative effort of many. In particular, this work would not have been possible without contributions from Todd Lipcon, Eli Collins, Uma Maheswara Rao G, Bikas Saha, Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay Radia, Mingjie Lai, and Gregory Chanan. Also thanks to Dhruba Borthakur and Konstantin Shvachko for helpful design discussions and recommendations on testing. Thanks also to Stephen Chu, Wing Yew Poon, and Patrick Ramsey for their help in testing the HA Name Node.

One special item of note: Michael Stack announced HBaseCon 2012, taking place this spring in the Bay Area.  This inaugural conference will focus on the growth and education of the HBase community.  While details of the event are not yet published, the call for speakers is currently open.  Submit your abstract here.

Many of the talks focused on HBase operations.  Here’s a summary of those presentations:

Aravind Gottipati discussed the HBase deployments at StumbleUpon, reflecting on hardware, requirements, configuration, and monitoring tools. Aravind also pointed out some operational challenges StumbleUpon has faced, and suggested some improvements for future HBase versions.  [slides]

Next, Paul Tuckfield presented on HBase operations at Facebook. He shared interesting facts about their deployment, such as how their clusters span multiple racks to avoid network uplinks as a single point of failure, and how their clusters are as slow as their slowest region server.  [slides]

eBay’s Swati Agarwal and Thomas Pan gave a talk on eBay’s HBase deployments, sharing many statistics about their pre-production deployment, and discussed their need for well-distributed keys and the impact on their rowkey schema. They also talked about their HBase-related challenges, including a need for more stability and how upgrades incur significant downtime.  [slides]

By now, the meeting was running a bit behind schedule, so J.D. Cryans gave a quick presentation about some experiments he did at StumbleUpon involving different caching configurations and datasets. He showed his numbers in a couple of different runs based on a snapshot of the upcoming CDH3u3 release from Cloudera, which is currently in production at StumbleUpon.  The runs were with with no block cache, short-circuited reads, and 100% block cache. The main takeaway was that it is very important to have a good understanding of how much data that needs to be read for your specific use case, and how this data fits into HBase.  [slides]

In addition to the above talks, Tomer Shiran from MapR gave an overview of MapR’s product, and Mikhail Bautin from Facebook concluded the meetup with some slides about the various optimizations that Facebook has contributed back to the HBase community in the area of scanner performance.

Slides for all presentations are available here, and the link to the meetup web page is here.

Thanks to eBay for inviting the HBase User Group to their building, and providing the free pizza and beer.  See you at HBaseCon 2012!

Those who attended Hadoop World know how difficult navigating a route between two days of five parallel tracks of compelling content can be—particularly since Hadoop World 2011 consisted of sixty-five informative sessions about Hadoop. Understanding that it is nearly impossible to obtain and/or retain all the valuable information shared live at the event, we have compiled all the Hadoop World presentation slides and videos for perusing, sharing and for reference at your convenience. You can turn to these resources for technical Hadoop help and real-world production Hadoop examples, as well as information about advanced data science analytics.

I’d like to take this opportunity to again thank all who participated in making Hadoop World 2011 a success: sponsors, attendees, the Sheraton New York Hotel & Towers and the Hadoop World production team.

Hadoop World Resource Links

Hadoop World Agenda: http://www.hadoopworld.com/agenda/

• The Hadoop World Agenda has both Video and Slide (PPT) links for each presentation located in the presentations designated time slot.
• Place your cursor over a time slot to learn more about the presentations content.

Hadoop World Resources on Cloudera.com: http://www.cloudera.com/resources/Hadoop+World/

• Each Hadoop World presentation video and slide deck is listed in the Resources section of the Cloudera web site.
• Scroll the list to find a presentation of interest to you.

Notes

• Larry Feinsmith’s keynote presentation will not be listed in conjunction with JPMorgan & Chase’s wishes.
• The session, “Life in Hadoop Ops – Tales From the Trenches” does not have a corresponding slide deck as this session was a free flowing discussion panel.
• Hadoop World 2012 will take place in the fall. Specific location and dates will be disclosed in the future.

Apache Sqoop (incubating) was created to efficiently transfer bulk data between Hadoop and external structured datastores, such as RDBMS and data warehouses, because databases are not easily accessible by Hadoop. Sqoop is currently undergoing incubation at The Apache Software Foundation. More information on this project can be found at http://incubator.apache.org/sqoop.

The popularity of Sqoop in enterprise systems confirms that Sqoop does bulk transfer admirably. That said, to enhance its functionality, Sqoop needs to fulfill data integration use-cases as well as become easier to manage and operate.

What is Sqoop?

As described in a previous blog post, Sqoop is a bulk data transfer tool that allows easy import/export of data from structured datastores such as relational databases, enterprise data warehouses, and NoSQL systems. Using Sqoop, you can provision the data from an external system into HDFS, as well as populate tables in Hive and HBase. Similarly, Sqoop integrates with the workflow coordinator Apache Oozie (incubating), allowing you to schedule and automate import/export tasks. Sqoop uses a connector-based architecture which supports plugins that provide connectivity to additional external systems.

Figure 1: Sqoop 1.4.0-incubating Architecture

Sqoop’s Challenges

Sqoop has enjoyed enterprise adoption, and our experiences have exposed some recurring ease-of-use challenges, extensibility limitations, and security concerns that are difficult to support in the original design:

• Cryptic and contextual command line arguments can lead to error-prone connector matching, resulting in user errors
• Due to tight coupling between data transfer and the serialization format, some connectors may support a certain data format that others don’t (e.g. direct MySQL connector can’t support sequence files)
• There are security concerns with openly shared credentials
• By requiring root privileges, local configuration and installation are not easy to manage
• Debugging the map job is limited to turning on the verbose flag
• Connectors are forced to follow the JDBC model and are required to use common JDBC vocabulary (URL, database, table, etc), regardless if it is applicable

These challenges have motivated the design of Sqoop 2, which is the subject of this post. That said, Sqoop 2 is a work in progress whose design is subject to change.

Sqoop 2 will continue its strong support for command line interaction, while adding a web-based GUI that exposes a simple user interface. Using this interface, a user can walk through an import/export setup via UI cues that eliminate redundant options. Various connectors are added in the application in one place and the user is not tasked with installing or configuring connectors in their own sandbox. These connectors expose their necessary options to the Sqoop framework which then translates them to the UI. The UI is built on top of a REST API that can be used by a command line client exposing similar functionality. The introduction of Admin and Operator roles in Sqoop 2 will restrict ‘create’ access for Connections to Admins and ‘execute’ access to Operators. This model will allow integration with platform security and restrict the end user view to only operations applicable to end users.

Figure 2: Sqoop 2 Architecture

Ease of Use

Whereas Sqoop requires client-side installation and configuration, Sqoop 2 will be installed and configured server-side. This means that connectors will be configured in one place, managed by the Admin role and run by the Operator role. Likewise, JDBC drivers will be in one place and database connectivity will only be needed on the server. Sqoop 2 will be a web-based service: front-ended by a Command Line Interface (CLI) and browser and back-ended by a metadata repository. Moreover, Sqoop 2′s service level integration with Hive and HBase will be on the server-side. Oozie will manage Sqoop tasks through the REST API. This decouples Sqoop internals from Oozie, i.e. if you install a new Sqoop connector then you won’t need to install it in Oozie also.

Ease of Extension

In Sqoop 2, connectors will no longer be restricted to the JDBC model, but can rather define their own vocabulary, e.g. Couchbase no longer needs to specify a table name, only to overload it as a backfill or dump operation.

Common functionality will be abstracted out of connectors, holding them responsible only for data transport. The reduce phase will implement common functionality, ensuring that connectors benefit from future development of functionality.

Sqoop 2′s interactive web-based UI will walk users through import/export setup, eliminating redundant steps and omitting incorrect options. Connectors will be added in one place, with the connectors exposing necessary options to the Sqoop framework. Thus, users will only need to provide information relevant to their use-case.

With the user making an explicit connector choice in Sqoop 2, it will be less error-prone and more predictable. In the same way, the user will not need to be aware of the functionality of all connectors. As a result, connectors no longer need to provide downstream functionality, transformations, and integration with other systems. Hence, the connector developer no longer has the burden of understanding all the features that Sqoop supports.

Security

Currently, Sqoop operates as the user that runs the ‘sqoop’ command. The security principal used by a Sqoop job is determined by what credentials the users have when they launch Sqoop. Going forward, Sqoop 2 will operate as a server based application with support for securing access to external systems via role-based access to Connection objects. For additional security, Sqoop 2 will no longer allow code generation, require direct access to Hive and HBase, nor open up access to all clients to execute jobs.

Sqoop 2 will introduce Connections as First-Class Objects. Connections, which will encompass credentials, will be created once and then used many times for various import/export jobs. Connections will be created by the Admin and used by the Operator, thus preventing credential abuse by the end user. Furthermore, Connections can be restricted based on operation (import/export). By limiting the total number of physical Connections open at one time and with an option to disable Connections, resources can be managed.

Summary

As detailed in this presentation, Sqoop 2 will enable users to use Sqoop effectively with a minimal understanding of its details by having a web-application run Sqoop, which allows Sqoop to be installed once and used from anywhere. In addition, having a REST API for operation and management will help Sqoop integrate better with external systems such as Oozie. Also, introducing a reduce phase allows connectors to be focused only on connectivity and ensures that Sqoop functionality is uniformly available for all connectors. This facilitates ease of development of connectors.

We encourage you to participate in and contribute to Sqoop 2′s Design and Development (SQOOP-365).