We (the Apache MRUnit team) have just released Apache MRUnit 0.9.0-incubating (tarball, nexus, javadoc). Apache MRUnit is an Apache Incubator project that is a Java library which helps developers unit test Apache Hadoop MapReduce jobs. Unit testing is a technique for improving project quality and reducing overall costs by writing a small amount of code that can automatically verify the software you write performs as intended. This is considered a best practice in software development since it helps identify defects early, before they’re deployed to a production system.

The MRUnit project is quite active, 0.9.0 is our fourth release since entering the incubator and we have added 4 new committers beyond the projects initial charter! We are very interested in having new contributors and committers join the project! Please join our mailing list to find out how you can help!

The MRUnit build process has changed to produce mrunit-0.9.0-hadoop1.jar and mrunit-0.9.0-hadoop2.jar instead of mrunit-0.9.0-hadoop020.jar, mrunit-0.9.0-hadoop100.jar and mrunit-0.9.0-hadoop023.jar. The hadoop1 classifier is for all Apache Hadoop versions based off the 0.20.X line including 1.0.X. The hadoop2 classifier is for all Apache Hadoop versions based off the 0.23.X line including the unreleased 2.0.X.

This release contains 2 new features, 15 improvements and 6 bug fixes. I will highlight a few below:

• Support custom counter checking in MRUNIT-68
• runTest() should optionally ignore output order in MRUNIT-91
• Driver.runTest throws RuntimeException should it throw AssertionError in MRUNIT-54
• o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner in MRUNIT-67
• Better support for other serializations besides Writable: MRUNIT-70, MRUNIT-86, MRUNIT-99, MRUNIT-77
• Better error messages from validate, null checking and forgetting to set mappers and reducers: MRUNIT-74, MRUNIT-66, MRUNIT-65
• add static convenience methods to PipelineMapReduceDriver class in MRUNIT-89
• Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods in MRUNIT-48

Support custom counter checking

It has always been possible to check the counter values like so:

assertEquals(2, mapDriver.getCounters().findCounter(CustomMapper.CustomCounter.NAME).getValue());

but this is quite tedious. As such Jarek Jarcec Cecho (our second newest committer) added this feature directly to the drivers:

.withCounter(CustomMapper.CustomCounter.Name, 2);

runTest() should optionally ignore output order

Previous to this change MRUnit required Mapper/Reducer classes to output key value pairs in the order specified on the test. Well defined output order is common, but strictly not universal. Dave Beech (our newest committer) contributed a patch so you optionally turn this ordered requirement off by using:

.runTest(false)

instead of

.runTest()

Driver.runTest throws RuntimeException should it throw AssertionError

Previous versions of MRUnit threw a RuntimeException when a test failed. This worked well, but it meant that testing frameworks saw the the test as having erred, not failed. We have changed this to AssertionError so that testing frameworks see the tests as failed. The distinction is small but important.

o.a.h.mrunit.mapreduce.MapReduceDriver should support a combiner

Previously the MRUnit only supported a combiner in the mapred MapReduceDriver class but now the mapreduce MapReduceDriver also supports a combiner by:

MapReduceDriver.newMapReduceDriver(mapper, reducer, combiner)

or

.withCombiner(combiner) or .setCombiner(combiner)

Better support for other serializations besides Writable

Previous versions of MRUnit did not support JavaSerialization, Avro or other Serialization frameworks well. We improved alternative serialization support by not forcing K2 in MapReduceDriver to be Comparable and supporting serializations that cannot clone into a object or that do not have default constructors.

Better error messages from validate, null checking and forgetting to set mappers and reducers

We have improved checking of parameters passed to MRUnit and the error messages when the parameters are invalid including throwing NullPointerException immediately when receiving a null value and throwing a IllegalStateExcpetion when no mapper or reducer class is provided instead of a NullPointerException.

Add static convenience methods to PipelineMapReduceDriver class

add static convenience constructors similar to those in the other driver classes:

PipelineMapReduceDriver.newPipelineMapReduceDriver()

or

PipelineMapReduceDriver.newPipelineMapReduceDriver(list of Pair)

Test and Deprecate Driver.{*OutputFromString,*InputFromString} Methods

The OutputFromString and InputFromString methods are now deprecated because they required Text inputs or outputs with no way to enforce that the inputs or outputs from a mapper or reducer were actually Text. These methods also provided little convenience as a user can just pass the string they intended to new Text(string)

Starting way back in June,  we presented Top Ten Tips and Tricks for Hadoop Success. In this webinar we explained some tips that the Cloudera Solutions Architect team has picked up from implementing, deploying, and running Hadoop with our customers.

Top Ten Tips and Tricks for Hadoop Success (Link to video recording)

In August,  Jeff Hammerbacher presented 10 Common Hadoop-able Problems. This webinar highlighted ten Hadoop use cases deemed “Hadoop-able problems.” The use cases were distilled from Cloudera’s experience’s helping customers to solve their emerging data and business challenges.

10 Common Hadoop-able Problems Webinar (Link to video recording)

In September, Cloudera’s Tom White, author of Hadoop: The Definitive Guide, presented The State of Hadoop . As with most open source projects Hadoop is continuously evolving and this webinar was an update on what was new, changed, and possible with Apache Hadoop.

The State of Hadoop (Link to Cloudera resources video version)

In November, Anil Madan, Director of Engineering at eBay, and Cloudera’s Chief Scientist, Jeff Hammerbacher, presented Integrating Hadoop In Your Existing Data Warehouse and Business Intelligence Environment. This webinar explained the standard model for business intelligence and data warehousing, the three stages of Hadoop adoption, integration with Cloudera partnerships and Hadoop analytics at eBay.

Integrating Hadoop In Your Existing Data Warehouse and Business Intelligence Environment (Link to video recording)

Cloudera’s most recent webinar, Production-izing Hadoop: Lessons Learned, presented on December 8^th by Cloudera Solution Architect, Eric Sammer, shared key insights for installing, configuring and running Cloudera’s Distribution for Apache Hadoop when scaling a cluster for full production use. These best practices were gained from the hundreds of Hadoop deployments that Cloudera has been involved with.

Production-izing Hadoop: Lessons Learned (Link to video recording)

Cloudera will continue to provide more educational webinars in 2011. You can check our events page for upcoming activities or subscribe to our newsletter in the left side bar to stay informed on the latest from Cloudera. If there is a specific topic you would like to learn more about we’d love to hear from you. Please email your suggestions to community@cloudera.com.

• Hadoop 0.20.1 – 73 more patches of extra Hadoop’y goodness (that is 225 total patches over vanilla 0.20.1)
• Lots of libhdfs and fusefs love resulting in stability and usability improvements currently in use at scale
• HDFS fixes that improve the write pipeline
• Lots of general stability fixes for Hadoop
• Pig 0.5.0 release – Working out of the box with our Hadoop 0.18 and 0.20 builds
• Hive 0.4.1 release – Works with both of our Hadoop 0.18 and 0.20
• HBase 0.20.3 – We worked with the HBase team to bring the latest rpms to a yum repo near you

We are excited about our CDH2 release. Its running at scale at some really great companies. We are looking forward to promoting it to stable shortly and moving on to the next big thing, CDH3. I’ll let you know as soon as this happens. When CDH2 becomes stable, it also means that CDH3 is ready to start its journey through testing. Stay tuned for more details as to what CDH3 will encompass; I’ll just say that I’m pretty excited about it.

You can subscribe to our CDH mailing list (cdh-announce-subscribe@cloudera.com) to get information about new releases as we push them out. Check out the new release, and remember to let us know what you think!

We plan on pushing new packages into the testing repository every 3 to 6 weeks.  And it just so happens it is just about 3 weeks after we announced the first testing release. So it must be time for a new one. Here are some of the highlights:

• Hadoop 0.20.1 – Bumps the hadoop package up to the 0.20.1 release and adds 133 patches worth of extra goodness
• Alternatives for Hadoop – Now you can have both 0.18 and 0.20 installed and use the alternatives system to pick a default
• Pig 0.50 pre-release – We included some magic to get things working out of the box with both 0.18 and 0.20
• Hive 0.40 pre-release - Integrated with the alternatives setup out the box works with 0.18 and 0.20
• HBase 0.20 – We worked with the HBase team to bring rpms to a yum repo near you

A project as large as Hadoop is a communal effort. Cloudera is proud to be part of that community and hope that our products and services make Hadoop even more accessible to a wider audience. We’d like to thank everyone who contributes to Hadoop, especially the Yahoo! team for all of their hard work on getting 0.20.1 released, the developers at Facebook and those working on Pig, Hive and HBase.

We are just getting the ball rolling here. You can subscribe to our CDH mailing list (cdh-announce-subscribe@cloudera.com) to get information about new releases as we push them out. Check out the new release, and remember to let us know what you think!

MR jobs are particularly difficult to test thoroughly because they run in a distributed environment.  This post will give specific advice on how an engineering team might QA test its MR jobs. Note that Chapter 5 of Hadoop: The Definitive Guide gives specific code examples for testing an MR job.

As is the case with most testing scenarios, there are certain practices one can follow that have a low barrier to entry; such practices might do a fairly sufficient job of testing. There are also practices one can follow that are more complicated but perhaps result in more thorough testing. Let’s walk through some good QA practices, starting with the easiest and ending with the most complicated.

Traditional Unit Tests – JUnit, PyUnit, Etc.

Your MR job will probably have some functionality that can be tested in isolation using a unit-testing framework such as JUnit or PyUnit. For example, if your MR job does some document parsing in Java, your parse method can be tested using JUnit.

Using a traditional unit-testing framework is perhaps the easiest way to get started testing your MR jobs for a few reasons. First, they are already used by a huge collection of developers. Second, they can be invoked by and integrated into most popular continuous integration servers. Finally, they are simple, effective, and don’t require Hadoop daemons to be running.

Unit tests do a great job of testing each individual part of your MR job, but they do not test your MR job as a whole, and they do not test your MR job within Hadoop.

To start using traditional unit tests to improve the quality of your MR jobs, simply write your map and reduce functions in such a way that functionality is extracted from these functions into “private” helper functions (or classes), which can be tested in isolation. For example, put all your parse code in a “parse” method. Then, choose a unit-testing framework that fits your use case.

Most Python unit testing uses PyUnit, and most Java unit testing uses either JUnit or TestNG. Most popular programming languages have a standard unit-testing tool, so do some research to learn which framework is best for your needs. Finally, write tests in the framework of choice that thoroughly test each helper function you’ve defined for your map and reduce functions. Unit tests can then be run either on the local machine or on a continuous integration to ensure that the post conditions of the tested helper functions are met for a particular input.

MRUnit – Unit Testing for MR Jobs

MRUnit is a tool that was developed here at Cloudera and released back to the Apache Hadoop project. It can be used to unit-test map and reduce functions. MRUnit lets you define key-value pairs to be given to map and reduce functions, and it tests that the correct key-value pairs are emitted from each of these functions. MRUnit tests are similar to traditional unit tests in that they are simple, isolated, and don’t require Hadoop daemons to be running. Aaron Kimball has written a very detailed blog post about MRUnit here.

Local Job Runner Testing – Running MR Jobs on a Single Machine in a Single JVM

Traditional unit tests and MRUnit should do a fairly sufficient job detecting bugs early, but neither will test your MR jobs with Hadoop. The local job runner lets you run Hadoop on a local machine, in one JVM, making MR jobs a little easier to debug in the case of a job failing.

To enable the local job runner, set “mapred.job.tracker” to “local” and “fs.default.name” to “file:///some/local/path” (these are the default values).

Remember, there is no need to start any Hadoop daemons when using the local job runner. Running bin/hadoop will start a JVM and will run your job for you. Creating a new hadoop-local.xml file (or mapred-local.xml and hdfs-local.xml if you’re using 0.20) probably makes sense. You can then use the –config parameter to tell bin/hadoop which configuration directory to use. If you’d rather avoid fiddling with configuration files, you can create a class that implements Tool and uses ToolRunner, and then run this class with bin/hadoop jar foo.jar com.example.Bar -D mapred.job.tracker=local -D fs.default.name=file:/// (args), where Bar is the Tool implementation.

To start using the local job runner to test your MR jobs in Hadoop, create a new configuration directory that is local job runner enabled and invoke your job as you normally would, remembering to include the –config parameter, which points to a directory containing your local configuration files.

The -conf parameter also works in 0.18.3 and lets you specify your hadoop-local.xml file instead of specifying a directory with –config. Hadoop will run the job happily. The difficulty with this form of testing is verifying that the job ran correctly. Note: you’ll have to ensure that input files are set up correctly and output directories don’t exist before running the job.

Assuming you’ve managed to configure the local job runner and get a job running, you’ll have to verify that your job completed correctly. Simply basing success on exit codes isn’t quite good enough. At the very least, you’ll want to verify that the output of your job is correct. You may also want to scan the output of bin/hadoop for exceptions. You should create a script or unit test that sets up preconditions, runs the job, diffs actual output and expected output, and scans for raised exceptions. This script or unit test can then exit with the appropriate status and output specific messages explaining how the job failed.

Note that the local job runner has a couple of limitations: only one reducer is supported, and the DistributedCache doesn’t work (a fix is in progress).

Pseudo-distributed Testing – Running MR Jobs on a Single Machine Using Daemons

The local job runner lets you run your job in a single thread. Running an MR job in a single thread is useful for debugging, but it doesn’t properly simulate a real cluster with several Hadoop daemons running (e.g., NameNode, DataNode, TaskTracker, JobTracker, SecondaryNameNode). A pseudo-distributed cluster is composed of a single machine running all Hadoop daemons. This cluster is still relatively easy to manage (though harder than local job runner) and tests integration with Hadoop better than the local job runner does.

To start using a pseudo-distributed cluster to test your MR jobs in Hadoop, follow the aforementioned advice for using the local job runner, but in your precondition setup include the configuration and start-up of all Hadoop daemons. Then, to start your job, just use bin/hadoop as you would normally.

Full Integration Testing – Running MR Jobs on a QA Cluster

Probably the most thorough yet most cumbersome mechanism for testing your MR jobs is to run them on a QA cluster composed of at least a few machines. By running your MR jobs on a QA cluster, you’ll be testing all aspects of both your job and its integration with Hadoop.

Running your jobs on a QA cluster has many of the same issues as the local job runner. Namely, you’ll have to check the output of your job for correctness. You may also want to scan the stdin and stdout produced by each task attempt, which will require collecting these logs to a central place and grepping them. Scribe is a useful tool for collecting logs, though it may be superfluous depending on your QA cluster.

We find that most of our customers have some sort of QA or development cluster where they can deploy and test new jobs, try out newer versions of Hadoop, and practice upgrading clusters from one version of Hadoop to another. If Hadoop is a major part of your production pipeline, then creating a QA or development cluster makes a lot of sense, and repeatedly running jobs on it will ensure that changes to your jobs continue to get tested thoroughly. EC2 may be a good host for your QA cluster, as you can bring it up and down on demand. Take a look at our beta EC2 EBS Hadoop scripts if you’re interested in creating a QA cluster in EC2.

You should choose QA practices based on the importance of QA for your organization and also on the amount of resources you have. Simply using a traditional unit-testing framework, MRUnit and the local job runner can test your MR jobs thoroughly in a simple way without using too many resources. However, running your jobs on a QA or development cluster is naturally the best way to fully test your MR jobs with the expenses and operational tasks of a Hadoop cluster.

Do you have any helpful advice on beneficial QA practices for MR jobs?  Leave a comment :).