Cloudera » pig

Using Apache Hadoop to Find Signal in the Noise: Analyzing Adverse Drug Events

Josh Wills — Wed, 16 Nov 2011 17:54:58 +0000

Last month at the Web 2.0 Summit in San Francisco, Cloudera CEO Mike Olson presented some work the Cloudera Data Science Team did to analyze adverse drug events. We decided to share more detail about this project because it demonstrates how to use a variety of open-source tools – R, Gephi, and Cloudera’s Distribution Including Apache Hadoop (CDH) – to solve an old problem in a new way.

Background: Adverse Drug Events

An adverse drug event (ADE) is an unwanted or unintended reaction that results from the normal use of one or more medications. The consequences of ADEs range from mild allergic reactions to death, with one study estimating that 9.7% of adverse drug events lead to permanent disability. Another study showed that each patient who experiences an ADE remains hospitalized for an additional 1-5 days and costs the hospital up to $9,000.

Some adverse drug events are caused by drug interactions, where two or more prescription or over-the-counter (OTC) drugs taken together leads to an unexpected outcome. As the population ages and more patients are treated for multiple health conditions, the risk of ADEs from drug interactions increases. In the United States, roughly 4% of adults older than 55 are at risk for a major drug interaction.

Because clinical trials study a relatively small number of patients, both regulatory agencies and pharmaceutical companies maintain databases in order to track adverse events that occur after drugs have been approved for market. In the United States, the FDA uses the Adverse Event Reporting System (AERS), where healthcare professionals and consumers may report the details of ADEs they experienced. The FDA makes a well-formatted sample of the reports available for download from their website, to the benefit of data scientists everywhere.

Methodology

Identifying ADEs is primarily a signal detection problem: we have a collection of events, where each event has multiple attributes (in this case, the drugs the patient was taking) and multiple outcomes (the adverse reactions that the patient experienced), and we would like to understand how the attributes correlate with the outcomes. One simple technique for analyzing these relationships is a 2×2 contingency table:

For All Drugs/Reactions:	Reaction = R_j	Reaction != R_j	Total
Drug = D_i	A	B	A + B
Drug != D_i	C	D	C + D
Total	A + C	B + D	A + B + C + D

Based on the values in the cells of the tables, we can compute various measures of disproportionality to find drug-reaction pairs that occur more frequently than we would expect if they were independent.

For this project, we analyzed interactions involving multiple drugs, using a generalization of the contingency table method that is described in the paper, “Empirical bayes screening for multi-item associations” by DuMouchel and Pregibon. Their model computes a Multi-Item Gamma-Poisson Shrinkage (MGPS) estimator for each combination of drugs and outcomes, and gives us a statistically sound measure of disproportionality even if we only have a handful of observations for a particular combination of drugs. The MGPS model has been used for a variety of signal detection problems across multiple industries, such as identifying fraudulent phone calls, performing market basket analyses and analyzing defects in automobiles.

Solving the Hard Problem with Apache Hadoop

At first glance, it doesn’t seem like we would need anything beyond a laptop to analyze ADEs, since the FDA only receives about one million reports a year. But when we begin to examine these reports, we discover a problem that is similar to what happens when we attempt to teach computers to play chess: a combinatorial explosion in the number of possible drug interactions we must consider. Even restricting ourselves to analyzing pairs of drugs, there are more than 3 trillion potential drug-drug-reaction triples in the AERS dataset, and tens of millions of triples that we actually see in the data. Even including the iterative Expectation Maximization algorithm that we use to fit the MGPS model, the total runtime of our analysis is dominated by the process of counting how often the various interactions occur.

The good news is that MapReduce running on a Hadoop cluster is ideal for this problem. By creating a pipeline of MapReduce jobs to clean, aggregate, and join our data, we can parallelize the counting problem across multiple machines to achieve a linear speedup in our overall runtime. The faster runtime for each individual analysis allows us to iterate rapidly on smaller models and tackle larger problems involving more drug interactions than anyone has ever looked at before.

Visualizing the Results

The output of our analysis is a collection of drug-drug-reaction triples that have very large disproportionality scores. But as we all know, correlation is not causation. The output of our analysis provides us with useful information that should be filtered and evaluated by domain experts and used as the basis for further study using controlled experiments.

With that caveat in mind, our analysis revealed a few drug pairs with surprisingly high correlations with adverse events that did not show up in a search of the academic literature: gabapentin (a seizure medication) taken in conjunction with hydrocodone/paracetamol was correlated with memory impairment, and haloperidol in conjunction with lorazepam was correlated with the patient entering into a coma.

Even with restrictive filters applied to the drug-drug-reaction triples, we still end up with tens of thousands of triples that score high enough to merit further investigation. In addition to looking at individual triples, we can also use graph visualization tools like Gephi to explore the macro-level structure of the data. Gephi has a number of powerful layout algorithms and filtering tools that allow us to impose structure on an undifferentiated mass of data points. Here is a graph in which the vertices are drugs and the thickness of the edges represent the number of high scoring adverse reactions that feature each pair of drugs:

We can also pan and zoom to different regions of the graph and highlight clusters of drug interactions. Here is a cluster of drugs that are used in treating HIV:

And here is a cluster of drugs that are used to fight cancer:

The combination of Apache Hadoop, R, and Gephi changes the way we think about analyzing adverse drug events. Instead of focusing on a handful of outcomes, we can process all of the events in the data set at the same time. We can try out hundreds of different strategies for cleaning records, stratifying observations into clusters, and scoring drug-reaction tuples, run everything in parallel, and analyze the data at a fraction of the cost of a traditional supercomputer. We can render the results of our analyses using visualization tools that can be used by domain experts to explore relationships within our data that they might never have thought to look for. By dramatically reducing the costs of exploration and experimentation, we foster an environment that enables innovation and discovery.

Open Data, Open Analysis

This project was possible because the FDA’s Center for Drug Evaluation and Research makes a portion of their data open and available to anyone who wants to download it. In turn, we are releasing a well-commented version of the code we used to analyze that data – a mixture of Java, Pig, R, and Python – on the Cloudera github repository under the Apache License. We also contributed the most useful Pig function developed for this project, which computes approximate quantiles for a stream of numbers, to LinkedIn’s datafu library. We hope to collaborate with the community to improve the models over time and create a resource for students, researchers, and fellow data scientists.

Hadoop World 2011: A Glimpse into Development

Jon Zuanich — Wed, 12 Oct 2011 13:00:42 +0000

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Abstract: CBS Interactive successfully adopted Hadoop as the web analytics platform, processing one Billion weblogs daily from hundreds of web site properties that CBS Interactive oversees. After introducing Lumberjack—the Extraction, Transformation and Loading framework we built based on python and streaming, which is under review for Open-Source release—Michael will talk about web metrics processing on Hadoop, focusing on weblog harvesting, parsing, dimension look-up, sessionization, and loading into a database. Since migrating processing from a proprietary platform to Hadoop, CBS Interactive achieved robustness, fault-tolerance and scalability, and significant reduction of processing time to reach SLA (over six hours reduction so far).

Gateway: Cluster Virtualization Framework
Konstantin Shvachko, eBay

Abstract: Access to Hadoop clusters through dedicated portal nodes (typically located behind firewalls and performing user authentication and authorization) can have several drawbacks — as shared multitenant resources they can create contention among users and increase the maintenance overhead for cluster administrators. This session will discuss the Gateway system, a cluster virtualization framework that provides multiple benefits: seamless access from users’ workplace computers through corporate firewalls; the ability to failover to active clusters for scheduled or unscheduled downtime, as well as the ability to redirect traffic to other clusters during upgrades; and user access to clusters running different versions of Hadoop.

SHERPASURFING – Open Source Cyber Security Solution
Wayne Wheeles, Novii Design

Abstract: Every day billions of packets, both benign and some malicious, flow in and out of networks. Every day it is an essential task for the modern Defensive Cyber Security Organization to be able to reliably survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform. SHERPASURFING is an open source platform built on the proven Cloudera’s Distribution including Apache Hadoop that enables organizations to perform the Cyber Security mission and at scale at an affordable price point. This session will include an overview of the solution and components, followed by a demonstration of analytics.

Integrating Hadoop with Enterprise RDBMS Using Apache SQOOP and Other Tools
Arvind Prabhakar, Cloudera
Guy Harrison, Quest Software

Abstract: As Hadoop graduates from pilot project to a mission critical component of the enterprise IT infrastructure, integrating information held in Hadoop and in Enterprise RDBMS becomes imperative. We’ll look at key scenarios driving Hadoop and RDBMS integration and review technical options. In particular, we’ll deep dive into the Apache SQOOP project, which expedites data movement between Hadoop and any JDBC database, as well as providing an framework which allows developers and vendors to create connectors optimized for specific targets such as Oracle, Netezza etc.

Next Generation Apache Hadoop MapReduce
Mahadev Konar, Hortonworks

Abstract: The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale, high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. We will be presenting the architecture and design of the next generation of map reduce and will delve into the details of the architecture that makes it much easier to innovate. We will also be presenting large scale and small scale comparisons on some benchmarks with MRV1.”

There are several training classes and certification sessions provided surrounding the Hadoop World conference. Don’t forget to register and become Cloudera Certified in Apache Hadoop.

New Features in Apache Pig 0.8

John Kreisa — Tue, 21 Dec 2010 14:34:58 +0000

This is a guest post contributed by Dmitriy Ryaboy (@squarecog) and was originally published in his blog on December 19th. We thought the information was valuable enough that it was worth reposting to spread the word even further.

The Pig 0.8 release includes a large number of bug fixes and optimizations, but at the core it is a feature release. It’s been in the works for almost a full year (most of the work on 0.7 was completed by January of 2009, although it took a while to actually get the release out), and the amount of time spent on 0.8 really shows.

I meant to describe these in detail in a series of posts, but it seems blogging regularly is not my forte. This release is so chock-full of great new features, however, that I feel compelled to at least list them. So, behold, in no particular order, a non-exhaustive list of new features I am excited about in Pig 0.8:

Support for UDFs in scripting languages

This is exactly what it sounds like — if your favorite language has a JVM implementation, it can be used to create Pig UDFs.

Pig now ships with support for UDFs in Jython, but other languages can be supported by implementing a few interfaces. Details about the Pig UDFs in Python can be found here: http://pig.apache.org/docs/r0.8.0/udf.html#Python+UDFs

This is the outcome of PIG-928; it was quite a pleasure to watch this develop over time — while most Pig tickets wind up getting worked on by at most one or two people, this turned into a collaboration of quite a few developers, many of them new to the project — Kishore Gopalakrishna’s patch was the initial conversation starter, which was then hacked on or merged into similar work by Woody Anderson, Arnab Nandi, Julien Le Dem, Ashutosh Chauhan and Aniket Mokashi (Aniket deserves an extra shout-out for patiently working to incorporate everyone’s feedback and pushing the patch through the last mile).

PigUnit

A contribution by Romain Rigaux, PigUnit is exactly what it sounds like — a tool that simplifies the Pig users’ lives by giving them a simple way to unit test Pig scripts.

The documentation at http://pig.apache.org/docs/r0.8.0/pigunit.html and the code at http://svn.apache.org/viewvc/pig/trunk/test/org/apache/pig/test/pigunit/TestPigTest.java?view=markup speak for themselves as far as usage.

PigStats

Pig can now provide much better visibility into what is going on inside a Pig job than it ever did before, thanks to extensive work by Richard Ding (see PIG-1333 and PIG-1478). This feature is a feature in three parts:

1. Script statistics.
This is the most easily visible change. At the end of running a script, Pig will output a table with some basic statistics regarding the jobs that it ran. It looks something like this:

Job Stats (time in seconds):

JobId	Maps	Reduces	Max Map Time	Min Map Time	Avg Map Time	Max Reduce Time	Min Reduce Time	Avg Reduce Time	Alias	Feature	Outputs
job_xxx	1654	218	84	6	14	107	87	99	counted_data, data, grouped_data	GROUP_BY, COMBINER
job_xxx	2	1	9	6	7	13	13	13	ordered_data	SAMPLER
job_xxx	2	1	26	18	22	31	31	31	ordered_data	ORDER_BY	hdfs://tmp/out,

This is extremely useful when debugging slow jobs, as you can immediately identify which stages of your script are slow, and correlate the slow Map-Reduce jobs with the actual Pig operators and relations in your script — something that was not trivial before (folks often resorted to setting parallelism to slightly different numbers for every join and group just to figure out which job was doing what. No more of this!)

2. Data in Job XML

Pig now inserts several interesting properties into the Hadoop jobs that it generates, including the relations being generated, Pig features being used, and ids of parent Hadoop jobs. This is quite helpful when monitoring a cluster, and is also handy when examining job history using the HadoopJobHistoryLoader , now part of piggybank (use Pig to mine your job history!).

3. PigRunner API

The same information that is printed out when Pig runs the script from a command line is available if one uses the Java API to start Pig jobs. If you start a script using the PigRunner.run(String args[], ProgressNotificationListener listener), you will get as a result a PigStats object that gives you access to the job hierarchy, the Hadoop counters from each job, and so on. You can implement the optional ProgressNotificationListener if you want to watch the job as it progresses; the listener will be notified as different component jobs start and finish.

Documentation of the API, new properties in the Job XML, and more, is available at http://pig.apache.org/docs/r0.8.0/piglatin_ref1.html#Pig+Statistics

Scalar values

It’s very common to need to use some calculated statistic in a calculation to inform other calculations. For example, consider a data set that consists of people and their eye color; we want to calculate the fraction of the total population that has a given eye color. The script looks something like this:

people = LOAD '/data/people' using PigStorage()
  AS (person_id:long, eye_color:chararray);
num_people = FOREACH (group people all)
  GENERATE COUNT(people) AS total;
eye_color_fractions = FOREACH ( GROUP people BY eye_color )
  GENERATE
    group as eye_color,
    COUNT(people) / num_people.total AS fraction;

Pretty straightforward, except it does not work. What’s happening in the above code is that we are referencing the relation num_people from inside another relation, eye_color_fractions and this doesn’t really make sense if Pig does not know that num_people only has one row.

In the past you had to do something hacky like joining the two relations on a constant to replicate the total into each row, and then generate the division. Needless to say, this was not entirely satisfactory. In PIG-1434 Aniket Mokashi tackled this, implementing an elegant solution that hides all of these details from the user — you can now simply cast a single-row relation as a scalar, and use it as desired. The above script becomes:

people = LOAD '/data/people' using PigStorage()
  AS (person_id:long, eye_color:chararray);
num_people = FOREACH (group people all)
  GENERATE COUNT(people) AS total;
eye_color_fractions = FOREACH ( GROUP people BY eye_color )
  GENERATE
    group as eye_color,
    COUNT(people) / (long) num_people.total AS fraction;

This makes the casting explicit, but Pig is now smart enough to do this implicitly as well. A runtime exception is generated if the relation being used as a scalar contains more than one tuple.

More documentation of this feature is available at http://pig.apache.org/docs/r0.8.0/piglatin_ref2.html#Casting+Relations+to+Scalars

Monitored UDFs

A new annotation has been added, @MonitoredUDF, which makes Pig spawn a watcher thread that kills an execution that is taking too long, and return a default value instead. This comes in handy when dealing with certain operations like complex regular expressions. More documentation is available at http://pig.apache.org/docs/r0.8.0/udf.html#Monitoring+long-running+UDFs

Automatic merge of small files

This is a simple one, but useful — when running Pig over many small files, instead of creating a map task per file (paying the overhead of scheduling and running a task for a computation that might only take a few seconds), we can merge the inputs and create a few map tasks that are a bit more hefty.

Two properties control this behavior: pig.maxCombinedSplitSize controls the maximum size of the resulting split, and pig.splitCombination controls whether or not the feature is activated in the first place (it is on by default).

This work is documented in the ticket PIG-1518; there are additional details in the release notes attached to the ticket.

Generic UDFs

I wrote about this one before — a small feature that allows you to invoke static Java methods as Pig UDFs without needing to wrap them in custom code.

The official documentation is available at http://pig.apache.org/docs/r0.8.0/piglatin_ref1.html#Dynamic+Invokers

Safeguards against missing PARALLEL keyword

One of the more common mistakes people make when writing Pig scripts is forgetting to specify parallelism for operators that need it. The default behavior used to be that this means parallelism of 1, which can lead to extremely inefficient jobs. A patch by Jeff Zhang in PIG-1249 changes this behavior to instead use a simple heuristic: if parallelism is not specified, derive the number of reducers by taking MIN(max_reducers, total_input_size / bytes_per_reducer). Max number of reducers is controlled by the property pig.exec.reducers.max (default 999) and bytes per reducer are controlled by pig.exec.reducers.bytes.per.reducer (default 1GB).

This is a safeguard, not a panacea; it only works with file-based input, estimates number of reducers based on input size, not the size of the intermediate data — so if you have a highly selective filter, or you are grouping a large dataset by a low-cardinality field, it will produce bad number — but it’s a nice safeguard against dramatic misconfigurations.

When porting to Apache Pig 0.8, remember to audit your scripts for parallelized operators that do not specify the PARALLEL keyword — if the intent is to use a single reducer, make this intent explicit by specifying PARALLEL 1.

HBaseStorage

HBaseStorage has been shored up in Pig 0.8. It can now read data stored in as bytes instead of requiring all numbers to be converted to Strings; it accepts a number of options — limit the number of rows returned, push down filters on HBase keys, etc. In addition, it can now be used to write to HBase in addition to reading from it. Details about the options, etc, can be found in the Release Notes section of PIG-1205.

Note that at the moment this only works with the HBase 0.20.{4,5,6} releases, and does not work with 0.89+. There is a patch in PIG-1680 that you can apply if you need 0.89 and 0.90 compatibility; it is not applied to the main codebase yet, as it is not backwards compatible.

We are very interested in help making this Storage engine more featureful, please feel free to jump in and contribute!

Support for custom Map-Reduce jobs in the flow

Although we try to make these a rarity, sometimes cases come up in which a custom Map-Reduce job fits the bill better than Pig. Weaving a Map-Reduce job into the middle of a Pig workflow was awkward before — you had to use something like Oozie or Azkaban, or write your own workflow application. Pig 0.8 introduces a simple “MAPREDUCE” operator which allows you to invoke an opaque MR job in the middle of the flow, and continue with Pig:

text = load 'WordcountInput.txt';
wordcount = MAPREDUCE wordcount.jar
  STORE text INTO 'inputDir'
  LOAD 'outputDir' AS (word:chararray, count: int)
  `org.myorg.WordCount inputDir outputDir`;

Details are available on the wiki page: http://wiki.apache.org/pig/NativeMapReduce

The ticket for this one has been open for a while, since Pig 0.2 days, and it’s nice to see it finally implemented. Thumbs up to Aniket Mokashi for this one.

Custom Partitioners

This feature, also implemented by the amazingly productive Aniket Mokashi, is also a bit of a power-user thing (and also an ancient ticket, PIG-282). It allows the Pig script author to control the function used to distribute map output among reducers. By default, Pig uses a random hash partitioner, but sometimes a custom algorithm is required when the script author knows something particularly unique about the reduce key distribution. When that is the case, a user can now specify the Hadoop Partitioner to swap in instead of the default:

B = group A by $0 PARTITION BY org.apache.pig.test.utils.SimpleCustomPartitioner parallel 2;

More specific documentation can be found in the Release Notes section of PIG-282

Hadoop World: NYC – Training

Jon Zuanich — Thu, 19 Aug 2010 15:00:23 +0000

Our vision for Hadoop World is a conference where both newcomers and experienced Hadoop users can learn and be part of the growing Hadoop community.

We are also offering training sessions for newcomers and experienced Hadoop users alike. Whether you are looking for an Introduction to Hadoop, Hadoop Certification, or you want to learn more about related Hadoop projects we have the training you are looking for.

Included with our top-notch Hadoop training you will have full access to Hadoop World free of charge.

Available Training Sessions include:

Oct 11:

Introduction to Hadoop: http://www.eventbrite.com/event/762326138

This one-day course provides a solid foundation for those seeking to understand large scale data processing with MapReduce and Hadoop. This session is designed for developers, analysts or system administrators that are new to Hadoop. This course provides the pre-requisite knowledge for the later classes: Developer Training, Administrator Training or Analyzing Data with Hive and Pig.

Hadoop Essentials For Managers: http://www.eventbrite.com/event/762237874

This one-day course will give decision-makers the information they need to know about Apache Hadoop, answering questions such as:

When is Hadoop appropriate?
What are people using Hadoop for?
How does Hadoop fit into our existing environment?
What do I need to know about choosing Hadoop?

Cloudera HUE SDK Training: http://www.eventbrite.com/event/764021208

Cloudera Hue provides developers with back end APIs to simplify interacting with Hadoop and front end APIs to deliver rich, web based, graphical user experiences. For this training, developers should have experience building web apps using modern MVC frameworks and Ajax. Experience with Python and Django is a strong plus. In this session we spend half the day covering the following topics, and the other half of the day interactively building applications with the Cloudera Hue team.

Oct 13 & 14:

Developer Training & Certification: http://www.eventbrite.com/event/762320120

In this two-day hands-on session, developers learn the MapReduce framework and how to write programs against its API. In addition to learning how to write individual MapReduce jobs, we discuss design techniques for larger workflows. This course also covers advanced skills for debugging MapReduce programs and optimizing their performance. At the end of the course, attendees have the option to take a certification exam documenting their understanding of the concepts taught during the training session.

Administrator Training & Certification: http://www.eventbrite.com/event/762677188

This two-day hands-on session covers the system administration aspects of Hadoop from installation and configuration to load balancing and tuning including diagnosing and solving problems in your deployment. At the end of the course, attendees have the option of taking a certification exam documenting their understanding of the concepts taught at the training session.

Analyzing Data with Hive and Pig: http://www.eventbrite.com/event/762318114

Cloudera’s two-day hands-on course on Hive and Pig is designed for people who have a basic understanding of how Hadoop works and want to utilize these languages for analysis of their data. Hive makes Hadoop accessible to users who already know SQL; Pig is similar to popular scripting languages. This course teachs you how to process data by using filters, joins, user-defined functions and more.

Oct 15:

HBase Training: http://www.eventbrite.com/event/762317111

This one-day hands-on course gives you the necessary knowledge for using HBase as a distributed data store to achieve low-latency queries and highly scalable throughput. This class covers the HBase architecture, data model, and Java API as well as advanced topics and best practices. This course is for developers who already have a basic understanding of Hadoop (Java experience is recommended).

Migrating to CDH

Eric Sammer — Tue, 03 Aug 2010 01:32:37 +0000

With the recent release of CDH3b2, many users are more interested than ever to try out Cloudera’s Distribution for Hadoop (CDH). One of the questions we often hear is, “what does it take to migrate?”.

Why Migrate?

If you’re not familiar with CDH3b2, here’s what you need to know.

All versions of CDH provide:

RPM and Debian packages for simple installation and management.
Clean integration with the host operating system. Logs are in /var/log, common binaries in /usr/bin, and configuration in /etc.
A Cloudera support-ready distribution. As Hadoop becomes a mission critical component of your production infrastructure, you’ll want the option of engaging Cloudera for support or consulting services. Running CDH makes this process simple.

CDH3b2 additionally is:

A complete platform with smooth integration of popular projects such as Hive, HBase, Pig, Zookeeper, Flume, Sqoop, Oozie, and HUE. HDFS and Hadoop Map Reduce are only two parts of a larger system. CDH3b2 brings together tools frameworks to get data in and out of HDFS, coordinate complex processing pipelines, as well as process and analyze your data. Learn more about this.
Based on Apache Hadoop 0.20.2 with 320 patches worth of feature back ports, stability enhancements, and bug fixes.

Overview

The migration process does require a moderate understanding of Linux system administration. You should make a plan before you start. You will be restarting some critical services such as the name node and job tracker, so some downtime is necessary. Given the value of the data on your cluster, you’ll also want to be careful to take recent back ups of any mission-critical data sets as well as the name node meta-data.

Backing up your data is most important if you’re upgrading from a version of Hadoop based on an Apache Software Foundation release earlier than 0.20. There were changes in the open source HDFS implementation prior to 0.20 that force this upgrade. See the section below on compatibility for more details.

The process I’ll outline here is as follows:

CDH version selection
Options for installation
Installation process
Migration of configuration data
Testing your cluster

Selecting a Branch

One of the first questions you should ask yourself is what level of stability versus new features you require from Hadoop. If you’re managing a production Hadoop cluster with jobs with SLAs, you need a rock solid, production-proven Hadoop distribution. This is Cloudera’s stable or production branch. At the time of this writing, this is CDH2 based on Hadoop 0.20.1+169.89. In certain cases, features may be of greater priority, in which case, CDH3 0.20.2+320 is appropriate.

It’s important to note that both CDH2 and CDH3 pass all functional and unit tests at Cloudera. The real difference between them is that CDH2 has been in the field longer. We generally promote a release to stable when we’ve seen it running production workloads for a substantial period of time, and when the rate of issues opened against the distro in our support group tails off. We have customers running in production today on both CDH2 an CDH3.

On Compatibility

Before we dive into the installation process I’ll highlight some points on compatibility. When upgrading to CDH from an older version or another distribution of Hadoop, it’s possible that HDFS data needs to be taken through an upgrade process. This is relatively simple, but as with any upgrade of critical data, it is absolutely necessary to back up your data.

Currently, it is not necessary to perform an HDFS upgrade if you’re upgrading to CDH3 from CDH2 or Apache Hadoop versions 0.20.0 or later. In fact, any distribution of Hadoop based on Apache 0.20.0 is likely to be a clean transition without an update to HDFS required, but you should always check with the distributor.

During RPC operations, all Hadoop daemons will check to ensure they are speaking to the same exact version as themselves. This means that you cannot, at present, perform a rolling upgrade of CDH. There has been some discussion about relaxing this requirement so compatible versions of Hadoop can communicate, but this has not yet been implemented.

Installation Options

CDH is available in three forms: RPMs, debs, and tarball distributions. The preferred method of installation is usually the RPM or deb packages as they automate a lot of the work required to get CDH up and running quickly. Tarballs of CDH are useful for users on systems that do not use yum/rpm or apt/dpkg, or where you do not have root access to the host operating system.

Installing CDH

When installing CDH from from RPMs or Debian packages you will definitely want to take advantage of Cloudera’s yum or apt repository support. If you’re on a system that is not rpm or deb format packages, you can still use Cloudera’s binary tarball packages.

You should follow the normal process for installing CDH on your systems. The CDH packages should be installed on all nodes in the cluster. The rpm and deb packages of CDH will automatically create a hadoop user and group as well as SYSV init scripts as part of the install process. The CDH tarballs do not contain the init scripts and obviously do not create the hadoop user and group.

Detailed installation instructions for all formats of CDH are available.

After the packages are installed, you’ll want to make sure you set the proper daemons to start on the proper machines upon boot. There is a separate init script for each Hadoop daemon so only what is necessary is started.

Redhat example:
% chkconfig --level 3 hadoop-0.20-namenode on

Debian example:
% update-rc.d hadoop-0.20-namenode start 80 3 .

Make sure you specify the correct run level. While run level 3 is common for multiuser Linux servers, this may not be the case in your installation. You can use the runlevel command to find the currently active run level.

For now, do not start any of the Hadoop daemons.

Migrating Your Configuration

If you’re coming from older version of CDH, your configuration should already be setup with alternatives. If not, now is a good time to bring your configuration layout in line with CDH by moving your conf directory to /etc/hadoop-0.20/conf.mycluster. You should also configure alternatives to know about your new configuration. The CDH documentation covers this in detail. For now, register your new configuration with alternatives and set it to be the preferred configuration.

% alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.mycluster 100 % alternatives --set hadoop-0.20-conf /etc/hadoop-0.20/conf.mycluster

Users who are on systems that don’t have alternatives or who are installing CDH from tarballs should simply update the configuration files in $HADOOP_HOME/conf. Normally, $HADOOP_HOME is /usr/local/hadoop-$VERSION or /opt/hadoop-$VERSION but you can put it wherever it makes sense. This includes running CDH from your home directory if you don’t have root access.

Testing CDH

Now that CDH is installed and you’ve migrated your cluster configuration it’s time to fire up a few nodes and make sure everything is working as expected. Rather than bring up all the daemons at once, let’s focus on the name node first.

Start by logging on to the name node machine. You may want to manually rotate the log file just to minimize the noise during testing. You can do this by simply moving today’s log file to a different name.

% mv /var/log/hadoop/hadoop-hadoop-namenode-nn.mycompany.com.log \ /var/log/hadoop/hadoop-hadoop-namenode-nn.mycompany.com.log.old

Next, start the CDH name node daemon using the provided init script. If an HDFS upgrade is required, you can use the upgrade argument in place of start below. This will be your last chance to grab a backup of the name node’s metadata prior to starting the daemon.

% /etc/init.d/hadoop-0.20-namenode start

Note that the CDH init scripts require you to be root whereas the Apache Hadoop start-all.sh / stop-all.sh scripts should not be run as root.

It’s a good idea to check the contents of the name node log file now to ensure it has come up cleanly. You should see a warning about the name node being in safe mode due to missing blocks. This is OK because we haven’t brought up any data nodes yet. If something doesn’t look right, jump ahead to the getting help section before proceeding.

Before you start any of your data nodes, you’ll want to place the name node in safe mode manually. This will prevent the name node from “panicking” and trying to repair missing block replicas as data nodes begin to register themselves. You’ll need to run this command as the hadoop user.

% hadoop dfsadmin -safemode enter

Next start one of the data nodes and watch its logs as you did for the name node.

% /etc/init.d/hadoop-0.20-datanode start

If everything is setup correctly, you should see the data node start up, register with the name node, and start its periodic block scanner thread. You should also check the name node logs to confirm you see the data node registration message there as well. Once you’ve confirmed that things look good, you should move on to starting additional data nodes checking them in batches as you go.

After all data nodes are up and running, you can use the Hadoop fsck tool to confirm that the file system is healthy.

% hadoop fsck /

Your cluster should still be in safe mode. If the file system is healthy, you can go ahead and take it out of safe mode.

% hadoop dfsadmin -safemode leave

Follow this with a quick test of HDFS by copying a file into the file system.

% date > now.txt % hadoop fs -put now.txt /now.txt % hadoop fs -cat /now.txt % hadoop fs -rm /now.txt % rm now.txt

Congratulations! You now have HDFS running on CDH.

If you had to upgrade the HDFS data – that is, you started the init script with the upgrade option – you should do some more extensive testing of your data. Once you’ve confirmed everything is working as expected, finalize the HDFS upgrade.

% hadoop namenode -finalize

Starting and testing the map reduce daemons follows a similar procedure but is a bit simpler. Start the job tracker daemon on the proper machine and monitor the logs as you did with the name node. Once you’ve confirmed the job tracker is running, proceed with starting the task tracker daemons in groups checking the job tracker UI as you go. You should see the map and reduce task capacity increasing with each node you start. Don’t panic if the job tracker doesn’t see the nodes immediately; it can take a few seconds.

Don’t forget to start the secondary name node daemon as well. It’s usually a good idea to wait an hour or so and check the modification time on the files in the configured fs.checkpoint.dir. You should see that the files have been updated within the last hour. You can also check the secondary name node logs; you’ll see an indication things are working there as well in the form of some log messages about performing the checkpoint.

Documentation and References

In addition to the community articles and blog posts on Hadoop, Cloudera provides CDH-specific documentation at docs.cloudera.com. Here you can find information on CDH including all of its components like Hadoop, Hive, Flume, Sqoop, HUE, and others.

How to Get Help

There are a number of ways to get help if you run into trouble during your migration or if you just have questions.

Cloudera Documentation
Cloudera mailing lists
Cloudera videos
IRC users can join #cloudera on freenode

Announcing Two New Training Classes from Cloudera: Introduction to HBase and Analyzing Data with Hive and Pig

John Kreisa — Wed, 28 Jul 2010 13:52:22 +0000

Cloudera is pleased to announce two new training courses: a one-day Introduction to HBase and a two-day session on Analyzing Data with Hive and Pig. These join a recently-expanded two-day Hadoop for Administrators course and our popular three-day Hadoop for Developers offering, any of which can be combined to provide extensive, customized training for your organization. Please contact sales@cloudera.com for more information regarding on-site training, or visit www.cloudera.com/hadoop-training to view our public course schedule.

Cloudera’s HBase course discusses use-cases for HBase, and covers the HBase architecture, schema modeling, access patterns, and performance considerations. During hands-on exercises, students write code to access HBase from Java applications, and use the HBase shell to manipulate data. Introduction to HBase also covers deployment and advanced features.

Our Hive and Pig course is designed for developers who are skilled with SQL or scripting languages, but who are not Java experts. Hive and Pig are two approaches which allow non-Java programmers to access and manipulate massive amounts of data while abstracting away the complexities of MapReduce. Hive offers an SQL-like interface, while Pig’s scripting language, named PigLatin, is very easy for developers learn. This course covers both technologies, and includes multiple hands-on exercises to reinforce key concepts.

Cloudera’s Hadoop for System Administrators course has recently been expanded from one day to two, and covers the important issues for System Administrators charged with looking after Hadoop clusters. Topics include planning and deploying the cluster, managing MapReduce jobs, scheduling jobs using the Fair Scheduler, cluster monitoring and troubleshooting, populating HDFS from existing relational database management systems with Sqoop, and using Flume to import logs and other files into HDFS.

Our most popular course, Hadoop for Developers, is a three-day offering which covers everything from an introduction to HDFS and MapReduce right through to advanced MapReduce APIs and algorithms. Students learn to build MapReduce jobs through a combination of instructor-led training and hands-on exercises; the course includes an exam offering students the chance to earn Cloudera Certified Hadoop Developer credentials.

A complete list of events including upcoming training is available here: http://www.cloudera.com/company/events/

What’s New in CDH3b2: Oozie

Arvind Prabhakar — Thu, 15 Jul 2010 15:00:23 +0000

Hadoop has emerged as an indispensable component of any data-intensive enterprise infrastructure. In many ways, working with large datasets on a distributed computing platform (powered by commodity hardware or cloud infrastructure) has never been easier. But because customers are running clusters consisting of hundreds or thousands of nodes, and are processing massive quantities of data from production systems every hour, the logistics of efficient platform utilization can quickly become overwhelming.

To deal with this challenge, the Yahoo! engineering team created Oozie – the Hadoop workflow engine. We are pleased to provide Oozie with Cloudera’s distribution for Hadoop starting with the beta-2 release.

Why create a new workflow system?

You might wonder why a new workflow system is necessary for Hadoop, given that there are quite a few existing commercial and open-source systems available. While it is possible to use existing general-purpose workflow systems with Hadoop, it is anything but simple. Intricacies such as monitoring long running jobs and interfacing with the distributed file system require extensive work to port general workflow systems to the Hadoop environment. Oozie, on the other hand, is designed specifically for the Hadoop platform and uses it as its execution environment. It has built-in support for Hadoop tasks and integrates with this environment cleanly. Oozie itself is fairly light-weight, requires minimal configuration, and scales linearly – thus offering a sustainable approach to building workflows in the Hadoop environment.

Still not convinced about Oozie? Consider these numbers for a moment: According to the Oozie presentation during Hadoop Summit in June – there are over 4800+ workflow applications deployed within Yahoo! at the moment, with largest workflow containing 2000 actions. There were roughly 55,000 workflow jobs that Yahoo! infrastructure team executed in the month of May 2010 alone, with workflows that could run up to many hours.

A Simple Use-Case

Consider the example of web log analysis. For a typical operation that deals with a few gigabytes of log data every day, the steps involved in analyzing it can be many. First, the files have to be moved into a certain location. Next, the files are used to create new tables or partitions in Hive which are then queried to see if certain criteria have been met. For instance, if the number of accesses for a particular resource exceeds a certain threshold, some notifications must be generated. Regardless of the outcome of this analysis, certain other queries need be run in order to populate other tables that record rolled-up information.

While these steps are not difficult to execute, they are repetitive and time consuming. Ideally, such steps should be automated in a manner that notifications are raised to operators when something interesting is discovered by the system or if there is a failure of some sort. That is exactly what Oozie does.

Using Oozie, all the steps outlined in this example can be modeled as a workflow which can be executed with a single command. Once the workflow takes off, you can sit back and relax while Oozie runs through each step of the flow.

Oozie Highlights

Oozie workflow bundles the workflow definition, any libraries necessary for the execution of workflow actions, and properties that are necessary to resolve parameterized values in the workflow. Together, this bundle is referred to as an Oozie application and informally – a workflow. These are deployed to the Oozie server using a command line utility. Once deployed, the workflows can be started and manipulated as necessary using the same utility. The web console for Oozie server can be used to monitor the progress of various workflow jobs being managed by the server.

Scalability

Oozie is a server-based web application that uses a transactional store to manage workflow metadata and execution states. It relies on HTTP-based notifications and polling mechanism to monitor the progress of workflows and to manage its runtime state. The Oozie server itself does not do any particular work other than this state management. All of the work is delegated to worker nodes within the cluster on which the workflow executes. This allows Oozie to scale horizontally by adding more Oozie servers pointing to the same workflow metadata store.

Resilience

When a workflow execution encounters a transient error condition, Oozie automatically attempts to execute the action again. In some situations, when the error requires user intervention, Oozie can suspend the workflow indefinitely allowing the administrator to step in, take corrective action, and resume the workflow. For long running workflows that fail, Oozie provides a mechanism by which the workflow can be restarted from the point of failure to avoid redoing the steps that may have already completed earlier.

Simple and Intuitive

Workflows in Oozie are expressed in a simple XML representation that is inspired by process definition language – JPDL. However, compared to the overal JPDL schema, the Oozie schema is extremely simplified and intuitive. The key concepts in a Oozie workflow is that of action and control flow nodes. Action nodes do the workflow tasks – such as moving files, running Map/Reduce jobs, running Hive Queries etc. Control-flow nodes govern the progress of the workflow from action to action, enabling things like error handling, conditional execution and branching logic.

Together, the action and control-flow nodes are arranged in a directed acyclic graph (DAG), which represents the overall workflow. This DAG is executed by the Oozie server in a controlled-dependency manner – implying that a node shall be executed if and only if all of the nodes that it depends upon have been executed successfully. This is very similar to how one would manually implement the workflow – by initiating actions and starting follow-up actions when the previous ones are complete with expected outcome.

The bottom line is that if you know the steps you need to take for managing data in your Hadoop environment, you can easily express them as a workflow and hand it off to Oozie for execution.

Rich Set of Features

The core feature set of Oozie is designed to take care of the most commonly-exercised functionality for the Hadoop platform. The key objective behind these features is to ensure that anything done manually can be implemented as a workflow task to the last detail. The following list, while far from being exhaustive, lists out some of the many features that Oozie has.

1. Parameterization: With parameterization support, workflows can be written once and executed many times with different parameter bindings. This allows reuse of workflows in a manner that promotes ease of maintenance and management.
2. Fine-Grain and Coarse-Grain Notification Support: Workflows in Oozie can be configured to notify external systems at varying degree of granularity. Notifications can be raised when a workflow changes its overall state, or when an individual action within it changes state. These notifications are implemented as HTTP GET requests which can pass extra information to the receiver such as job identifier. Using this mechanism, external systems can be integrated at various stages of the workflow as necessary.
3. User Propagation: The user and group information associated with the workflow job is propagated by Oozie to the underlying action execution and cannot be overwritten. This allows Oozie to work together with Hadoop security to ensure that actions are authorized to access and manipulate data where applicable.
4. Java Client API: A programmatic client API is provided by Oozie that can be called from external systems to better integrate with the Oozie system. This API provides equivalent functionality as provided by the command line client utility.
5. Web Services API: Oozie also provides a rich REST/JSON API for web-services integration. Clients that prefer to access Oozie via this API can directly access and manipulate workflows running on Oozie server using this interface.
6. Built in actions: The default actions provided by Oozie cover a vast majority of the use-cases for workflows including:
- Map/Reduce action: Allows you to model Map/Reduce jobs.
- Streaming Map/Reduce action: Allows you to specify executable mapper and reducers that can be plugged in using the streaming support.
- Pig action: Allows you to run custom Pig scripts and tasks.
- FS action: Allows you to manipulate the Hadoop file system as necessary.
- SSH action: Allows you to securely execute commands over SSH connections.
- Sub-Workflow action: Allows a workflow to be executed within another workflow.
- Java action: Allows you to plug in any Java program that has a main method for direct execution.
- Hive action: Allows you to run Hive commands from within the workflow. This action is contributed by Cloudera.
- Sqoop action: Allows you to run Sqoop commands from within the workflow. This action is contributed by Cloudera.
7. Built in control-flow nodes: The control-flow nodes provided by Oozie allow you to create sophisticated workflow graphs. Together with built-in support for JSP Expression Language functions, the control-flow nodes can be used for creating conditional executions where necessary. These include the following:
- Start node: Indicates the starting point of a workflow.
- End node: Indicates a completion point of a workflow.
- Kill node: Allows a workflow action outcome to cause workflow termination where necessary.
- Decision node: Allows a branching construct similar to traditional switch-case statement using JSP Expression Language expressions as predicates.
- Fork and Join node: Fork node allows the splitting of a workflow execution path into multiple concurrent paths of execution. Join nodes allow the branches created by fork node to merge back into a single execution path. Fork and join nodes must be used in pairs and allow efficient parallel execution of tasks that do not have a direct-controlled dependency order.
8. Custom Extensions: Oozie provides an extension mechanism that allows the implementation of custom actions. This mechanism should be used when the extension cannot be modeled as a regular Java action.

Get up and running!

Now that you have a good feel for what Oozie is and how simple it is, it is time for you to get your instance of Oozie installed and configured. Follow our Quick Start Guide to get up and running with Oozie in a matter of minutes. You can reach out to the Oozie user group by sending a mail to oozie-users@yahoogroups.com.

What’s New in CDH3b2: Pig

Carl Steinbach — Wed, 14 Jul 2010 19:41:54 +0000

CDH3 beta 2 includes Pig 0.7.0, the latest and greatest version of the popular dataflow programming environment for Hadoop. In this post I’ll review some of the bigger changes that went into Pig 0.7.0, describe the motivations behind these changes, and explain how they affect users. Readers in search of a canonical list of changes in this new version of Pig should consult the Pig 0.7.0 Release Notes as well as the list of backward incompatible changes.

Load-Store Redesign

The biggest change to appear in Pig 0.7.0 is the complete redesign of the LoadFunc and StoreFunc interfaces. The Load-Store interfaces were first introduced in version 0.1.0 and have remained largely unchanged up to this point. Pig uses a concrete instance of the LoadFunc interface to read Pig records from the underlying storage layer, and similarly uses an instance of the StoreFunc interface when it needs to write a record. Pig provides different LoadFunc and StoreFunc implementations in order to support different storage formats, and since this is a public interface users may provide their own implementations as well.

The primary motivation for redesigning these interfaces is to bring them into closer alignment with Hadoop’s InputFormat and OutputFormat interfaces, with the goal of making it much easier to write new LoadFunc and StoreFunc implementations based on existing Hadoop InputFormat and OutputFormat classes. At the same time the new interfaces were also made a lot more powerful by providing direct access to configurations as well as the ability to selectively read individual columns.

In the short span of time since these new interfaces appeared the Pig community has responded by writing a variety of custom Loaders including ones for Cassandra, Voldemort, and Hive’s RCFile columnar storage format. It is important to note that these new plugins were written without any direct involvement from the Pig core team, which is a significant validation of the work that went into the redesign effort. A list of third-party Pig Loaders is maintained on the Pig Intoperability page. Users who are interested in writing their own LoadFuncs or StoreFuncs should first read the updated Load-Store HowTo.

If you are upgrading from an earlier version of Pig you need to be aware that the new Load/Store interfaces are not backward compatible with the old interfaces. Users who have written custom LoadFuncs or StoreFuncs that work with an earlier version will need to upgrade these functions to use the new interfaces. For more details about this process please consult the Load-Store Migration Guide on the Pig wiki.

Use the Distributed Cache to Improve Performance

Pig 0.7.0 includes a set of important performance enhancements that aim to make queries run faster by leveraging Hadoop’s Distributed Cache. The key observation that motivated these changes is that Pig query plans often involve directing a large number of tasks to read the same sample of data. One can observe this access pattern in the Fragment-Replicate Join, SkewedJoin, and GroupBy operators. Earlier versions of Pig read this data directly from the underlying distributed file system, an approach that is inefficient, but also has the potential to cause a cluster-wide failure if a large number of concurrent Map tasks swamp the NameNode with read requests. PIG-872 and PIG-1218 remedy this problem by loading the common data into the Distributed Cache. This allows tasks to perform a local disk read instead of having to wait while the data is retrieved from HDFS, and also allows tasks that run on the same node to share the same data.

Use Hadoop’s Local Mode for Pig Local Mode

One of things that has made Pig especially easy for new users to pick up is its support for a local mode that does not require an Hadoop installation. Unfortunately, maintaining this feature has turned into a major headache for the Pig developers as it requires a large body of custom code and execution paths that are not shared with the rest of the system. A direct consequence of this is that many of the new features that have been added to Pig do not work in local mode, and this has caused a lot of confusion within the Pig user community. Based on these factors the Pig developers decided that it made sense to replace Pig’s custom local mode implementation with one that depends on Hadoop’s local mode. This change benefits Pig users since they can now test a script in local mode and be confident that it will run correctly in distributed mode, or vice-versa. However, users should be aware that there is one unfortunate side-effect of this change: Pig now runs roughly an order of magnitude slower in local mode.

Making Pig 0.7.0 Even Better

Pig 0.7.0 was released in mid-May, and since that time several important patches have appeared for bugs that were found in the original release. These patches include a fix that allows UDFs to access counters, as well as another fix that adds a counter to track the number of output rows in each output file. I think you’ll be glad to hear that we have included these patches as well as others in the version of Pig 0.7.0 that is included in CDH3 beta 2.

For More Information

We hope you’ll give CDH and Pig a spin. The CDH Quick Start Guide is the best place to begin, followed with the Pig installation instructions in the Pig Installation Guide.

CDH3 Beta 1 Now Available

Eli Collins — Wed, 24 Mar 2010 15:00:25 +0000

It’s official – Cloudera’s Distribution for Hadoop Version 2, which we often shorthand as CDH2, has been released. CDH2 is the product we recommend to our current production customers. It’s a stable version that has undergone a long cycle of time in the field with a variety of customers, in addition to Cloudera’s internal QA process.

And with the CDH2 release, the Cloudera engineering team is excited to start the feedback and development process for the next version of Cloudera’s Distribution for Hadoop – Version 3. CDH3 includes a Pig package with additional bug fixes and performance improvements, and the Hive package is now based on the latest Apache release. One of the most notable aspects of CDH3 beta 1 is what has not changed: CDH3 remains based on the Apache 0.20 release. However, we have already bundled many new improvements and bug fixes in CDH3. The release notes cover these changes in detail.

What the release notes don’t share, though, is what we plan on putting into upcoming CDH3 releases. Cloudera is working hard with the rest of the Apache community to deliver additional features in CDH3 including the following noteworthy items:

A new Pig package based on the latest Apache release.
HBase and ZooKeeper, previously only supported as part of our contrib repository, will become first class packages in CDH3.
The security work Yahoo! is contributing, which should significantly impact Hadoop adoption.

As you’d expect, we will continue to test and integrate many other improvements, bug fixes and features throughout the release. Please check out the beta and tell us what you think!

CDH2 is released

Chad Metcalf — Wed, 24 Mar 2010 14:59:22 +0000

We’re proud to announce that Cloudera’s Distribution for Hadoop Version 2 (CDH2) is officially released.

We’ve come a long way to get to a production quality release. At the beginning of September we announced the first beta of CDH2. After 6 months of additional testing we announced a release candidate. The release candidate spent over a month hardening in Cloudera’s internal QA process and on a wide variety of customer clusters. CDH2 is now stable and ready for use – we are pleased to recommend it to all our production users.

CDH2 is based on Apache Hadoop 0.20 – a release that has been available for almost a year. During this time, the Apache Hadoop community has produced hundreds of bug fixes, improvements and features. Cloudera is proud to have contributed many of these and incorporated them into CDH2. For more information, please review the following resources:

The release notes for CDH2. All bug fixes and improvements are covered in detail.
For new features you’ll want to checkout CDH3, which is now in beta.
For how to get started, please have a look at CDH documentation, which includes a helpful bit on determining which version is right for you.

Hadoop is a community effort. We’d like to thank everyone who contributes to Hadoop, especially the substantial contribution made by the big team at Yahoo! and all the other users who have contributed to this release. We appreciate the feedback on Get Satisfaction, twitter and IRC (#cloudera on freenode.net). Keep it coming, and thanks for using Cloudera’s Distribution for Hadoop!