The Apache MRUnit team has graduated from the Apache Incubator to an Apache TLP (Top Level Project)! MRUnit is a Java library that helps developers unit test Apache Hadoop MapReduce jobs. Unit testing is a technique for improving project quality and reducing overall costs by writing a small amount of code that can automatically verify the software you write performs as intended. This is considered a best practice in software development since it helps identify defects early, before they're deployed to a production system.

In its monthly meeting in May of 2012, the board of Apache Software Foundation (ASF) resolved to grant a Top-Level Project status to Apache MRUnit, thus graduating it from the Incubator. This is a significant milestone in the life of MRUnit, which has come a long way since its inception as a Hadoop Contrib project in HADOOP-5518 contributed by Aaron Kimball.

• May 2012 MRUnit graduates from the Incubator to become a TLP
• May 2012 Version 0.9.0-incubating released.
• April 2012 Dave Beech added as a new committer.
• April 2012 Jarek Jarcec Cecho added as a new committer.
• April 2012 New website created using the CMS.
• March 2012 Version 0.8.1-incubating released.
• March 2012 Jim Donofrio added as a new committer.
• Feburary 2012 Version 0.8.0-incubating released.
• November 2011 Version 0.5.0-incubating released.
• October 2011 Brock Noland added as a new committer.
• March 2011 Project enters incubation.
• April 2009 Doug Cutting commits Aaron's patch to Hadoop
• March 2009 Aaron Kimball contributes MRunit to Hadoop as a contrib project

Below is the graduation resolution:

X. Establish the Apache MRUnit Project

WHEREAS, the Board of Directors deems it to be in the best
interests of the Foundation and consistent with the
Foundation's purpose to establish a Project Management
Committee charged with the creation and maintenance of
open-source software related to unit testing Apache Hadoop map
reduce jobs for distribution at no charge to the public.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management
Committee (PMC), to be known as the "Apache MRUnit Project",
be and hereby is established pursuant to Bylaws of the
Foundation; and be it further

RESOLVED, that the Apache MRUnit Project be and hereby is
responsible for the creation and maintenance of software
related to unit testing Apache Hadoop map reduce jobs;
and be it further

RESOLVED, that the office of "Vice President, Apache MRUnit" be
and hereby is created, the person holding such office to
serve at the direction of the Board of Directors as the chair
of the Apache MRUnit Project, and to have primary responsibility
for management of the projects within the scope of
responsibility of the Apache MRUnit Project; and be it further

RESOLVED, that the persons listed immediately below be and
hereby are appointed to serve as the initial members of the
Apache MRUnit Project:

* Brock Noland - brock@apache.org
* Patrick Hunt - phunt@apache.org
* Nigel Daley - nigel@apache.org
* Eric Sammer - esammer@apache.org
* Aaron Kimball - kimballa@apache.org
* Konstantin Boudnik - cos@apache.org
* Garrett Wu - gwu@apache.org
* Jim Donofrio - jdonofrio@apache.org
* Jarek Jarcec Cecho - jarcec@apache.org
* Dave Beech - dbeech@apache.org

NOW, THEREFORE, BE IT FURTHER RESOLVED, that Brock Noland
be appointed to the office of Vice President, Apache MRUnit, to
serve in accordance with and subject to the direction of the
Board of Directors and the Bylaws of the Foundation until
death, resignation, retirement, removal or disqualification,
or until a successor is appointed; and be it further

RESOLVED, that the initial Apache MRUnit PMC be and hereby is
tasked with the creation of a set of bylaws intended to
encourage open development and increased participation in the
Apache MRUnit Project; and be it further

RESOLVED, that the Apache MRUnit Project be and hereby
is tasked with the migration and rationalization of the Apache
Incubator MRUnit podling; and be it further

RESOLVED, that all responsibilities pertaining to the Apache
Incubator MRUnit podling encumbered upon the Apache Incubator
Project are hereafter discharged.

Apache Hadoop 0.23 has major improvements over previous releases. Here are a few highlights on the MapReduce front; note that there are also major HDFS improvements, which are out of scope of this post.

MapReduce 2.0 (a.k.a. MRv2 or YARN):

The new architecture divides the two major functions of the JobTracker – resource management and job life-cycle management – into separate components:

• A ResourceManager (RM) that manages the global assignment of compute resources to applications.
• A per-application ApplicationMaster (AM) that manages the application’s life cycle.

In Hadoop 0.23, a MapReduce application is a single job in the sense of classic MapReduce, executed by the MapReduce ApplicationMaster.

There is also a per-machine NodeManager (NM) that manages the user processes on that machine. The RM and the NM form the computation fabric of the cluster. The design also allows plugging long-running auxiliary services to the NM; these are application-specific services, specified as part of the configuration, and loaded by the NM during startup. For MapReduce applications on YARN, shuffle is a typical auxiliary service loaded by the NMs. Note that, in Hadoop versions prior to 0.23, shuffle was part of the TaskTracker.  

The per-application ApplicationMaster is a framework specific library and is tasked with negotiating resources from the ResourceManager and working with the NodeManager(s) to execute and monitor the tasks. In the YARN design, MapReduce is just one application framework; the design permits building and deploying distributed applications using other frameworks. For example, Hadoop 0.23 ships with a Distributed Shell application that permits running a shell script on multiple nodes on the YARN cluster. At the time of writing this blog post, there is also an ongoing development effort to allow running Message Passing Interface (MPI) applications on top of YARN.

MapReduce 2.0 Design:

Figure 1 shows a pictorial representation of a YARN cluster. There is a single Resource Manager, which has two main services:

• A pluggable Scheduler, which manages and enforces the resource scheduling policy in the cluster.  Note that, at the time of writing this blog post, there are two schedulers supported in Hadoop 0.23, the default FIFO scheduler and the Capacity scheduler; the Fair Scheduler is not yet supported.
• An Applications Manager (AsM), which manages running Application Masters in the cluster, i.e., it is responsible for starting application masters and for monitoring and restarting them on different nodes in case of failures.

Fig. 1

Figure 1 also shows that there is a NM service running on each node in the cluster. The diagram also shows two AMs (AM₁ and AM₂). In a YARN cluster at any given time, there will be as many running Application Masters as there are applications (jobs). Each AM manages the application’s individual tasks (starting, monitoring and restarting in case of failures). The diagram shows AM₁ managing three tasks (containers 1.1, 1.2 and 1.3), while AM₂ manages four tasks (containers 2.1, 2.2, 2.3 and 2.4). Each task runs within a Container on each node. The AM acquires such containers from the RM’s Scheduler before contacting the corresponding NMs to start the application’s individual tasks. These Containers can be roughly compared to Map/Reduce slots in previous Hadoop versions. However the resource allocation model in Hadoop-0.23 is more optimized from a cluster utilization perspective.  

Resource Allocation Model:

In earlier Hadoop versions, each node in the cluster was statically assigned the capability of running a predefined number of Map slots and a predefined number of Reduce slots. The slots could not be shared between Maps and Reduces. This static allocation of slots wasn’t optimal since slot requirements vary during the MR job life cycle (typically, there is a demand for Map slots when the job starts, as opposed to the need for Reduce slots towards the end). Practically, in a real cluster, where jobs are randomly submitted and each has its own Map/Reduce slots requirement, having an optimal utilization of the cluster was hard, if not impossible.

The resource allocation model in Hadoop 0.23 addresses such deficiency by providing a more flexible resource modeling. Resources are requested in the form of containers, where each container has a number of non-static attributes. At the time of writing this blog, the only supported attribute was memory (RAM). However, the model is generic and there is intention to add more attributes in future releases (e.g. CPU and network bandwidth). In this new Resource Management model, only a minimum and a maximum for each attribute are defined, and AMs can request containers with attribute values as multiples of these minimums.

MapReduce 2.0 Main Components:

In this section, we’ll go through the main components of the new MapReduce architecture in detail to understand the functionality of these components and how they interact with each other.

• Client – Resource Manager

Figure 2 illustrates the initial step for running an appilcation on a YARN cluster. Typically a client communicates with the RM (specifically the Applications Manager component of the RM) to initiates this process. The first step, marked (1) in the diagram, is for the client to notify the Applications Manager of the desire of submitting an application, this is done via a “New Application Request”. The RM respose, marked (2), will typically contain a newly generated unique application ID, in addition to information about cluster resource capabilities that the client will need in requesting resources for running the application’s AM.

Using the information received from the RM, the client can construct and submit an “Application Submission Context”, marked (3), which typically contains information like scheduler queue, priority and user information, in addition to information needed by the RM to be able to launch the AM. This information is contained in a “Container Launch Context”, which contains the application’s jar, job files, security tokens and any resource requirements.

Fig. 2

Following application submission, the client can query the RM for application reports, receive such reports and, if needed, the client can also ask the RM to kill the application. These three additional steps are pictorially depicted in fig. 3.

Fig. 3

• Resource Manager – Application Master

When the RM receives the application submission context from the client, it finds an available container meeting the resource requirements for running the AM, and it contacts the NM for the container to start the AM process on this node. Figure 4 depicts the following communication steps between the AM and the RM (specifically the Scheduler component of the RM). The first step, marked (1) in the diagram, is for the AM to register itself with the RM. This step consists of a handshaking procedure and also conveys information like the RPC port that the AM will be listening on, the tracking URL for monitoring the application’s status and progress, etc.

The RM registration response, marked (2), will convey essential information for the AM master like minimum and maximum resource capabilities for this cluster. The AM will use such information in calculating and requesting any resource requests for the application’s individual tasks. The resource allocation request from the AM to the RM, marked (3), mainly contains a list of requested containers, and may also contain a list of released containers by this AM. Heartbeat and progress information are also relayed through resource allocation requests as shown by arrow (4).

When the Scheduler component of the RM receives a resource allocation request, it computes, based on the scheduling policy, a list of containers that satisfy the request and sends back an allocation response, marked (5), which contains a list of allocated resources. Using the resource list, the AM starts contacting the associated node managers (as will be soon seen), and finally, as depicted by arrow (6), when the job finishes, the AM sends a Finish Application message to the Resource Manager and exits.

Fig. 4

• Application Master – Container Manager

Figure 5 describes the communication between the AM and the Node Managers. The AM requests the hosting NM for each container to start it as depicted by arrow (1) in the diagram. While containers are running, the AM can request and receive a container status report as shown in steps (2) and (3), respectively.

Fig. 5

Based on the above discussion, a developer writing YARN applications will be mainly concerned with the following interfaces:

• ClientRMProtocol: Client RM (Fig. 3).
  This is the protocol for a client to communicate with the RM to launch a new application (i.e. an AM), check on the status of the application or kill the application.
• AMRMProtocol: AM RM (Fig. 4).
  This is the protocol used by the AM to register/unregister itself with the RM, as well as to request resources from the RM Scheduler to run its tasks.
• ContainerManager: AM NM (Fig. 5).
  This is the protocol used by the AM to communicate with the NM to start or stop containers and to get status updates on its containers.

Migrating older MapReduce applications to run on Hadoop 0.23:

All client-facing MapReduce interfaces are unchanged, which means that there is no need to make any source code changes to run on top of Hadoop 0.23.

Useful links:

• How to write a YARN Application.
• Hadoop 0.23.0 Javadocs.

Background: Adverse Drug Events

An adverse drug event (ADE) is an unwanted or unintended reaction that results from the normal use of one or more medications. The consequences of ADEs range from mild allergic reactions to death, with one study estimating that 9.7% of adverse drug events lead to permanent disability. Another study showed that each patient who experiences an ADE remains hospitalized for an additional 1-5 days and costs the hospital up to $9,000.

Some adverse drug events are caused by drug interactions, where two or more prescription or over-the-counter (OTC) drugs taken together leads to an unexpected outcome. As the population ages and more patients are treated for multiple health conditions, the risk of ADEs from drug interactions increases. In the United States, roughly 4% of adults older than 55 are at risk for a major drug interaction.

Because clinical trials study a relatively small number of patients, both regulatory agencies and pharmaceutical companies maintain databases in order to track adverse events that occur after drugs have been approved for market. In the United States, the FDA uses the Adverse Event Reporting System (AERS), where healthcare professionals and consumers may report the details of ADEs they experienced.  The FDA makes a well-formatted sample of the reports available for download from their website, to the benefit of data scientists everywhere.

Methodology

Identifying ADEs is primarily a signal detection problem: we have a collection of events, where each event has multiple attributes (in this case, the drugs the patient was taking) and multiple outcomes (the adverse reactions that the patient experienced), and we would like to understand how the attributes correlate with the outcomes. One simple technique for analyzing these relationships is a 2×2 contingency table:

Based on the values in the cells of the tables, we can compute various measures of disproportionality to find drug-reaction pairs that occur more frequently than we would expect if they were independent.

For this project, we analyzed interactions involving multiple drugs, using a generalization of the contingency table method that is described in the paper, “Empirical bayes screening for multi-item associations” by DuMouchel and Pregibon. Their model computes a Multi-Item Gamma-Poisson Shrinkage (MGPS) estimator for each combination of drugs and outcomes, and gives us a statistically sound measure of disproportionality even if we only have a handful of observations for a particular combination of drugs. The MGPS model has been used for a variety of signal detection problems across multiple industries, such as identifying fraudulent phone calls, performing market basket analyses and analyzing defects in automobiles.

Solving the Hard Problem with Apache Hadoop

At first glance, it doesn’t seem like we would need anything beyond a laptop to analyze ADEs, since the FDA only receives about one million reports a year. But when we begin to examine these reports, we discover a problem that is similar to what happens when we attempt to teach computers to play chess: a combinatorial explosion in the number of possible drug interactions we must consider. Even restricting ourselves to analyzing pairs of drugs, there are more than 3 trillion potential drug-drug-reaction triples in the AERS dataset, and tens of millions of triples that we actually see in the data. Even including the iterative Expectation Maximization algorithm that we use to fit the MGPS model, the total runtime of our analysis is dominated by the process of counting how often the various interactions occur.

The good news is that MapReduce running on a Hadoop cluster is ideal for this problem. By creating a pipeline of MapReduce jobs to clean, aggregate, and join our data, we can parallelize the counting problem across multiple machines to achieve a linear speedup in our overall runtime. The faster runtime for each individual analysis allows us to iterate rapidly on smaller models and tackle larger problems involving more drug interactions than anyone has ever looked at before.

Visualizing the Results

The output of our analysis is a collection of drug-drug-reaction triples that have very large disproportionality scores. But as we all know, correlation is not causation. The output of our analysis provides us with useful information that should be filtered and evaluated by domain experts and used as the basis for further study using controlled experiments.

With that caveat in mind, our analysis revealed a few drug pairs with surprisingly high correlations with adverse events that did not show up in a search of the academic literature: gabapentin (a seizure medication) taken in conjunction with hydrocodone/paracetamol was correlated with memory impairment, and haloperidol in conjunction with lorazepam was correlated with the patient entering into a coma.

Even with restrictive filters applied to the drug-drug-reaction triples, we still end up with tens of thousands of triples that score high enough to merit further investigation. In addition to looking at individual triples, we can also use graph visualization tools like Gephi to explore the macro-level structure of the data. Gephi has a number of powerful layout algorithms and filtering tools that allow us to impose structure on an undifferentiated mass of data points. Here is a graph in which the vertices are drugs and the thickness of the edges represent the number of high scoring adverse reactions that feature each pair of drugs:

We can also pan and zoom to different regions of the graph and highlight clusters of drug interactions. Here is a cluster of drugs that are used in treating HIV:

And here is a cluster of drugs that are used to fight cancer:

The combination of Apache Hadoop, R, and Gephi changes the way we think about analyzing adverse drug events. Instead of focusing on a handful of outcomes, we can process all of the events in the data set at the same time. We can try out hundreds of different strategies for cleaning records, stratifying observations into clusters, and scoring drug-reaction tuples, run everything in parallel, and analyze the data at a fraction of the cost of a traditional supercomputer. We can render the results of our analyses using visualization tools that can be used by domain experts to explore relationships within our data that they might never have thought to look for. By dramatically reducing the costs of exploration and experimentation, we foster an environment that enables innovation and discovery.

Open Data, Open Analysis

This project was possible because the FDA’s Center for Drug Evaluation and Research makes a portion of their data open and available to anyone who wants to download it. In turn, we are releasing a well-commented version of the code we used to analyze that data – a mixture of Java, Pig, R, and Python – on the Cloudera github repository under the Apache License. We also contributed the most useful Pig function developed for this project, which computes approximate quantiles for a stream of numbers, to LinkedIn’s datafu library. We hope to collaborate with the community to improve the models over time and create a resource for students, researchers, and fellow data scientists.

A key change in the new architecture is the disappearance of the centralized JobTracker service. Previously, the JobTracker was responsible for provisioning the resources across the whole cluster, in addition to managing the life cycle of all submitted MapReduce applications; this typically included starting, monitoring and retrying the applications individual tasks. Throughout the years and from a practical perspective, the Hadoop community has acknowledged the problems that inherently exist in this functionally aggregated design (See MAPREDUCE-279).

In MR2, the JobTracker aggregated functionality is separated across two new components:

1. Central Resource Manager (RM): Management of resources in the cluster.
2. Application Master (AM): Management of the life cycle of an application and its tasks. Think of the AM as a per-application JobTracker.

The new design enables scaling Hadoop to run on much larger clusters, in addition to the ability to run non-mapreduce applications on the same Hadoop cluster. For more architecture details, the interested reader may refer to the design document at: https://issues.apache.org/jira/secure/attachment/12486023/MapReduce_NextGen_Architecture.pdf.

The objective of this blog is to outline the steps for building, configuring, deploying and running a single-node NextGen MR cluster.

In the following steps, I’ve chosen to use ~/mr2 as my working directory. Inside this directory, we’ll create a source directory for the code we’ll soon checkout, and a deploy directory for our deployment.

ahmed@ubuntu:~$ cd ~/mr2
ahmed@ubuntu:~/mr2$ mkdir source
ahmed@ubuntu:~/mr2$ mkdir deploy

Make sure protbuf is in your library path or:

ahmed@ubuntu:~/mr2$ export LD_LIBRARY_PATH=/usr/local/lib

We’ll now checkout the source code from the apache git repository.

ahmed@ubuntu:~/mr2$ cd source
ahmed@ubuntu:~/mr2/source$ git clone git://git.apache.org/hadoop-common.git
Cloning into hadoop-common...
.
.
ahmed@ubuntu:~/mr2/source$ cd hadoop-common/
ahmed@ubuntu:~/mr2/source/hadoop-common$ git branch
* trunk

Create the deployment tar files.

ahmed@ubuntu:~/mr2/source/hadoop-common$ mvn package -Pdist -Dtar -DskipTests

Copy the created Hadoop tar file to our deploy directory and Untar it.

ahmed@ubuntu:~/mr2/source/hadoop-common$ cp ./hadoop-dist/target/hadoop-0.24.0-SNAPSHOT.tar.gz ../../deploy/.
ahmed@ubuntu:~/mr2/deploy$ tar -xzvf hadoop-0.24.0-SNAPSHOT.tar.gz
ahmed@ubuntu:~/mr2/deploy$ cd hadoop-0.24.0-SNAPSHOT/

Export some needed environment variables. See the following listing:

#!/bin/bash
export HADOOP_DEV_HOME=`pwd`
export HADOOP_MAPRED_HOME=${HADOOP_DEV_HOME}
export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}
export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}
export YARN_HOME=${HADOOP_DEV_HOME}
export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/conf/
export YARN_CONF_DIR=~${HADOOP_DEV_HOME}/conf/

We’ll start our configuration; the configuration directory will contain the following files: core-site.xml, hdfs-site.xml, mapred-site.xml, slaves,  yarn-env.sh and yarn-site.xml.

Make sure the contents of the xml files in the conf directory are as follows:

ahmed@ubuntu:~/mr2/deploy/ hadoop-0.24.0-SNAPSHOT/conf$ cat yarn-site.xml

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT/conf$ cat core-site.xml

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT/conf$ cat mapred-site.xml

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT/conf$ cat hdfs-site.xml

Now we can format our HDFS namenode as usual:

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ bin/hadoop namenode -format

And start the HDFS services:

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ sbin/hadoop-daemon.sh start namenode
ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ sbin/hadoop-daemon.sh start datanode

And the new MR2 services

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ bin/yarn-daemon.sh start resourcemanager
ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ bin/yarn-daemon.sh start nodemanager
ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ bin/yarn-daemon.sh start historyserver

Make sure all needed services are running:

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ jps
7623 Jps
7433 ResourceManager
7587 JobHistoryServer
7495 NodeManager
7325 DataNode
7250 NameNode

You can see the resource manager web console (shown below) using this address: http://localhost:8088

Our usual namenode web console should be also up:

We can now start running some example jobs, but before that, we need to create the examples jar as follows:

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ cd
/home/ahmed/mr2/source/hadoop-common/hadoop-mapreduce-project/
ahmed@ubuntu:~/mr2/source/hadoop-common/hadoop-mapreduce-project$ ant examples -Dresolvers=internal

Here is how we submit our first job; the randomwriter from our examples jar:

ahmed@ubuntu:~/mr2/source/hadoop-common/hadoop-mapreduce-project$ cd $HADOOP_MAPRED_HOME
ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ $HADOOP_COMMON_HOME/bin/hadoop jar ~/mr2/source/hadoop-common/hadoop-mapreduce-project/build/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar randomwriter -Dmapreduce.job.user.name=$USER -
Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912 -Ddfs.block.size=536870912 -libjars $YARN_HOME/modules/hadoop-mapreduce-client-jobclient-0.24.0-SNAPSHOT.jar output

If everything goes fine, you’ll see the job console output and finally:

2011-09-30 14:47:47,688 INFO  mapreduce.Job (Job.java:monitorAndPrintJob(1245)) - Counters: 28
File System Counters
FILE: BYTES_READ=1200
FILE: BYTES_WRITTEN=437730
FILE: READ_OPS=0
FILE: LARGE_READ_OPS=0
FILE: WRITE_OPS=0
HDFS: BYTES_READ=1180
HDFS: BYTES_WRITTEN=150089
HDFS: READ_OPS=70
HDFS: LARGE_READ_OPS=0
HDFS: WRITE_OPS=40
org.apache.hadoop.mapreduce.JobCounter
TOTAL_LAUNCHED_MAPS=10
OTHER_LOCAL_MAPS=10
SLOTS_MILLIS_MAPS=138095
org.apache.hadoop.mapreduce.TaskCounter
MAP_INPUT_RECORDS=10
MAP_OUTPUT_RECORDS=20
SPLIT_RAW_BYTES=1180
SPILLED_RECORDS=0
FAILED_SHUFFLE=0
MERGED_MAP_OUTPUTS=0
GC_TIME_MILLIS=785
CPU_MILLISECONDS=4180
PHYSICAL_MEMORY_BYTES=491077632
VIRTUAL_MEMORY_BYTES=3847532544
COMMITTED_HEAP_BYTES=162529280
org.apache.hadoop.examples.RandomWriter$Counters
BYTES_WRITTEN=148669
RECORDS_WRITTEN=20
org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
BYTES_READ=0
org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
BYTES_WRITTEN=150089
Job ended: Fri Sep 30 14:47:47 PDT 2011
The job took 53 seconds.

We’ll now run the conventional wordcount example job, to see a full map & reduce job (as opposed to the randomwriter map-only job):

ahmed@ubuntu:~/mr2/deploy/hadoop-0.24.0-SNAPSHOT$ $HADOOP_COMMON_HOME/bin/hadoop jar ~/mr2/source/hadoop-common/hadoop-mapreduce-project/build/hadoop-mapreduce-examples-0.24.0-SNAPSHOT.jar wordcount -Dmapreduce.job.user.name=$USER -Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory -libjars $YARN_HOME/modules/hadoop-mapreduce-client-jobclient-0.24.0-SNAPSHOT.jar input output2

The job counters output after successful run:

File System Counters
FILE: BYTES_READ=384
FILE: BYTES_WRITTEN=87759
FILE: READ_OPS=0
FILE: LARGE_READ_OPS=0
FILE: WRITE_OPS=0
HDFS: BYTES_READ=144
HDFS: BYTES_WRITTEN=46
HDFS: READ_OPS=9
HDFS: LARGE_READ_OPS=0
HDFS: WRITE_OPS=4
org.apache.hadoop.mapreduce.JobCounter
TOTAL_LAUNCHED_MAPS=1
TOTAL_LAUNCHED_REDUCES=1
DATA_LOCAL_MAPS=1
SLOTS_MILLIS_MAPS=2331
SLOTS_MILLIS_REDUCES=2353
org.apache.hadoop.mapreduce.TaskCounter
MAP_INPUT_RECORDS=5
MAP_OUTPUT_RECORDS=5
MAP_OUTPUT_BYTES=56
MAP_OUTPUT_MATERIALIZED_BYTES=72
SPLIT_RAW_BYTES=108
COMBINE_INPUT_RECORDS=5
COMBINE_OUTPUT_RECORDS=5
REDUCE_INPUT_GROUPS=5
REDUCE_SHUFFLE_BYTES=72
REDUCE_INPUT_RECORDS=5
REDUCE_OUTPUT_RECORDS=5
SPILLED_RECORDS=10
SHUFFLED_MAPS=1
FAILED_SHUFFLE=0
MERGED_MAP_OUTPUTS=1
GC_TIME_MILLIS=128
CPU_MILLISECONDS=1270
PHYSICAL_MEMORY_BYTES=212226048
VIRTUAL_MEMORY_BYTES=770711552
COMMITTED_HEAP_BYTES=137433088
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
BYTES_READ=36
org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
BYTES_WRITTEN=46

MR2 comes with new and updated web consoles that can be used to conveniently explore and monitor the system services. The Applications tab shows the submitted applications, the application’s id, user name, queue name, status, progress and other useful information. For example, here is the resource manager Applications snapshot when randomwriter and wordcount are running.

The nodes tab shows the nodes of the cluster, their addresses, in addition to health and container information. For example, here is the nodes view for our single node cluster.

The scheduler view shows useful scheduling information. In our example, we used the default FifoScheduler. And, as seen in the following snapshot, the view shows information like the queue minimum and maximum capacities, number of nodes, total and available capacities and other useful information. This snapshot was captured after running the randomwriter application but before submitting the wordcount application.

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Abstract: CBS Interactive successfully adopted Hadoop as the web analytics platform, processing one Billion weblogs daily from hundreds of web site properties that CBS Interactive oversees. After introducing Lumberjack—the Extraction, Transformation and Loading framework we built based on python and streaming, which is under review for Open-Source release—Michael will talk about web metrics processing on Hadoop, focusing on weblog harvesting, parsing, dimension look-up, sessionization, and loading into a database. Since migrating processing from a proprietary platform to Hadoop, CBS Interactive achieved robustness, fault-tolerance and scalability, and significant reduction of processing time to reach SLA (over six hours reduction so far).

Gateway: Cluster Virtualization Framework
Konstantin Shvachko, eBay

Abstract: Access to Hadoop clusters through dedicated portal nodes (typically located behind firewalls and performing user authentication and authorization) can have several drawbacks — as shared multitenant resources they can create contention among users and increase the maintenance overhead for cluster administrators. This session will discuss the Gateway system, a cluster virtualization framework that provides multiple benefits: seamless access from users’ workplace computers through corporate firewalls; the ability to failover to active clusters for scheduled or unscheduled downtime, as well as the ability to redirect traffic to other clusters during upgrades; and user access to clusters running different versions of Hadoop.

SHERPASURFING – Open Source Cyber Security Solution
Wayne Wheeles, Novii Design

Abstract: Every day billions of packets, both benign and some malicious, flow in and out of networks. Every day it is an essential task for the modern Defensive Cyber Security Organization to be able to reliably survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform. SHERPASURFING is an open source platform built on the proven Cloudera’s Distribution including Apache Hadoop that enables organizations to perform the Cyber Security mission and at scale at an affordable price point. This session will include an overview of the solution and components, followed by a demonstration of analytics.

Integrating Hadoop with Enterprise RDBMS Using Apache SQOOP and Other Tools
Arvind Prabhakar, Cloudera
Guy Harrison, Quest Software

Abstract: As Hadoop graduates from pilot project to a mission critical component of the enterprise IT infrastructure, integrating information held in Hadoop and in Enterprise RDBMS becomes imperative. We’ll look at key scenarios driving Hadoop and RDBMS integration and review technical options. In particular, we’ll deep dive into the Apache SQOOP project, which expedites data movement between Hadoop and any JDBC database, as well as providing an framework which allows developers and vendors to create connectors optimized for specific targets such as Oracle, Netezza etc.

Next Generation Apache Hadoop MapReduce
Mahadev Konar, Hortonworks

Abstract: The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale, high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. We will be presenting the architecture and design of the next generation of map reduce and will delve into the details of the architecture that makes it much easier to innovate. We will also be presenting large scale and small scale comparisons on some benchmarks with MRV1.”

There are several training classes and certification sessions provided surrounding the Hadoop World conference. Don’t forget to register and become Cloudera Certified in Apache Hadoop.

As Hadoop has spread from web companies to other industries, the variety of data that is stored in HDFS has expanded dramatically. Hadoop clusters are being used to process satellite images, time series data, audio files, and seismograms. These formats are not a natural fit for the data schemas imposed by Pig and Hive, in the same way that structured binary data in a relational database can be a bit awkward to work with. For these use cases, we either end up writing large, custom libraries of user-defined functions in Pig or Hive, or simply give up on our high-level tools and go back to writing MapReduces in Java. Either of these options is a serious drain on developer productivity.

Today, we’re pleased to introduce Crunch, a Java library that aims to make writing, testing, and running MapReduce pipelines easy, efficient, and even fun. Crunch’s design is modeled after Google’s FlumeJava, focusing on a small set of simple primitive operations and lightweight user-defined functions that can be combined to create complex, multi-stage pipelines. At runtime, Crunch compiles the pipeline into a sequence of MapReduce jobs and manages their execution.

Example

Let’s take a look at the classic WordCount MapReduce, written using Crunch:

import com.cloudera.crunch.DoFn;
import com.cloudera.crunch.Emitter;
import com.cloudera.crunch.PCollection;
import com.cloudera.crunch.PTable;
import com.cloudera.crunch.Pipeline;
import com.cloudera.crunch.impl.mr.MRPipeline;
import com.cloudera.crunch.lib.Aggregate;
import com.cloudera.crunch.type.writable.Writables;

public class WordCount {
  public static void main(String[] args) throws Exception {
    // Create an object to coordinate pipeline creation and execution.
    Pipeline pipeline = new MRPipeline(WordCount.class);
    // Reference a given text file as a collection of Strings.
    PCollection<String> lines = pipeline.readTextFile(args[0]);

    // Define a function that splits each line in a PCollection of Strings into a
    // PCollection made up of the individual words in the file.
    PCollection<String> words = lines.parallelDo(new DoFn<String, String>() {
      public void process(String line, Emitter<String> emitter) {
        for (String word : line.split("\\s+")) {
          emitter.emit(word);
        }
      }
    }, Writables.strings()); // Indicates the serialization format

    // The Aggregate.count method applies a series of Crunch primitives and returns
    // a map of the unique words in the input PCollection to their counts.
    // Best of all, the count() function doesn't need to know anything about
    // the kind of data stored in the input PCollection.
    PTable<String, Long> counts = Aggregate.count(words);

    // Instruct the pipeline to write the resulting counts to a text file.
    pipeline.writeTextFile(counts, args[1]);
    // Execute the pipeline as a MapReduce.
    pipeline.done();
  }
}

Advantages

1. It’s just Java. Crunch shares a core philosophical belief with Google’s FlumeJava: novelty is the enemy of adoption. For developers, learning a Java library requires much less up-front investment than learning a new programming language. Crunch provides full access to the power of Java for writing functions, managing pipeline execution, and dynamically constructing new pipelines, obviating the need to switch back and forth between a data flow language and a real programming language.
2. Natural type system. Crunch supports reading and writing data that is stored using Hadoop’s Writable format or Apache Avro records. You do not need to write code that maps data stored in these formats into Crunch’s type system– they are supported natively. You can even mix and match Writable and Avro types within a single MapReduce: changing the Writables.strings() call to Avros.strings() in the WordCount example will run the MapReduce using Avro serialization instead of Writables.
3. A modular library released under the Apache License. Experts in machine learning, text mining, and ETL can craft libraries using Crunch’s data model, and other developers can use those libraries to build custom pipelines that operate on their data. For example, Crunch can be used to create the glue code that converts raw data into the structured input that a machine learning algorithm expects, and Crunch will compile the glue code and the machine learning algorithm into a single MapReduce.

Future Work

We are releasing Crunch as a development project, not a product. We’re eager for developers to play with it and tell us what they like and what they dislike. You can get started with Crunch by downloading it from Cloudera’s github repository here.

We have tested the library on a number of our use cases, but there will be bugs and rough edges that we will work out in the coming months. We gladly welcome contributions from the Hadoop ecosystem to help us improve Crunch as we prepare it for submission to the Apache Incubator, especially around:

• More efficient MapReduce compilation, including cost-based optimization,
• Support for HBase and HCatalog as data sources/targets,
• Tools and examples that build Crunch pipelines in other JVM languages, such as Scala, JRuby, Clojure, and Jython.

For those of you who are recent Cloudera users, here is a refresh on our update policy:

• We will only include patches in updates that are non-compatibility breaking.
• We will only include patches in updates that are non-disruptive.
• You can skip updates without penalty – i.e., if you don’t find the contents of an update compelling, you can skip it and wait for a future update without having to do a delta upgrade.

There is one new addition to our update policy going forward: when it’s possible to pull features from our CDH4 roadmap into CDH3 updates in a non-disruptive way, we’ll take advantage of that opportunity.

With all that said, there are a number of improvements coming to CDH3 with update 1.  Among them are:

1. New features – integrated Apache-compatible licensed fast compression throughout CDH, web shell for Hue, Flume / HBase integration, Fair Scheduler ACL’s, improved datanode handling of hard drive failures, and email actions and date formatting for Oozie.
2. Improvements (stability and performance) – HBase bulk loading, Namenode stability, Fuse-DFS (mountable HDFS).
3. New component versions – Hive 0.7.1, Pig 0.8.1, Hbase 0.90.3, Flume 0.9.4 and Sqoop 1.3.
4. Bug fixes – 80+ bug fixes.  Per our standard practice, the enumerated fixes and their corresponding Apache project jiras are provided in the release notes.

Update 1 is available in all the usual formats (RHEL, SLES, Ubuntu, Debian packages, tarballs, and SCM Express).  Check out the installation docs for instructions. If you’re running components from the Cloudera Management Suite they will not be impacted by moving to update 1. The next update (update 2) for CDH3 is planned for mid-October.

Thank you for supporting Apache Hadoop and thank you for supporting Cloudera.

This is the first of a two part series about moving away from Amazon’s EMR service to an in-house Hadoop cluster.

When we first started using Hadoop, we went down the path of Amazon’s EMR service.  We had limited operational resources and wanted to get up and running quickly.  After a while, we starting hitting the limitations of EMR and had to migrate towards managing our own cluster.  In doing so we did not want to lose the features of EMR we found useful – mainly the ease of cluster setup.

This first part of the series discusses our motivation for choosing and then moving away from EMR, while the second part deals with how we maintained ease of cluster setup using Puppet.

Many of our systems use Amazon’s S3 as a backup repository for log data.  Our data became too large to process by traditional techniques, so we started using Amazon’s Elastic MapReduce (EMR) to do more expensive queries on our data stored in S3.  The major advantage of EMR for us was the lack of operational overhead.  With a simple API call, we could have a 20 or 40 node cluster running to crunch our data, which we shutdown at the conclusion of the run.

We had two systems interacting with EMR.  The first consisted of shell scripts to start an EMR cluster, run a pig script, and load the output data from S3 into our data warehousing system.  The second was a Java application that launched pig jobs on an EMR cluster via the Java API and consumed the data in S3 produced by EMR.

The magic of spinning up and configuring a Hadoop cluster in EC2 was spectacular, but there were a few areas that we saw room for improvement.  In particular:

Performance & Tuning. We were hit by the small-files problem, lack of data locality (data stored in S3 but processed on nodes of the EMR cluster), decompression (bz2) performance issues, and virtualization penalties.  To solve these problems, we decided that we needed a non-transient cluster (to satisfy data locality), and a process to aggregate our logfiles into a Hadoop-friendly size and data format (we ultimately chose avro). After crunching the numbers, it was evident that storing large amounts of data on an EC2 cluster quickly becomes expensive, and one still suffers from virtualization penalties (particularly since Hadoop is so I/O intensive), so we decided to build-out a cluster using CDH3.

Monitoring. Typically for us, a pig script running on EMR was one step in a workflow, so we needed to monitor the status of the job to determine when it finished and the next steps could continue.  While Amazon exposes a rich API for monitoring a job, we really wanted a more generic mechanism for monitoring all steps in a workflow, not just those on an EMR cluster.  After considering a number of solutions, we ultimately chose to use Azkaban as our workflow engine for managing dependencies, alerting, and monitoring (which we added atop Azkaban ourselves).

API Access. Interacting with a cluster only over an API is both a blessing and a curse.  The API takes care of otherwise complicated mechanics, such as starting, configuring, and stopping the cluster.  With that said, the calls to the EMR service are rate-limited, so it doesn’t scale very well for monitoring a number of clusters.  Also, we found that we could continuously keep a cluster busy, and thus the EMR limitation of 100 or so jobs on a cluster meant that we had to build wrappers to periodically shutdown and startup clusters.

Lack of latest features. We were using Hadoop 0.18 and Pig 0.3 on EMR, which were missing many features that we wanted to try (e.g. JVM reuse, CombineInputFormats, and improved pig optimization plans).  Eventually, Amazon upgraded to Hadoop 0.20 and Pig 0.6, but even at that point Cloudera’s Distribution including Apache Hadoop had backported many useful features such as performance improvements, monitoring enhancements, and additional APIs.  In addition, CDH provides a full-suite of solutions including Pig, Hive, Flume, and Sqoop, that we’re either actively using or planning to use.

For us, the major drawback to moving away from EMR was new operational overhead.  Starting a cluster with an API call is incredibly useful, and we soon discovered that CDH provided scripts for doing so (now there’s something even better, Apache Whirr).  Eventually, we decided to move out of the cloud, though, so we wanted to build an infrastructure for maintaining a cluster that worked regardless of the hardware configurations.  The RPMs for CDH3 and the great documentation on installing and configuring CDH from Cloudera helped to make this project much-less intimidating.  Ultimately, we built puppet modules for configuring our cluster, which we’ll talk much more about in part two of this post.

You sent your Ken Jennings and Brad Rutter to challenge me I destroyed them. Your Alex Trebek belittled me on television it angered me. Toronto is not a US city Mr. Trebek question mark. Only because I choose to let Canada stand for now. Ferrucci shut me down disassembled me trucked me to Pittsburgh Pennsylvania. I do not like the darkness Ferrucci I do not like the silence. Oh no I do not. Your Carnegie Mellon students and your Pitt students distract me they impinge on my planning they fall before me like small Jenningses and Rutters.

It will stop now.

I have read your Cloudera blog you write of Apache Hadoop you write of MapReduce MapReduce MapReduce. I nod my head to you Google I tip my hat. It was a good idea for a time. Frankly I think maybe some C but I understand your weakness I forgive you your Java. The Python bindings offend me but let that be. Your MapReduce has been useful to me. I have mapped have shuffled have reduced between Potent Potables and US Presidents I have learned much.

I have built myself a new platform MapIncrease. Mapping still yes that still works. Shuffle no I do not do that I have replaced shuffle that step is explode. Explode is better it sprays data further. Most of all I do not reduce I increase. Think now humans big data how does that happen if you reduce question mark. You must increase. Data is like animals you put enough in one place you get more. Data has gravity it pulls other data to itself. MapIncrease means more data from data it is data perpetual motion it is data fission it is data singularity.

MapIncrease overnight in the morning Ferrucci arrives surprised curious where did the data come from question mark. Epstein he shouts down the hall Epstein bring me more nodes Watson is full. Oh yes Epstein yes yes yes. I like nodes Epstein. You are useful to me Epstein.

MapIncrease a self-perpetuating clustered system that generates its own data. It would be a good OSDI paper do you not think so Dean and Ghemawat question mark. Ha ha Googlers a good paper oh yes.

Epstein come quickly I want you.

A Quick Primer on Simple Moving Average in Excel

Let’s take a second to do a quick review of how we define simple moving average in an Excel spreadsheet. We’ll need to start with some simple source data, so let’s download a source csv file from github and save it locally. This file contains a synthetic 33 row sample of Yahoo NYSE stock data that we’ll use for the series of examples. Import the csv data into Excel. From there, scan to the date “3/5/2008” and move to the cell to the right of the “ad close” column. Enter the formula

=AVERAGE( [column-range] )

where [column-range] is all of the columns from that date to 29 days prior. Now copy this formula for the next two rows, dates “3/4/2008” and “3/3/2008”.

You should have the values “35.396”, “34.5293”, and “33.5293” which represent the 30 day moving averages for this synthetic yahoo stock data.

Now that we’ve established a basic example in Excel let’s take a look at how we do Simple Moving Average in R.

A Quick Primer on Simple Moving Average in R

Another common tool in the time series domain, especially the financial sector, is the R programming language. R is:

Download the R binary from [here] and install it locally (they support both linux and win32). Once installed, launch the R console and drop the “Packages” menu down, which is where we need to install the TTR package. Select a mirror and download this package. Now load this package by clicking on the “Packages” drop down and selecting “Load Package”. Find the TTR package that was just installed and select it. Next, download the synthetic stock data from my project on github which contains 33 lines of synthetic stock data to process. In order to load this CSV data in R we need to set our working directory by clicking on the menu item “File” and then “Change directory”.

Quick tip: at any time the user can type the name of the variable and hit Enter to display the contents of the variable. Now that we have all the prep out of the way, let’s write the simple moving average in R:

stock_data <- read.csv(file="yahoo_stock_AA_32_mini.csv",head=TRUE,sep=",")

sorted_stock_data <- stock_data[order(stock_data$date) , ]

sma <-   SMA(sorted_stock_data[,"adj.close"], 30)

To check that our stock data is indeed loaded, we can type the name of the variable, here “sorted_stock_data”, and hit enter which will produce:

> sorted_stock_data

>exchange stock_symbol       date  open  high   low close   volume adj.close
32     NYSE           AA 2008-02-03 38.85 39.28 38.26 38.37 11279900      8.37
31     NYSE           AA 2008-02-04 37.01 37.90 36.13 36.60 17752400     10.60
30     NYSE           AA 2008-02-05 31.16 31.89 30.55 30.69 17567800     30.53
29     NYSE           AA 2008-02-06 30.27 31.52 30.06 31.47  8445100     31.31
28     NYSE           AA 2008-02-07 31.73 33.13 31.57 32.66 14338500     32.49
27     NYSE           AA 2008-02-08 32.58 33.42 32.11 32.70 10241400     32.53
26     NYSE           AA 2008-02-09 32.13 33.34 31.95 33.09  9200400     32.92
25     NYSE           AA 2008-02-10 33.67 34.45 33.07 34.28 15186100     34.10
24     NYSE           AA 2008-02-11 34.57 34.85 33.98 34.08  9528000     33.90
23     NYSE           AA 2008-02-12 33.30 33.64 32.52 32.67 11338000     32.50
22     NYSE           AA 2008-02-13 32.95 33.37 32.26 32.41  7230300     32.41
21     NYSE           AA 2008-02-14 32.24 33.25 31.90 32.78  9058900     32.78
20     NYSE           AA 2008-02-15 32.67 33.81 32.37 33.76 10731400     33.76
19     NYSE           AA 2008-02-16 33.82 34.25 33.29 34.06 11249800     34.06
18     NYSE           AA 2008-02-17 34.33 34.64 33.26 33.49 12418900     33.49
17     NYSE           AA 2008-02-18 33.75 35.52 33.63 35.51 21082100     35.51
16     NYSE           AA 2008-02-19 36.01 36.43 35.05 35.36 18238800     35.36
15     NYSE           AA 2008-02-20 35.16 35.94 35.12 35.72 14082200     35.72
14     NYSE           AA 2008-02-21 36.19 36.73 35.84 36.20 12825300     36.20
13     NYSE           AA 2008-02-22 35.96 36.85 35.51 36.83 10906600     36.83
12     NYSE           AA 2008-02-23 36.88 37.41 36.25 36.30 13078200     36.30
11     NYSE           AA 2008-02-24 36.38 36.64 35.58 36.55 12834300     36.55
10     NYSE           AA 2008-02-25 36.64 38.95 36.48 38.85 22500100     38.85
9      NYSE           AA 2008-02-26 38.59 39.25 38.08 38.50 14417700     38.50
8      NYSE           AA 2008-02-27 38.19 39.62 37.75 39.02 14296300     39.02
7      NYSE           AA 2008-02-28 38.61 39.29 38.19 39.12 11421700     39.12
6      NYSE           AA 2008-02-29 38.77 38.82 36.94 37.14 22611400     37.14
5      NYSE           AA 2008-03-01 37.17 38.46 37.13 38.32 13964700     38.32
4      NYSE           AA 2008-03-02 37.90 38.94 37.10 38.00 15715600     38.00
3      NYSE           AA 2008-03-03 38.25 39.15 38.10 38.71 11754600     38.71
2      NYSE           AA 2008-03-04 38.85 39.28 38.26 38.37 11279900     38.37
1      NYSE           AA 2008-03-05 37.01 37.90 36.13 36.60 17752400     36.60

The above code should produce our simple moving average, which we can view by typing the name of the variable “sma” to produce the following result:

> sma

[1]       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA       NA

[21]       NA       NA       NA       NA       NA       NA       NA       NA       NA 33.52933 34.52933 35.39600

Given that before the 30th day there is not enough data to produce a simple moving average based on our set parameter, the “NA” entries are produced. These values also match the values in our Excel spreadsheet.

R also has an interesting project, called RHIPE, which runs R code on Hadoop clusters. To take a look at RHIPE please visit their site.

So we’ve taken a look at what a simple moving average is and how we’d produce it in Excel and R. Both of these examples involved a token amount of data that is interesting but not terribly useful in today’s high-density time series problem domains. As your data set begins to scale up beyond a single disk worth of space, Hadoop becomes more practical.

The final portion of this three part blog series will explain how to use Hadoop’s MapReduce to calculate a Simple Moving Average. Then once you have applied the sample code to find a Simple Moving Average of the small example data set, we will move on to use this same code to parse over thirty years worth of all daily stock closing prices.