Today’s “Meet the Presenters” interview features two speakers: Aaron Myers from Cloudera and Suresh Srinivas from Hortonworks. Aaron and Suresh will be presenting on HDFS NameNode High Availability, one of the hottest topics in the Apache Hadoop space today.

Question: Tell us about your current role and how you interact with Apache Hadoop?

Aaron: I work full-time developing Hadoop and supporting Hadoop’s many users. My efforts are primarily focused on HDFS and Hadoop’s security infrastructure.

Suresh: I have been working on Hadoop for about 4 years. Currently I am on HDFS full-time, with focus on improving reliability, scalability and developing enterprise features. I also work on expanding Apache Hadoop APIs and interfaces to enable new use cases and simplify integration of other solutions with HDFS.

Question: Tell us about your Hadoop Summit presentation?

Suresh: The HDFS NameNode is a robust and reliable service as seen in practice in production at Yahoo! and other organizations. However, the NameNode does not have automatic failover support. A hot failover solution called HA NameNode is currently under active development (HDFS-1623). This talk will cover the architecture, design and setup. We will also discuss the future direction for HA NameNode.

Question: What do you expect will be the key takeaway for folks attending your session? 

Aaron: Because we will be sharing best practices and architectural details, we expect attendees to walk away with a good understanding of what’s required to deploy and operate a highly available HDFS NameNode.

Question: What are you most looking forward to at Hadoop Summit?

Aaron: Chatting in-person with the Hadoop developers and other community members who I interact with frequently, but don’t get to see often.

Suresh: Interacting with the community and learning from them their Hadoop experiences. I’m also interested in getting feedback on things we can improve and important new features desired by the community.

Question: What other presentations are you most looking forward to attending?

Aaron:

•  I accidentally the Namenode: Hadoop Distributed Filesystem Reliability and Durability at Facebook by Andrew Ryan
• Optimizing MapReduce Job Performance by Todd Lipcon

 Suresh:

• Like Aaron,  I accidentally the Namenode: Hadoop Distributed Filesystem Reliability and Durability at Facebook by Andrew Ryan
• Apache Hadoop and Virtual Machines by Richard McDougall and Sanjay Radia

Apache Hadoop consists of two primary components: HDFS and MapReduce. HDFS, the Hadoop Distributed File System, is the primary storage system of Hadoop, and is responsible for storing and serving all data stored in Hadoop. MapReduce is a distributed processing framework designed to operate on data stored in HDFS.

HDFS has long been considered a highly reliable file system.  An empirical study done at Yahoo! concluded that across Yahoo!’s 20,000 nodes running Apache Hadoop in 10 different clusters in 2009, HDFS lost only 650 blocks out of 329 million total blocks. The vast majority of these lost blocks were due to a handful of bugs which have long since been fixed.

Despite this very high level of reliability, HDFS has always had a well-known single point of failure which impacts HDFS’s availability: the system relies on a single Name Node to coordinate access to the file system data. In clusters which are used exclusively for ETL or batch-processing workflows, a brief HDFS outage may not have immediate business impact on an organization; however, in the past few years we have seen HDFS begin to be used for more interactive workloads or, in the case of HBase, used to directly serve customer requests in real time. In cases such as this, an HDFS outage will immediately impact the productivity of internal users, and perhaps result in downtime visible to external users. For these reasons, adding high availability (HA) to the HDFS Name Node became one of the top priorities for the HDFS community.

The remainder of this post discusses the implementation of a new feature for HDFS, called the “HA Name Node.” For a detailed discussion of other issues surrounding the availability of Hadoop as a whole, take a look at this excellent blog post by my colleague Eli Collins.

High-level Architecture

The goal of the HA Name Node project is to add support for deploying two Name Nodes in an active/passive configuration. This is a common configuration for highly-available distributed systems, and HDFS’s architecture lends itself well to this design. Even in a non-HA configuration, HDFS already requires both a Name Node and another node with similar hardware specs which performs checkpointing operations for the Name Node. The design of the HA Name Node is such that the passive Name Node is capable of performing this checkpointing role, thus requiring no additional Hadoop server machines beyond what HDFS already requires.

The HDFS Name Node is primarily responsible for serving two types of file system metadata: file system namespace information and block locations. Because of the architecture of HDFS, these must be handled separately.

Namespace Information

All mutations to the file system namespace, such as file renames, permission changes, file creations, block allocations, etc, are written to a persistent write-ahead log by the Name Node before returning success to a client call. In addition to this edit log, periodic checkpoints of the file system, called the fsimage, are also created and stored on-disk on the Name Node. Block locations, on the other hand, are stored only in memory. The locations of all blocks are received via “block reports” sent from the Data Nodes when the Name Node is started.

The goal of the HA Name Node is to provide a hot standby Name Node that can take over serving the role of the active Name Node with no downtime. To provide this capability, it is critical that the standby Name Node has the most complete and up-to-date file system state possible in memory. Empirically, starting a Name Node from cold state can take tens of minutes to load the namespace information (fsimage and edit log) from disk, and up to an hour to receive the necessary block reports from all Data Nodes in a large cluster.

The Name Node has long supported the ability to write its edit logs to multiple, redundant local directories. To address the issue of sharing state between the active and standby Name Nodes, the HA Name Node feature allows for the configuration of a special shared edits directory. This directory should be available via a network file system, and should be read/write accessible from both Name Nodes. This directory is treated as being required by the active Name Node, meaning that success will not be returned to a client call unless the file system change has been written to the edit log in this directory. The standby Name Node polls the shared edits directory frequently, looking for new edits written by the active Name Node, and reads these edits into its own in-memory view of the file system state.

Note that requiring a single shared edits directory does not necessarily imply a new single point of failure. It does, however, mean that the filer providing this shared directory must itself be HA, and that multiple network routes should be configured between the Name Nodes and the service providing this shared directory. Plans to improve this situation are discussed further below.

Block Locations

The other part of keeping the standby Name Node hot is making sure that it has up-to-date block location information. Since block locations aren’t written to the Name Node edit log, reading from the shared edits directory is not sufficient to share this file system metadata between the two Name Nodes. To address this issue, when HA is enabled, all Data Nodes in the cluster are configured with the network addresses of both Name Nodes. Data Nodes send all block reports, block location updates, and heartbeats to both Name Nodes, but Data Nodes will only act on block commands issued by the currently-active Name Node.

With both up-to-date namespace information and block locations in the standby Name Node, the system is able to perform a failover from the active Name Node to the standby with no delay.

Client Failover

Since multiple distinct daemons are now capable of serving as the active Name Node for a single cluster, the HDFS client must be able to determine which Name Node to communicate with at any given time. The HA Name Node feature does not support an active-active configuration, and thus all client calls must go to the active Name Node in order to be served.

To implement this feature, the HDFS client was extended to support the configuration of multiple network addresses, one for each Name Node, which collectively represent the HA name service. The name service is identified by a single logical URI, which is mapped to the two network addresses of the HA Name Nodes via client-side configuration. These addresses are tried in order by the HDFS client. If a client makes a call to the standby Name Node, a special result is returned to the client, indicating that it should retry elsewhere. The configured addresses are tried in order by the client until an active Name Node is found.

In the event that the active Name Node crashes while in the middle of processing a request, the client will be unable to determine whether or not the request was processed. For many operations such as reads (or idempotent writes such as setting permissions, setting modification time, etc), this is not a problem — the client may simply retry after the failover has completed. For others, the error must be bubbled up to the caller to be correctly handled. In the course of the HA project, we extended the Hadoop IPC system to be able to classify each operation’s idempotence using special annotations.

Current Status

Active development work began on the HA Name Node in August 2011, in a branch off of Apache Hadoop trunk. Development was done under the umbrella JIRAs HDFS-1623 and HADOOP-7454. Last Friday, March 2nd 2012 we merged this branch back into Apache Hadoop trunk. We closed over 170 individual JIRAs in the course of implementing this feature. The stated intention of the community is to merge this work from HDFS trunk into the 0.23 branch, where it will be released as an update of the Apache Hadoop 0.23 release line. Much of this work is already available as part of CDH4 beta 1, released on February 13th, 2012.

Once a failover has been initiated, the actual process of stopping the active and starting the standby Name Node takes a matter of seconds or less. This speed allows for little or no detectable service disruption during a failover. I’ve personally run hundreds of MR jobs over a running HA cluster, doing failovers back and forth between two HA Name Nodes, without any job failures.

This first implementation of the HA Name Node supports only manual failover — that is, failure of one of the Name Nodes is not automatically detected by the system, but rather requires intervention by an operator to initiate a failover between the Name Nodes. Though this is an obvious limitation, this version should still be useful to eliminate the need for planned HDFS downtime in many cases, e.g. changing the configuration of the Name Node, scheduled hardware maintenance of a Name Node, or scheduled OS upgrade of a Name Node.

Next Up

The highest priority feature to add to the HA Name Node implementation is support for automatically detecting the failure of the Active Name Node and initiating a failover to the Standby when it is determined that the Active is no longer functional. HDFS-3042 and its sub-tasks are actively being worked on to provide this functionality.

The dependence on an HA filer for HDFS edit logs is a limitation that we’d like to address in the near to medium term as well. Several different options have been discussed to address this:

• BookKeeper – BookKeeper is a highly available write-ahead logging system. Work has already been done to allow the HDFS Name Node to be able to write its edits log to BookKeeper, though this has not yet been tested with the HA Name Node.
• Multiple, non-HA filers – the HA Name Node presently only supports logging to a single shared edits directory. Perhaps the easiest improvement from the current situation would be to allow the Name Node to log to several shared edits directories, and require that all edits be logged to a quorum of shared edits directories. This proposal is being tracked by HDFS-2782.
• Stream edits to remote NNs – in addition to writing edits to a local file system, edit log entries could be sent directly to other Name Nodes over the network. The active Name Node would require a quorum of the involved Name Nodes to acknowledge receipt of the edits before responding with success to the client call.
• Store edit logs in HDFS itself – systems such as HBase already use HDFS to store a write-ahead log of all data mutations. If HDFS were extended to have a modicum of bootstrapping information, it is not inconceivable that HDFS edit logs could be stored in HDFS itself. This proposal is being discussed on HDFS-2601.

In the next few weeks, we will be evaluating all of these options and selecting one to implement.

Currently, deploying HA Name Nodes is somewhat cumbersome, requiring the operator to manually synchronize the on-disk metadata of the two Name Nodes. HDFS-2731 aims to improve the user experience of this deployment process by having the second Name Node automatically synchronize itself with the state of the first Name Node. This feature will make the process faster and less error prone.

Further Reading

Take a look at the CDH4 docs for detailed information on configuring the HA Name Node in CDH4.

Be on the lookout for an upcoming blog post from my colleague Todd Lipcon, which will go into greater detail about some of the specific challenges encountered while implementing the HA Name Node feature, and how these issues were overcome.

Acknowledgments

This work has been a community effort from the start, and represents the work of many contributors. Both the architecture and implementation were the collaborative effort of many. In particular, this work would not have been possible without contributions from Todd Lipcon, Eli Collins, Uma Maheswara Rao G, Bikas Saha, Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay Radia, Mingjie Lai, and Gregory Chanan. Also thanks to Dhruba Borthakur and Konstantin Shvachko for helpful design discussions and recommendations on testing. Thanks also to Stephen Chu, Wing Yew Poon, and Patrick Ramsey for their help in testing the HA Name Node.

Before getting into indexing within Hadoop, let us review the features of Lucene and Solr:

Apache Lucene and Apache Solr

Apache Lucene is a mature, high performance, full-featured Java API used for indexing and searching that has been around since the late nineties — it supports field-specific indexing and searching, sorting, highlighting, and wildcard searches, to name only a few. Everything in Lucene boils down to creating a document using artifacts such as email messages, HTML, PDF, XML, Word, Excel, etc, the contents of which will end up being parsed and added to Lucene documents as name/value pairs.  There are a number of libraries available for extracting actual content, depending on what the artifact is. When extracting content from .msg email files, for instance, TIKA and POI are some useful libraries.

Once you have added name/value pairs from the email content to the document, the index portion is taken care of. We can then use IndexSearcher to search through the indexed contents as illustrated below, in Figure 1:

Figure 1: Indexing and Searching using Lucene.

Apache Solr, on the other hand, is a Lucene-based full text search server with XML, JSON, and HTTP APIs, which has a web admin interface and provides extensive caching, replication, search distribution, as well as the ability to add customized plugins. Solr already includes various parsing libraries, including Tika, POI, and TagSoup, among others.

Figure 2 below illustrates the Solr components and deployment architecture:

Figure 2: Solr Components and Deployment Architecture.

Next, let’s explore how you can use both Solr and Lucene within the Hadoop environment for indexing and searching massive amounts of data:

First, you need to get data into HDFS, as covered in Hadoop for Archiving Email – Part 1. Once the data is there, you can start to run MapReduce to create indexes in parallel that can then be dumped into HDFS or into a Local File System. If an index is stored within a Local File System, simply serve it from there by pointing Solr to it. It can be run either in single, replicated or distributed mode, depending on the size of the index to serve.  However, if you need to make search available for only for a small number of users, you can simply store data directly in  HDFS and provide an interface for your users to access it directly.

One tool that I find very handy for providing such an interface is Luke. Luke was built for development and diagnostic purposes, and can be used to search, display and browse the results of a Lucene index. With Luke, you can view documents, analyze results, copy and delete them, or optimize indexes that have already been built. The best part: you can easily operate on multipart indexes stored directly in HDFS, as illustrated in Figure 3:

Figure 3: Indexing and Searching within HDFS or Local Filesystem.

Having discussed design at a high level, let’s now dive deeper into the details of MapReduce for creating an index.

Here is how the configure portion of each mapper could look:

//initialize indexWriter..
Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_33);
IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_33, analyzer);
//if we are writing to hdfs, then use RAMDirectory
if (toHDFS){
      iwc.setOpenMode(OpenMode.CREATE);
      idx =  new RAMDirectory();
      writer = new IndexWriter(idx, iwc);
} else {
     //use CREATE_OR_APPEND so if index already exists it will simply be appended to
     iwc.setOpenMode(OpenMode.CREATE_OR_APPEND);
     idx = FSDirectory.open(new File(outputDir));
     writer = new IndexWriter(idx, iwc);
}

1. Initialize the analyzer and index writer config.
2. If writing to HDFS, you can use RAMDirectory to hold the indexes created; and once complete, flush to HDFS.
3. If writing to a local file system, simply create FSDirectory with the location.

Having configured the mapper, let’s look at the map method:

public void map(LongWritable key, Text value, OutputCollector output,
Reporter reporter) throws IOException {
try {
     Document doc = new Document();
     //add email file path
     String path = key.toString();
     Fieldable field = new Field("path", path,
     Field.Store.YES, Field.Index.ANALYZED);
     doc.add(field);

     //convert content into MapiMessage
     InputStream input = new
     ByteArrayInputStream(value.getBytes());
     MAPIMessage msg = new MAPIMessage(input);
     //add recipient as stored and analyzed field so we can
     //search based on recipient and display recipient name in the results
     String recipient = msg.getRecipientEmailAddress();
     field = new Field("receipient",
     recipient,Field.Store.YES, Field.Index.ANALYZED );
     doc.add(field);

     String subject = msg.getSubject();
     field = new Field("subject", subject, Field.Store.YES,
     Field.Index.ANALYZED );
     doc.add(field);

     String content = msg.getTextBody();
     field = new Field("content", content, Field.Store.YES,
     Field.Index.ANALYZED );
     doc.add(field);

     //add more fine grained fields based on search criteria needed
     ...

     writer.addDocument(doc);
} catch (Exception e) {
     e.printStackTrace();
}
}

Once it’s configured within each map, the exercise boils down to parsing the content and adding it to the writer. The above code does the following:

1. Create a Lucene document.
2. Parse email content passed into MAPIMessage.
3. Extract necessary fields and add it to the index. (In this example I only extract recipient, subject and content. You can add more fields as necessary and use Hbase to store the content if needed.)

At this point, the index has been created – either in memory or the Local File System. The final task is to close the index to make it searchable, which can be done within the close method of the mapper, as demonstrated below:

public void close() {
try {
writer.optimize();
writer.close();
if (toHDFS) {
     String files[]= idx.listAll();
     FileSystem dfs = FileSystem.get(localConf);
     for (int i = 0; i < files.length; i++){
          //Read data into byte array, create file in HDFS, write bytes to that file
           ....
          fs.write(array);
          fs.close();
     }
}
} catch (CorruptIndexException e) {
     e.printStackTrace();
} catch (IOException e) {
     e.printStackTrace();
}
}

1. Close the created index.
2. In the case of HDFS, walk through the index in memory and write it to HDFS.

The index should now have been created, either in the Local File System of each of the DataNodes, or in HDFS directly. If it is in the Local File System, you can opt to make the directory part of the “www” directory and enable Solr to serve it from there. If it is in HDFS, one could load the index in RAMDirectory within each mapper and search, use a tool like Luke to provide a search interface, or put a mechanism in place to copy it to the Local File System to point Solr at it.

Appending

Appending to an existing index can be a bit tricky. If the index sits in a Local File System, this can be accomplished by setting the index writer to APPEND mode and adding new documents. This can get a bit more complicated, however, when the index is in HDFS. One option would be to write an index to a new directory in HDFS, then merge with the existing index.

SolrCloud and Katta

Since we are discussing fast search options, it also makes sense to touch on components like SolrCloud and Katta.

SolrCloud enables clusters of Solr instances to be created, with a central configuration, automatic load balancing, resizing, rebalancing and fail-over.

Katta serves indexes in a distributed manner similar to HDFS. It has built in replication for fail-over and performance, is easy to integrate with Hadoop clusters and has master fail-over. However, it does not provide real-time updates, nor is it an indexer – it is simply a serving tool for Lucene indexes.

In Part 3 of this series, I will cover ways to ingest such email messages and ways to put the steps involved in a workflow. In the meantime, drop us a line if you have any questions on storing email message in Hadoop and index and search them using Solr and Lucene.

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Abstract: CBS Interactive successfully adopted Hadoop as the web analytics platform, processing one Billion weblogs daily from hundreds of web site properties that CBS Interactive oversees. After introducing Lumberjack—the Extraction, Transformation and Loading framework we built based on python and streaming, which is under review for Open-Source release—Michael will talk about web metrics processing on Hadoop, focusing on weblog harvesting, parsing, dimension look-up, sessionization, and loading into a database. Since migrating processing from a proprietary platform to Hadoop, CBS Interactive achieved robustness, fault-tolerance and scalability, and significant reduction of processing time to reach SLA (over six hours reduction so far).

Gateway: Cluster Virtualization Framework
Konstantin Shvachko, eBay

Abstract: Access to Hadoop clusters through dedicated portal nodes (typically located behind firewalls and performing user authentication and authorization) can have several drawbacks — as shared multitenant resources they can create contention among users and increase the maintenance overhead for cluster administrators. This session will discuss the Gateway system, a cluster virtualization framework that provides multiple benefits: seamless access from users’ workplace computers through corporate firewalls; the ability to failover to active clusters for scheduled or unscheduled downtime, as well as the ability to redirect traffic to other clusters during upgrades; and user access to clusters running different versions of Hadoop.

SHERPASURFING – Open Source Cyber Security Solution
Wayne Wheeles, Novii Design

Abstract: Every day billions of packets, both benign and some malicious, flow in and out of networks. Every day it is an essential task for the modern Defensive Cyber Security Organization to be able to reliably survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform. SHERPASURFING is an open source platform built on the proven Cloudera’s Distribution including Apache Hadoop that enables organizations to perform the Cyber Security mission and at scale at an affordable price point. This session will include an overview of the solution and components, followed by a demonstration of analytics.

Integrating Hadoop with Enterprise RDBMS Using Apache SQOOP and Other Tools
Arvind Prabhakar, Cloudera
Guy Harrison, Quest Software

Abstract: As Hadoop graduates from pilot project to a mission critical component of the enterprise IT infrastructure, integrating information held in Hadoop and in Enterprise RDBMS becomes imperative. We’ll look at key scenarios driving Hadoop and RDBMS integration and review technical options. In particular, we’ll deep dive into the Apache SQOOP project, which expedites data movement between Hadoop and any JDBC database, as well as providing an framework which allows developers and vendors to create connectors optimized for specific targets such as Oracle, Netezza etc.

Next Generation Apache Hadoop MapReduce
Mahadev Konar, Hortonworks

Abstract: The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale, high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. We will be presenting the architecture and design of the next generation of map reduce and will delve into the details of the architecture that makes it much easier to innovate. We will also be presenting large scale and small scale comparisons on some benchmarks with MRV1.”

There are several training classes and certification sessions provided surrounding the Hadoop World conference. Don’t forget to register and become Cloudera Certified in Apache Hadoop.

For those of you who are recent Cloudera users, here is a refresh on our update policy:

• We will only include patches in updates that are non-compatibility breaking.
• We will only include patches in updates that are non-disruptive.
• You can skip updates without penalty – i.e., if you don’t find the contents of an update compelling, you can skip it and wait for a future update without having to do a delta upgrade.

There is one new addition to our update policy going forward: when it’s possible to pull features from our CDH4 roadmap into CDH3 updates in a non-disruptive way, we’ll take advantage of that opportunity.

With all that said, there are a number of improvements coming to CDH3 with update 1.  Among them are:

1. New features – integrated Apache-compatible licensed fast compression throughout CDH, web shell for Hue, Flume / HBase integration, Fair Scheduler ACL’s, improved datanode handling of hard drive failures, and email actions and date formatting for Oozie.
2. Improvements (stability and performance) – HBase bulk loading, Namenode stability, Fuse-DFS (mountable HDFS).
3. New component versions – Hive 0.7.1, Pig 0.8.1, Hbase 0.90.3, Flume 0.9.4 and Sqoop 1.3.
4. Bug fixes – 80+ bug fixes.  Per our standard practice, the enumerated fixes and their corresponding Apache project jiras are provided in the release notes.

Update 1 is available in all the usual formats (RHEL, SLES, Ubuntu, Debian packages, tarballs, and SCM Express).  Check out the installation docs for instructions. If you’re running components from the Cloudera Management Suite they will not be impacted by moving to update 1. The next update (update 2) for CDH3 is planned for mid-October.

Thank you for supporting Apache Hadoop and thank you for supporting Cloudera.

Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and write) over HTTP/S.

Hoop can be used to:

Hoop has a Hoop client and a Hoop server component:

Hoop and Hadoop HDFS Proxy

Hoop server is a full rewrite of Hadoop HDFS Proxy. Although it is similar to Hadoop HDFS Proxy (runs in a servlet-container, provides a REST API, pluggable authentication and authorization), Hoop server improves many of Hadoop HDFS Proxy shortcomings by providing:

Accessing HDFS files -via Hoop- using Unix ‘curl’ command

Assuming Hoop is running on http://hoopbar:14000, the following examples show how the Unix ‘curl’ command can be used to access data in HDFS via Hoop using pseudo authentication.

Getting the home directory:

$ curl -i "http://hoopbar:14000?op=homedir&user.name=babu"
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"homeDir":"http:\/\/hoopbar:14000\/user\/babu"}
$

Reading a file:

$ curl -i "http://hoopbar:14000?/user/babu/hello.txt&user.name=babu"
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Hello World!
$

Writing a file:

$ curl -i -X POST "http://hoopbar:14000/user/babu/data.txt?op=create" --data-binary @mydata.txt --header "content-type: application/octet-stream"
HTTP/1.1 200 OK
Location: http://hoopbar:14000/user/babu/data.txt
Content-Type: application/json
Content-Length: 0
$

Listing the contents of a directory:

$ curl -i "http://hoopbar:14000?/user/babu?op=list&user.name=babu"
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

[
  {
    "path" : "http:\/\/hoopbar:14000\/user\/babu\/data.txt"
    "isDir" : false,
    "len" : 966,
    "owner" : "babu",
    "group" : "supergroup",
    "permission" : "-rw-r--r--",
    "accessTime" : 1310671662423,
    "modificationTime" : 1310671662423,
    "blockSize" : 67108864,
    "replication" : 3
  }
]
$

Click this link for more details about the Hoop HTTP REST API.

Getting Hoop

Hoop is distributed with an Apache License 2.0.

The source code is available at http://github.com/cloudera/hoop.

Instructions on how to build, install and configure Hoop server and the rest of documentation is available at http://cloudera.github.com/hoop.

Contributing Hoop to Apache Hadoop

The goal is to contribute Hoop to Apache Hadoop as the next generation of Hadoop HDFS proxy. We are just waiting on the Mavenization of Hadoop Common and Hadoop HDFS which will make integration easier.

Background

When discussing Hadoop availability people often start with the NameNode since it is a single point of failure (SPOF) in HDFS, and most components in the Hadoop ecosystem (MapReduce, HBase, Pig, Hive etc) rely on HDFS directly, and are therefore limited by its availability. However, Hadoop availability is a larger, more general issue, so it’s helpful to establish some context before diving in.

Availability is the proportion of time a system is functioning [1], which is commonly referred to as “uptime” (vs downtime, when the system is not functioning).

Note that availability is a stricter requirement than fault tolerance – the ability for a system to perform as designed and degrade gracefully in the presence of failures. A system that requires an hour to restart (eg for a configuration change or software upgrade) but has no single point of failure is fault tolerant but not highly available (HA). Adding redundancy in all SPOFs is a common way to improve fault tolerance, which helps [2], but is just a part of, improving Hadoop availability. Note also that fault tolerance is distinct from durability, even though the NameNode is a SPOF no single failure results in data loss as copies of NameNode persistent state (the image and edit log) are replicated both within and across hosts.

Availability is also often conflated with reliability. Reliability in distributed systems is a more general issue than availability [3]. A truly reliable distributed system must be highly available, fault tolerant, secure, scalable, and perform predictably, etc. I’ll limit this post to Hadoop availability.

Reasons for downtime

An important part of improving availability and articulating requirements is understanding the causes of downtime. There are many types of failures in distributed systems, ways to classify them, and analyses of how failures result in downtime. Rather than go into depth here, I’ll briefly summarize some general categories of issues that may cause downtime:

1. Maintenance – Hardware and software may need to be upgraded, configuration changes may require a system restart, and operational tasks for dependent systems. Hadoop can handle most maintenance to slave hosts without downtime; however maintenance to a master host normally requires a restart of the entire system.

2. Hardware failures – Hosts and their connections may fail. Without redundant devices, or redundant components within devices, a single component failure may cause the entire device to fail. Hadoop can tolerate hardware failures (even silent failures like corruption) to slave hosts without downtime; however some hardware failures on the master host (or a failure in the connection between the master and the majority of the slaves) can cause system downtime [4].

3. Software failures – Software bugs may cause a component in the system to stop functioning or require a restart. For example, a bug in upgrade code could result in downtime due to data corruption. A dependent software component may become unavailable (eg the Java garbage collector enters a stop-the-world phase). Hadoop can tolerate some software bugs without downtime; however components are generally designed to fail-fast – to stop and notify other components of failure rather than attempt to continue a possibly-flawed process. Therefore a software bug in a master service will likely cause downtime.

4. Operator errors – People make mistakes. From disconnecting the wrong cable, to mis-configured hosts, to typos in configuration files, operator errors can cause downtime. Hadoop attempts to limit operator error by simplifying administration, validating its configuration, and providing useful messages in logs and UI components; however operator mistakes may still cause downtime.

Use cases

In order for a system to be highly available, its design needs to anticipate these various failures. Removing single points of failure, enabling rolling upgrades, faster restarts, making the system robust and user friendly, etc are all necessary to improve availability. Given that improving availability requires a multi-prong approach, let’s take a look at the relevant use cases for limiting downtime.

1. Host maintenance – If an operator needs to upgrade or replace the primary host hardware or upgrade its operating system, they should be able to manually fail over to a hot standby, perform the upgrade and optionally fail back to the primary. The fail-over should be transparent to clients accessing the system (eg active jobs continue to run). Host maintenance to slave hosts can be handled without downtime today by de-commissioning the host.

2. Configuration changes – Ideally configuration changes to masters should not require a system restart — the configuration can be updated in-place or fail-over to a hot standby with an updated configuration is supported. In cases when they do, the operator should be able to restart the system with minimal impact to running workloads.

3. Software upgrades – An operator should be able to upgrade Hadoop’s software in-place (a “rolling upgrade”) on slave nodes and via fail-over on the master hosts so there is little or no downtime. If a restart is required it should be accelerated by quickly re-constructing the system’s state.

4. Host failures – If a non-redundant hardware component fails, the operating system crashes, a disk partition runs out of space, etc. the system should detect the failure, and, depending on the service and failure, (a) recover, (b) de-commission itself, or (c) fail over to a hot standby. Hadoop currently tolerates slave host failures without downtime, however master host failures often cause downtime. In practice, for a number of reasons, master hardware failures do not cause as much downtime as you might expect:

Highly Available Hadoop

A number of efforts are under way to improve Hadoop availability, and implement missing functionality required by the above use cases. Tasks related to HDFS availability are tracked here, tasks related to MapReduce availability are tracked here.

1. Improvements to Hadoop’s failure handling code. Hadoop’s native fault injection framework and other related frameworks continue to make Hadoop more robust in the face of failures. Recent advances in failure testing applied have been successfully applied to Hadoop [5] to identify software bugs (eg HDFS-1231, HDFS-1225, and HDFS-1228).

2. Work was recently started to allow Hadoop configuration changes without restart. As Hadoop incorporates this change configuration parameter changes will increasingly be possible without downtime.

3. A number of changes (eg HDFS-1070, HDFS-1295, and HDFS-1391) are underway to significantly improve the time it takes to restart HDFS.

4. Work has started to allow Hadoop client and server software of different versions to co-exist, with the goal of enabling in-place Hadoop software upgrades.

5. There have been efforts to make existing releases of HDFS more highly-available, as well as several research prototypes (eg UpRight-HDFS, NameNode Cluster, and HDFS-dnn) that examine HDFS availability. HDFS developers are currently working on a hot standby for the NameNode to improve on the existing NameNode fail-over. Like the Google File System which has “shadow masters”, this allows HDFS to fail the NameNode process from one host to another by actively replicating all NameNode state required to quickly restart the process. Integrating the BackupNode (edits are streamed from the primary NameNode to one or more BackupNodes) or using BookKeeper (a replicated service to reliably log streams of records that can be used for the edits log) with the AvatarNode (which replicates block reports across a primary and backup host) results in a standby NameNode that can be activated if the NameNode fails. Automatic hot fail-over can be achieved by integrating both clients and servers with ZooKeeper. A similar approach has been successfully used by Google to make GFS highly available [6]:

Initially, GFS had no provision for automatic master failover. It was a manual process. Although it didn’t happen a lot, whenever it did, the cell might be down for an hour. Even our initial master-failover implementation required on the order of minutes. Over the past year, however, we’ve taken that down to something on the order of tens of seconds.

This Active/Passive design enables both high availability and evolution towards being a better storage layer for systems like HBase, which in turn could be used to store metadata for a new version of HDFS (similar to Google’s GFS2). Like HDFS federation, this provides an evolutionary path to high scalability without the complexity of modifying HDFS to use multi-master replication.

6. The MapReduce master (JobTracker) state is stored in HDFS and is therefore limited by its availability. The JobTracker can be-restarted, however works needs to be to integrate it with a service like ZooKeeper to handle fail-over to a separate host.

Hopefully this post has helped frame the various tasks behind Hadoop’s march towards high availability in a useful context. The development community understands this is one of the most high priority issues for users, and is looking forward to providing a highly available Hadoop in up-coming releases. Similarly, Cloudera is committed to improving availability in CDH4 – it’s our primary focus for the release.

Thanks to Dhruba Borthakur, Doug Cutting, Sanjay Radia, and Konstantin Shvachko for reading drafts of this post.

Footnotes

[1] A common way to define availability is the ratio of the expected value of the uptime of the system to the aggregate of the expected values of uptime and downtime. Common metrics used are:

• Mean time between failures (MTBF) – the expected time between failures of a system during operation.
• Mean time to recovery (MTTR) – the average time the system will take to recover from any failure.

Using these metrics availability can be defined as MTBF / (MTBF + MTTR).

[2] I say “helps” because one the most common reasons for downtime (misconfiguration, operator error, and software bugs) are all exacerbated by system complexity, and making systems more fault tolerant often increases their complexity.

[3] “Reliability and availability are different: Availability is doing the right thing within the specified response time. Reliability is not doing the wrong thing.” from WHY DO COMPUTERS STOP AND WHAT CAN BE DONE ABOUT IT? by Jim Gray

[4] People often configure master hosts with redundant hardware components (nics, disks, IO controllers, and power units) so that an individual component failure does not cause the system to fail.

[5] Towards Automatically Checking Thousands of Failures with Micro-specifications. H Gunawi, T Do, et al. UC Berkeley TR EECS-2010-98.

[6] GFS: Evolution on Fast-Forward. Marshall Kirk McKusick, Sean Quinlan in the ACM Queue.

We appreciate feedback! Get in touch with us on the CDH user list, twitter or IRC (#cloudera on freenode.net) and let us know how the update is working for you.

From the statistical or technical perspective, fraud is a rare event that results in a significant financial impact to the organization.

Both definitions emphasize that the event is rare (assuming that most of the population is law-abiding citizens), is intentional (there is no “accidental” fraud), as well as imply a significant damage caused to the defrauded party (otherwise why bother).  Fraud detection is difficult from statistical point of view for exactly these reasons: (a) the events are rare and it is difficult to build a predictive model and (b) fraud assumes a real human being behind it and incorporates elements of game theory since the fraudster is often an insider who knows how to game the system.

Fraud and Rare Events

By definition, fraud is an unexpected or rare event with significant financial or other damage.  Fraud assumes that the fraudster has some prior information how the current system works including previous successful and unsuccessful fraud cases and possibly the fraud detection mechanisms.  The above breaks the standard statistical modeling assumption, the variable independence or i.i.d. assumption, making building a reliable statistical model difficult.  Often the fraudster is working in the same industry that the fraud detection is supposed to protect, is intimately familiar with the fraud detection methods, and is actively trying to avoid detection by masquerading.

Rare event detection problem is also applicable to online advertising and marketing, particularly with predicting “long tail” events and terrorism detection.

One common example of fraud is associated with Taleb distribution where a seemingly high probability of a small gain shadows a small probability of a large loss that more than outweighs the gains.  Relatively long periods of slightly better than moderate gains are interrupted by a rare event of large losses.  It is easy to defraud investors by presenting the results of partial analysis excluding the “rare events”.

Fraud Prevention

Since fraud is so hard to prove in courts, most organizations and individuals try to prevent fraud from happening by blanket measures.  This includes limiting the amount of damage the fraudster can impact on the organization as well as early detection of fraud patterns.  For example, credit card companies can cut the credit card limit across the board in anticipation of a few negative fraud cases.  Advertisers can prevent advertising campaigns with low number of qualifying events.  And anti-terrorism agencies can prevent people with bottles of pure water from boarding the planes.  These actions are often in contrast with the company efforts to attract more customers and result in general dissatisfaction.  To the rescue are new technologies like Hadoop, Influence Diagrams and Bayesian Networks which are computationally expensive (these are NP-hard in computer science terminology) but are more accurate and predictive.

Why Hadoop?

Hadoop is a distributed system for processing large amounts of data.  In a recent Hadoop Summit 2010 Yahoo, Facebook, and other companies announced that they currently process a few TBs of data per day and the volumes are growing at exponential rates.  Hadoop can be vital for solving the fraud detection problem because:

1. Sampling does not work for rare events since the chance of missing a positive fraud case leads to significant deterioration of model quality.
2. Hadoop can solve much harder problems by leveraging multiple cores across thousands of machines and search through much larger problem domains.
3. Hadoop can be combined with other tools to manage moderate to low response latency requirements.

Let’s go through these reasons one by one.  Sampling is a common technique for modeling rare events.  One of the problems with sampling is that we cannot afford to throw away rare positive cases.  Even in a stratified or proportional sampling scheme one has to retain all positive cases since the model accuracy heavily depends on them (one can usually discard some negative cases though).  Given the above, the system still has to go through the whole dataset to sieve through the positive and negative cases.

Hadoop is known for its gnawing power.  Nothing can compare with the throughput power of thousands of machines each of which has multiple cores.  As was reported recently at the Hadoop Summit 2010, the largest installations of Hadoop have 2,000 to 4,000 computers with 8 to 12 cores each, amounting to up to 48,000 active threads looking for a pattern at the same time.  This allows either (a) looking through larger periods of time to incorporate events across a larger time frame or (b) taking more sources of information into account.  It is quite common among social network companies to comb through twitter blogs in search of relevant data.

Finally, one of the fraud prevention problems is latency.  The agencies want to react to an event as soon as possible, often within a few minutes of the event.  Yahoo recently reported that it can adjust its behavioral model in a response to a user click event within 5-7 minutes across several hundred of millions of customers and billions of events per day.  Cloudera has developed a tool, Flume, that can load billions of events into HDFS within a few seconds and analyze them using MapReduce.

Often fraud detection is akin to “finding a needle in a haystack”.  One has to go through mountains of relevant and seemingly irrelevant information, build dependency models, evaluate the impact and thwart the fraudster actions.  Hadoop helps with finding patterns by processing mountains of information on thousands of cores in a relatively short amount of time.

Where to look next?

Techniques for fraud detection are industry-specific as a rule and often are guarded since they obviously represent valuable information for potential fraudsters.  They are often kept confidential for this reason.  Moreover, the fraud detection techniques are usually a moving target since the fraudsters quickly adjust to the new fraud detection mechanisms.

One of the most publicized technical frauds is click fraud in on-line advertising.  Since advertisers are often charged on the per-click basis — so called PPC campaigns; there is a way to charge advertisers on a per-conversion basis, which we will cover shortly, but a different type of fraud emerges there where the advertiser tries to conceal the conversions — the traffic provider like a search web site has a clear incentive to inflate the number.  Additionally, an advertiser competitor may be incentivized to inflate the number to skew the original advertiser margin.  This can be achieved by a human or software agent that generates extra traffic and clicks on the competitor site.  Fraud management companies like Anchor Intelligence and Click Forensics estimate that approximately 20% to 30% of all clicks are fraud.  How do we know that a click is a fraud?

Decline in the number of conversions — first and most important, if your conversion rate is normally positive (that is, you are making a profit on your ad), and all of a sudden, conversion dives into negative numbers, this could be a sign of click fraud in action.  Click fraud causes extra clicks on your ad with no actual purchases, and your conversion rate will fall accordingly.

An abnormal number of clicks from the same IP address or a pattern in the access times — although this is the most obvious and easily identified form of click fraud, it is amazing how many fraudsters still use this method, particularly for quick attacks.  They may choose a to strike over a long weekend when they figure you may not be watching your log files carefully, clicking on your ad repeatedly so that when you return to work on Tuesday, your account is significantly depleted.  Part of this fraud might be unintentional when a user tries to reload a page.

Large “abandonment rate”, or numbers of visitors who leave your site quickly — another indication of click fraud can be a pattern of visitors clicking on your ad, spending the minimum amount of time on your site required by your PPC search engine to establish it as a valid click (usually 30 seconds or more), and then leaving without having left the landing page at all.

A large number of impressions, without the follow-through clicks or click on your ad — if you notice that there are a lot more impressions (views) of your website; this could indicate the impression fraud we discussed earlier. Artificial inflation of your ad impressions may cause your clickthrough rates to drop below the Google minimum, and your ad will be disabled.  Until you realize this, your competitors have free reign to use your keywords, sometimes at bargain prices.  As well, your relevancy ratings for search engines may drop as they record numerous impressions, but no interest shown via visits to other parts of your website, which could lead to a shutdown of your campaign.

Abnormally high clicks and impressions on affiliate websites — although affiliates themselves are sometimes involved in conducting click fraud schemes, they can be victims of click fraud themselves.  If one of their competitors uses this same method of excessive clicks and impressions on an affiliate’s site, the PPC search engine will soon notice an abnormally high payment to a certain affiliate and perhaps go as far as canceling that affiliate’s account, even though he or she was not engaging in any form of click fraud.

A large number of clicks coming from countries outside of your normal market area — using IP geo-location services, you can identify which country an IP address is probably coming from.

In the case of performance-based advertising, the advertiser himself is interested in concealing some of the traffic, not inflating it.  Since most of the performance-based measurements is based in beacons or pixels placed on the advertiser conversion page, advertiser has an incentive to (temporarily) block the traffic from the beacon or to completely remove it from their web-site.

Fraud is prevalent in telecom industry.  One of the leading commercially available fraud detection products is HP FMS system on which the author had a pleasure to work personally.  The types of telecom fraud include:

Subscription fraud — involves the acquisition of telecommunications services using stolen or false credentials and/or identity with no intention of paying. With subscription fraud, not only do service providers lose revenue, but also individual consumers are vulnerable to having their identity stolen and credit rating tarnished.

Technical/network fraud — occurs when someone uses equipment or technology to gain access to a service without paying. Fraudulent calls are typically billed to the legitimate owner of the line or service.  Wireless examples include cloning of cell phones or subscriber identity module (SIM) cards. Fixed line examples include clip on or line tapping, private branch exchange (PBX) hacking and calling card fraud. Prepaid services also have a large exposure to fraud with terminal tampering via magnetic strips or SIM chips, or recharging with stolen credit card numbers.

Insider fraud — occurs when individuals inside the operator provide fraudulent access to networks or otherwise thwart the ability of the operator to be paid for services used.

Handset abuse — is what takes place when stolen or lost handsets are used to consume telecommunications services that are in turn paid for by the service provider.  This is an expensive liability for carriers who absorb the costs.

Social engineering — is an effective fraud technique in which people unwittingly help perpetrators by providing sensitive data, illicit access or simply forwarding their calls without ever knowing they have done anything wrong.

All these patterns can be detected with special MapReduce pattern detection techniques. Flume offers low-latency stream processing capabilities.

Needless to say, the fraudsters also explore the potential market and invent new innovative ways to generate fraud.  One of them is deployed by Click Monkeys which deploys a vessel with animals next to the coast of California to generate seemingly random traffic.

Enter the code “london_10pct” when registering and receive a 10% discount!

Hadoop is a rapidly growing field. Prove your expertise by attaining certification from the world’s foremost Hadoop training and consulting company.

.