In my summer internship project at Cloudera, I added RPC tracing as a first-order feature of Apache Avro. Avro is a platform for data storage and exchange that caters to data-intensive, dynamic applications. My project focused on Avro’s RPC functionality.

It is common knowledge that tracing in distributed systems can be difficult. In user-facing web services, a front-end function may recursively trigger several function calls to mid and back-tier services. In offline processing, data-center storage layers may distribute data across several hosts, querying one or many of them when a client requests a file. In either case, the inter-dependency of components makes it difficult to pinpoint the source of a slowdown or hang-up when they inevitably occur.

What can attendees expect learn about Hadoop from your presentation at Hadoop World?

We’re now able to quantify aspects of human behavior never before accessible. Twitter, the News stream, the Smart Grid, are exquisite lab instruments for measuring ‘Conversation’, ‘Interest’, ‘Activity’. What’s more, with enough data machine-learning algorithms and big data tools let us expose insight using only the *structure*, not the content of the data. The massive quantity and connectivity required demands industrial-strength tools such as Hadoop.

We do *all* our data processing in high level tools (chiefly Pig and Wukong) — “black boxes with flexible glue”. We use ‘programmer fun’ + ‘programmer time’ as our primary development  metrics. Together, writing simple loosely coupled scripts lets us run the fast experiment-driven design cycles that a lean startup demands. It has also let us grow our own talent and recruit outside CS (physicists, in particular, dream in map reduce). I think this approach should have strong appeal to small- and medium-sized businesses, or anyone looking for low barrier-to-adoption of Hadoop.

Do you have Hadoop in production use today?

We have Hadoop in heavy production use for ad-hoc analysis and for automated processes digesting terabytes of data.

Can you describe some use cases for Hadoop in your business?

We have scraped data from around the web, principally Social Networks. We use Hadoop for processing it on its own and to mash it up with other open & commercial datasets.

Examples:

• We have a collection of 3 billion tweets (twitter messages) from 60+million users that we tokenize into 16B+ usages of 65M terms — more than a terabyte of data on its own. Using Pig and Wukong we can identify whom to follow, to understand how events and news stories resonate, and even to find dates.
• MLB has released a dataset describing the trajectory and full game state for every pitch of every game for the past several seasons.  Smashing this against the hourly weather data produces a laboratory able with the potential to describe the physics of a knuckleball or the performance for pitcher’s age vs. game-time temperature.

How do you support Hadoop?

Operationally,  we use the Amazon cloud and a collection of Chef recipes (that we’ve open-sourced). These let us spin up, use, and spin down clusters of one to hundreds of machines, using either local (persistent) HDFS or just push/pull from Amazon S3.

We have also been supporting Hadoop by giving back to the Hadoop open-source community.

• Wukong (our Ruby-language toolkit for Hadoop), which we believe is the easiest and most fun way to write map-reduce programs.
• At Hadoop World we’ll be announcing Chimpmark, a target benchmark for implementers and users of big data tools. It’s a collection of large scale datasets, accompanying challenges, and reference implementations that let you profile, tune and more deeply understand your hadoop system.
• ClusterChef, the cluster management toolkit I described above.

How has Hadoop improved your business?

Most of the stuff we use Hadoop for would be otherwise impossible.

What are you hoping to get out of your time at Hadoop World?

• Learn Ideas.
• Popularize and receive feedback on the development of a data marketplace.
• Hear where the world of Big Data is going.

At Hadoop World you can hear more from Philip Kromer as well as any of the thirty-five other presenters! Click here to register right away!

If you are a manager trying to decide whether Hadoop is an appropriate technology for your organization, Hadoop Essentials for Managers will answer your questions. We will show you when using Hadoop is appropriate, what Hadoop is being used for in a range of industries, how Hadoop fits into your existing environment and what you need to know in order to deploy it within your organization.

Why not turn your Hadoop World trip into a multiple day Hadoop learning extravaganza by attending one of our two-day sessions? Both the developer and administrator training courses culminate in an exam which, when passed, confers Cloudera Certified Hadoop Developer or Administrator status.

For the developer with an existing understanding of Hadoop and ready to utilize Hive and Pig for their data analysis, there is a two-day class teaching you how to process data using filters, joins, user-defined functions and more.

For those looking to deploy HBase, consider our one-day HBase training session. Learn how to use HBase as a distributed data store to achieve low-latency queries and highly scalable throughput. This class covers HBase architecture, data modeling, and the Java API as well as some advanced topics and best practices.

If you’re a developer who is completely new to Hadoop, we have put together a course that will provide you with a solid foundation in large scale data processing using MapReduce and Hadoop. This course is purposely offered the day before Hadoop World, so that while in attendance you will be able to better grasp the topics at the conference with your fresh Hadoop knowledge. Once you have taken this course and are comfortable with Hadoop, feel free to also enroll in a training course followed by Certification to document your new-found Hadoop knowledge.

For developers who wish to simplify interacting with Hadoop, Cloudera HUE provides back- and front-end APIs to deliver a rich, web-based, graphical user experience. This class covers using the HUE APIs to develop your own rich, graphical applications built on top of the HUE platform.

Once again, you will receive free entry to Hadoop World if you are registered in any of the training sessions surrounding the event! Don’t miss out on this opportunity to broaden your knowledge, and we hope to see you there!

GE, Product Manager, Linden Hillenbrand, will be talking about how Hadoop has improved GE’s Marketing & Communications functions.  One capability GE has implemented is assessing the external perception of GE–positive, neutral, or negative–through various marketing campaigns.

Managing Director, Big Data & Analytics at Bank of America, Abhishek Mehta, will  present “The Business of Big Data.” This presentation will discuss how an organization with established and legacy infrastructure, technology and business processes can adopt Hadoop technologies and processes to find groundbreaking solutions to known problems.

eBay Engineering director of Analytical Platform Development, Anil Madan, is presenting “Hadoop at eBay.” One of eBay’s largest assets is the large amount of user data they have collected. By sourcing huge volumes of this data into the HDFS cluster and running click stream and transactional data analysis eBay gets a better understanding of user behavior as well as search quality.

Hadoop World is a great way to learn how Hadoop is being used to power today’s modern enterprises. These presentations will help you understand how Hadoop improves your data storage and processing environment and directly impacts your business.

Don’t miss out! Register now

With such a large delta from the last release, it is difficult to grasp the important new features and changes. This post is intended to give a high-level view of some of the more significant features introduced in the 0.21.0 release. Of course, it can’t hope to cover everything, so please consult the release notes (Common, HDFS, MapReduce) and the change logs (Common, HDFS, MapReduce) for the full details. Also, please let us know in the comments of any features, improvements, or bug fixes that you are excited about.

You can download Hadoop 0.21.0 from an Apache Mirror. Thanks to everyone who contributed to this release!

Project Split

Organizationally, a significant chunk of work has arisen from the project split, which transformed a single Hadoop project (called Core) into three constituents: Common, HDFS, and MapReduce. HDFS and MapReduce both have dependencies on Common, but (other than for running tests) MapReduce has no dependency on HDFS. This separation emphasizes the fact that MapReduce can run on alternative distributed file systems (although HDFS is still the best choice for sheer throughput and scalability), and it has made following development easier since there are now separate lists for each subproject. There is one release tarball still, however, although it is laid out a little differently from previous releases, since it has a subdirectory containing each of the subproject source files.

From a user’s point of view little has changed as a result of the split. The configuration files are divided into core-site.xml, hdfs-site.xml, and mapred-site.xml (this was supported in 0.20 too), and the control scripts are now broken into three (HADOOP-4868): in addition to the bin/hadoop script, there is a bin/hdfs script and a bin/mapreduce script for running HDFS and MapReduce daemons and commands, respectively. The bin/hadoop script still works as before, but issues a deprecation warning. Finally, you will need to set the HADOOP_HOME environment variable to have the scripts work smoothly.

Common

The 0.21.0 release is technically a minor release (traditionally Hadoop 0.x releases have been major, and have been allowed to break compatibility with the previous 0.x-1 release) so it is API compatible with 0.20.2. To make the intended stability and audience of a particular API in Hadoop clear to users, all Java members with public visibility have been marked with classification annotations to say whether they are Public, or Private (there is also LimitedPrivate which signifies another, named, project may use it), and whether they are Stable, Evolving, or Unstable (HADOOP-5073). Only elements marked as Public appear in the user Javadoc (Common, MapReduce; note that HDFS is all marked as private since it is accessed through the FileSystem interface in Common). The classification interface is descibed in detail in Towards Enterprise-Class Compatibility for Apache Hadoop by Sanjay Radia.

This release has seen some significant improvements to testing. The Large-Scale Automated Test Framework, known as Herriot (HADOOP-6332), allows developers to write tests that run against a real (possibly large) cluster. While there are only a dozen or so tests at the moment, the intention is that more tests will be written over time so that regression tests can be shared and run against new Hadoop release candidates, thereby making Hadoop upgrades more predictable for users.

Hadoop 0.21 also introduces a fault injection framework, which uses AOP to inject faults into a part of the system that is running under test (e.g. a datanode), and asserts that the system reacts to the fault in the expected manner. Complementing fault injection is mock object testing, which tests code “in the small”, at the class-level rather than the system-level. Hadoop has a growing number of Mockito-based tests for this purpose (MAPREDUCE-1050).

Among the many other improvements and new features, a couple of small ones stand out: the ability to retrieve metrics and configuration from Hadoop daemons by accessing the URLs /metrics and /conf in a browser (HADOOP-5469, HADOOP-6408).

HDFS

Support for appends in HDFS has had a rocky history. The feature was introduced in the 0.19.0 release, and then disabled in 0.19.1 due to stability issues. The good news is that the append call is back in 0.21.0 with a brand new implementation (HDFS-265), and may be accessed via FileSystem’s append() method. Closely related—and more interesting for many applications, such as HBase—is the Syncable interface that FSDataOutputStream now implements, which brings sync semantics to HDFS (HADOOP-6313).

Hadoop 0.21 has a new filesystem API, called FileContext, which makes it easier for applications to work with multiple filesystems (HADOOP-4952). The API is not in widespread use yet (e.g. it is not integrated with MapReduce), but it has some features that the old FileSystem interface doesn’t, notably support for symbolic links (HADOOP-6421, HDFS-245).

The secondary namenode has been deprecated in 0.21. Instead you should consider running a checkpoint node (which essentially acts like a secondary namenode) or a backup node (HADOOP-4539). By using a backup node you no longer need an NFS-mount for namenode metadata, since it accepts a stream of filesystem edits from the namenode, which it writes to disk.

New in 0.21 is the offline image viewer (oiv) for HDFS image files (HADOOP-5467). This tool allows admins to analyze HDFS metadata without impacting the namenode (it also works with older versions of HDFS). There is also a block forensics tool for finding corrupt and missing blocks from the HDFS logs (HDFS-567).

Modularization continues in the platform with the introduction of pluggable block placement (HDFS-385), an expert-level interface for developers who want to try out new placement algorithms for HDFS.

Other notable new features include:

• Support for efficient file concatenation in HDFS (HDFS-222)
• Distributed RAID filesystem (HDFS-503) – an erasure coding filesystem running on HDFS, designed for archival storage since the replication factor is reduced from 3 to 2, while keeping the likelihood of data loss about the same. (Note that the RAID code is a MapReduce contrib module since it has a dependency on MapReduce for generating parity blocks.)

MapReduce

The biggest user-facing change in MapReduce is the status of the new API, sometimes called “context objects”. The new API is now more broadly supported since the MapReduce libraries (in org.apache.hadoop.mapreduce.lib) have been ported to use it (MAPREDUCE-334). The examples all use the new API too (MAPREDUCE-271). Nevertheless, to give users more time to migrate to the new API, the old API has been un-deprecated in this release (MAPREDUCE-1735), which means that existing programs will compile without deprecation warnings.

The LocalJobRunner (for trying out MapReduce programs on small local datasets) has been enhanced to make it more like running MapReduce on a cluster. It now supports the distributed cache (MAPREDUCE-476), and can run mappers in parallel (MAPREDUCE-1367).

Distcp has seen a number of small improvements too, such as preserving file modification times (HADOOP-5620), input file globbing (HADOOP-5472), and preserving the source path (MAPREDUCE-642).

Continuing the testing theme, this release is the first to feature MRUnit, a contrib module that helps users write unit tests for their MapReduce jobs (HADOOP-5518).

Other new contrib modules include Rumen (MAPREDUCE-751) and Mumak (MAPREDUCE-728), tools for modelling MapReduce. The two are designed to work together: Rumen extracts job data from historical logs, which Mumak then uses to simulate MapReduce applications and clusters on a cluster. Gridmix3 is also designed to work with Rumen traces. The job history log analyzer is another tool that gives information about MapReduce cluster utilization (HDFS-459).

On the job scheduling front there have been updates to the Fair Scheduler, including global scheduling (MAPREDUCE-548), preemption (MAPREDUCE-551), and support for FIFO pools (MAPREDUCE-706). Similarly, the Capacity Scheduler now supports hierarchical queues (MAPREDUCE-824), and admin-defined hard limits (MAPREDUCE-532). There is also a brand new scheduler, the Dynamic Priority Scheduler, which dynamically changes queue shares using a pricing model (HADOOP-4768).

Smarter speculative execution has been added to all schedulers using a more robust algorithm, called Longest Approximate Time to End (LATE) (HADOOP-2141).

Finally, a couple of smaller changes:

• Streaming combiners are now supported, so that the -combiner option may specify any streaming script or executable, not just a Java class. (HADOOP-4842)
• On the successful completion of a job, the MapReduce runtime creates a _SUCCESS file in the output directory. This may be useful for applications that need to see if a result set is complete just by inspecting HDFS. (MAPREDUCE-947)

What’s Not In

Finally, it bears mentioning what didn’t make it into 0.21.0. The biggest omission is the new Kerberos authentication work from Yahoo! While a majority of the patches are included, security is turned off by default, and is unlikely to work if enabled (certainly there is no guarantee that it will provide any level of security, since it is incomplete). A full working security implementation will be available in 0.22, and also the next version of CDH.

Also, Sqoop, which was initially developed as a Hadoop contrib module, is not in 0.21.0, since it was moved out to become a standalone open source project hosted on github.

From the statistical or technical perspective, fraud is a rare event that results in a significant financial impact to the organization.

Both definitions emphasize that the event is rare (assuming that most of the population is law-abiding citizens), is intentional (there is no “accidental” fraud), as well as imply a significant damage caused to the defrauded party (otherwise why bother).  Fraud detection is difficult from statistical point of view for exactly these reasons: (a) the events are rare and it is difficult to build a predictive model and (b) fraud assumes a real human being behind it and incorporates elements of game theory since the fraudster is often an insider who knows how to game the system.

Fraud and Rare Events

By definition, fraud is an unexpected or rare event with significant financial or other damage.  Fraud assumes that the fraudster has some prior information how the current system works including previous successful and unsuccessful fraud cases and possibly the fraud detection mechanisms.  The above breaks the standard statistical modeling assumption, the variable independence or i.i.d. assumption, making building a reliable statistical model difficult.  Often the fraudster is working in the same industry that the fraud detection is supposed to protect, is intimately familiar with the fraud detection methods, and is actively trying to avoid detection by masquerading.

Rare event detection problem is also applicable to online advertising and marketing, particularly with predicting “long tail” events and terrorism detection.

One common example of fraud is associated with Taleb distribution where a seemingly high probability of a small gain shadows a small probability of a large loss that more than outweighs the gains.  Relatively long periods of slightly better than moderate gains are interrupted by a rare event of large losses.  It is easy to defraud investors by presenting the results of partial analysis excluding the “rare events”.

Fraud Prevention

Since fraud is so hard to prove in courts, most organizations and individuals try to prevent fraud from happening by blanket measures.  This includes limiting the amount of damage the fraudster can impact on the organization as well as early detection of fraud patterns.  For example, credit card companies can cut the credit card limit across the board in anticipation of a few negative fraud cases.  Advertisers can prevent advertising campaigns with low number of qualifying events.  And anti-terrorism agencies can prevent people with bottles of pure water from boarding the planes.  These actions are often in contrast with the company efforts to attract more customers and result in general dissatisfaction.  To the rescue are new technologies like Hadoop, Influence Diagrams and Bayesian Networks which are computationally expensive (these are NP-hard in computer science terminology) but are more accurate and predictive.

Why Hadoop?

Hadoop is a distributed system for processing large amounts of data.  In a recent Hadoop Summit 2010 Yahoo, Facebook, and other companies announced that they currently process a few TBs of data per day and the volumes are growing at exponential rates.  Hadoop can be vital for solving the fraud detection problem because:

1. Sampling does not work for rare events since the chance of missing a positive fraud case leads to significant deterioration of model quality.
2. Hadoop can solve much harder problems by leveraging multiple cores across thousands of machines and search through much larger problem domains.
3. Hadoop can be combined with other tools to manage moderate to low response latency requirements.

Let’s go through these reasons one by one.  Sampling is a common technique for modeling rare events.  One of the problems with sampling is that we cannot afford to throw away rare positive cases.  Even in a stratified or proportional sampling scheme one has to retain all positive cases since the model accuracy heavily depends on them (one can usually discard some negative cases though).  Given the above, the system still has to go through the whole dataset to sieve through the positive and negative cases.

Hadoop is known for its gnawing power.  Nothing can compare with the throughput power of thousands of machines each of which has multiple cores.  As was reported recently at the Hadoop Summit 2010, the largest installations of Hadoop have 2,000 to 4,000 computers with 8 to 12 cores each, amounting to up to 48,000 active threads looking for a pattern at the same time.  This allows either (a) looking through larger periods of time to incorporate events across a larger time frame or (b) taking more sources of information into account.  It is quite common among social network companies to comb through twitter blogs in search of relevant data.

Finally, one of the fraud prevention problems is latency.  The agencies want to react to an event as soon as possible, often within a few minutes of the event.  Yahoo recently reported that it can adjust its behavioral model in a response to a user click event within 5-7 minutes across several hundred of millions of customers and billions of events per day.  Cloudera has developed a tool, Flume, that can load billions of events into HDFS within a few seconds and analyze them using MapReduce.

Often fraud detection is akin to “finding a needle in a haystack”.  One has to go through mountains of relevant and seemingly irrelevant information, build dependency models, evaluate the impact and thwart the fraudster actions.  Hadoop helps with finding patterns by processing mountains of information on thousands of cores in a relatively short amount of time.

Where to look next?

Techniques for fraud detection are industry-specific as a rule and often are guarded since they obviously represent valuable information for potential fraudsters.  They are often kept confidential for this reason.  Moreover, the fraud detection techniques are usually a moving target since the fraudsters quickly adjust to the new fraud detection mechanisms.

One of the most publicized technical frauds is click fraud in on-line advertising.  Since advertisers are often charged on the per-click basis — so called PPC campaigns; there is a way to charge advertisers on a per-conversion basis, which we will cover shortly, but a different type of fraud emerges there where the advertiser tries to conceal the conversions — the traffic provider like a search web site has a clear incentive to inflate the number.  Additionally, an advertiser competitor may be incentivized to inflate the number to skew the original advertiser margin.  This can be achieved by a human or software agent that generates extra traffic and clicks on the competitor site.  Fraud management companies like Anchor Intelligence and Click Forensics estimate that approximately 20% to 30% of all clicks are fraud.  How do we know that a click is a fraud?

Decline in the number of conversions — first and most important, if your conversion rate is normally positive (that is, you are making a profit on your ad), and all of a sudden, conversion dives into negative numbers, this could be a sign of click fraud in action.  Click fraud causes extra clicks on your ad with no actual purchases, and your conversion rate will fall accordingly.

An abnormal number of clicks from the same IP address or a pattern in the access times — although this is the most obvious and easily identified form of click fraud, it is amazing how many fraudsters still use this method, particularly for quick attacks.  They may choose a to strike over a long weekend when they figure you may not be watching your log files carefully, clicking on your ad repeatedly so that when you return to work on Tuesday, your account is significantly depleted.  Part of this fraud might be unintentional when a user tries to reload a page.

Large “abandonment rate”, or numbers of visitors who leave your site quickly — another indication of click fraud can be a pattern of visitors clicking on your ad, spending the minimum amount of time on your site required by your PPC search engine to establish it as a valid click (usually 30 seconds or more), and then leaving without having left the landing page at all.

A large number of impressions, without the follow-through clicks or click on your ad — if you notice that there are a lot more impressions (views) of your website; this could indicate the impression fraud we discussed earlier. Artificial inflation of your ad impressions may cause your clickthrough rates to drop below the Google minimum, and your ad will be disabled.  Until you realize this, your competitors have free reign to use your keywords, sometimes at bargain prices.  As well, your relevancy ratings for search engines may drop as they record numerous impressions, but no interest shown via visits to other parts of your website, which could lead to a shutdown of your campaign.

Abnormally high clicks and impressions on affiliate websites — although affiliates themselves are sometimes involved in conducting click fraud schemes, they can be victims of click fraud themselves.  If one of their competitors uses this same method of excessive clicks and impressions on an affiliate’s site, the PPC search engine will soon notice an abnormally high payment to a certain affiliate and perhaps go as far as canceling that affiliate’s account, even though he or she was not engaging in any form of click fraud.

A large number of clicks coming from countries outside of your normal market area — using IP geo-location services, you can identify which country an IP address is probably coming from.

In the case of performance-based advertising, the advertiser himself is interested in concealing some of the traffic, not inflating it.  Since most of the performance-based measurements is based in beacons or pixels placed on the advertiser conversion page, advertiser has an incentive to (temporarily) block the traffic from the beacon or to completely remove it from their web-site.

Fraud is prevalent in telecom industry.  One of the leading commercially available fraud detection products is HP FMS system on which the author had a pleasure to work personally.  The types of telecom fraud include:

Subscription fraud — involves the acquisition of telecommunications services using stolen or false credentials and/or identity with no intention of paying. With subscription fraud, not only do service providers lose revenue, but also individual consumers are vulnerable to having their identity stolen and credit rating tarnished.

Technical/network fraud — occurs when someone uses equipment or technology to gain access to a service without paying. Fraudulent calls are typically billed to the legitimate owner of the line or service.  Wireless examples include cloning of cell phones or subscriber identity module (SIM) cards. Fixed line examples include clip on or line tapping, private branch exchange (PBX) hacking and calling card fraud. Prepaid services also have a large exposure to fraud with terminal tampering via magnetic strips or SIM chips, or recharging with stolen credit card numbers.

Insider fraud — occurs when individuals inside the operator provide fraudulent access to networks or otherwise thwart the ability of the operator to be paid for services used.

Handset abuse — is what takes place when stolen or lost handsets are used to consume telecommunications services that are in turn paid for by the service provider.  This is an expensive liability for carriers who absorb the costs.

Social engineering — is an effective fraud technique in which people unwittingly help perpetrators by providing sensitive data, illicit access or simply forwarding their calls without ever knowing they have done anything wrong.

All these patterns can be detected with special MapReduce pattern detection techniques. Flume offers low-latency stream processing capabilities.

Needless to say, the fraudsters also explore the potential market and invent new innovative ways to generate fraud.  One of them is deployed by Click Monkeys which deploys a vessel with animals next to the coast of California to generate seemingly random traffic.

Enter the code “london_10pct” when registering and receive a 10% discount!

Hadoop is a rapidly growing field. Prove your expertise by attaining certification from the world’s foremost Hadoop training and consulting company.

.

Orbitz Worldwide (NYSE:OWW) is composed of a global portfolio of online consumer travel brands including Orbitz, Cheaptickets, The Away Network, ebookers and HotelClub, Additionally, the company operates business-to-business service: Orbitz Worldwide Distribution provides third parties such as Amtrak, Delta, LAN, KLM, Air France and a number of other leading airlines hotel booking capabilities, and Orbitz for Business provides corporate travel services to a number of Fortune 100 clients. The Orbitz Worldwide sites process millions of searches and transactions every day, which not surprisingly results in hundreds of gigabytes of log data per day. Not all of that data necessarily has value, but much of it does. Unfortunately storing and processing all of that data in our existing data warehouse infrastructure is impractical because of expense and space considerations.

Hadoop was selected to provide a solution to the problem of long-term storage and processing of these large quantities of un-structured and semi-structured data. We deployed our first Hadoop clusters in late 2009 running Cloudera’s Distribution for Hadoop (CDH), and in early 2010 deployed Hive to provide structure and SQL-like access to Hadoop data. In the short period of time since our initial deployment we’ve seen Hadoop rapidly adopted as a component in a wide range of applications across the organization due to its power, ease of use, and suitability for solving big data problems.

One of the applications that Hadoop facilitates is an effort to improve the hotel search results. Currently, when a user performs a hotel search on the Orbitz site the ranking of the search results returned (at least for larger markets) is influenced by a set of parameters manually tuned by an administrator. This leads to the question: can we use automation to optimize the ranking of hotels in order to increase bookings? In other words, can we identify consumer preferences in order to determine the best performing hotels to display to users, thus leading to more bookings? Further, for markets that are too small to be manually managed, can we implement a method to automatically rank hotel search results?

To answer this question, it was decided to turn to machine learning techniques, specifically using a trained classifier to determine a ranking of hotels that more closely follows consumer preferences. Performing this analysis requires having data on consumer interactions when shopping for hotels. Fortunately, we have a rich source of this session data in web analytics logs that are collected as users browse the sites. Unfortunately, although parts of this data are loaded into the data warehouse, it turned out that the specific fields we require are not loaded because of space restrictions. Our only alternative was to turn to the raw logs to extract the required fields. Just to further complicate things, the available archive of these logs only went back several days – not nearly enough data to perform the required analysis.

Hadoop of course provided a solution to the storage problem by providing a repository where we could download and archive logs. The next step was to extract the data we needed from the raw logs. We began with a set of shell and Perl scripts that were run manually to serially process logs on the local file system. This process worked fine for a while, but as the size of the data grew it was obvious that this process wouldn’t scale. Once again Hadoop provided a solution. Since we were already storing the logs in HDFS, by moving the most time-consuming portions of the data extraction into MapReduce, we were able to dramatically decrease processing time.  A test run against a small subset of data showed a greater than four time improvement for the MapReduce processing vs. the scripts. Now that we’ve accumulated several terabytes of data the performance disparity would be even more dramatic, assuming we even had access to a storage system large enough to hold all of the data for manual processing.

After the data is extracted through MapReduce, we load the resulting records into a set of Hive tables. Hive allows us to perform ad hoc querying and further analysis of this data, such as:

• Obtaining useful metrics, many of which were unavailable with our existing data stores.
• Creating data exports for further analysis with R scripts, allowing us to derive more complex statistics and visualizations of our data.
• Aggregating data for import into our data warehouse for creation of new data cubes, providing analysts access to data unavailable in existing data cubes.

 In addition to assisting with hotel rank optimization, a few examples of other ways Hadoop is being applied at Orbitz Worldwide are:

• Measuring page download performance: using web analytics logs as input, a set of MapReduce scripts are used to derive detailed client side performance metrics which allow us to track trends in page download times.
• Searching production logs: an effort is underway to utilize Hadoop to store and process our large volume of production logs, allowing developers and analysts to perform tasks such as troubleshooting production issues.
• Data aggregation for the data warehouse: further exploration is being done to expand the use of Hadoop and Hive as a means to aggregate previously unavailable data for import into our data warehouse, making it available for access by our existing data analysis tools.
• Cache analysis: extraction and aggregation of data to provide input to analyses intended to improve the performance of data caches utilized by our web sites.

 Again, these are just a few examples of how Hadoop is being utilized at Orbitz Worldwide, and we’re still just scratching the surface. Each week seems to bring a new team with a big data challenge to be solved by Hadoop, a trend which I expect to continue as more teams discover the possibilities that Hadoop provides to store and process data.

I’d like to thank my co-workers who have all made significant contributions to the work discussed here, including Rob Lancaster, Ramesh Venkataramaiah, Wai Gen Yee, Steve Hoffman, Matt Haddock and Andrew Yates.  Also a big thanks to Vice President of Technology Roger Liew, who was an early and enthusiastic champion of Hadoop.

We are also offering training sessions for newcomers and experienced Hadoop users alike. Whether you are looking for an Introduction to Hadoop, Hadoop Certification, or you want to learn more about related Hadoop projects we have the training you are looking for.

Included with our top-notch Hadoop training you will have full access to Hadoop World free of charge.

Available Training Sessions include:

Oct 11:

Introduction to Hadoop: http://www.eventbrite.com/event/762326138

This one-day course provides a solid foundation for those seeking to understand large scale data processing with MapReduce and Hadoop. This session is designed for developers, analysts or system administrators that are new to Hadoop. This course provides the pre-requisite knowledge for the later classes: Developer Training, Administrator Training or Analyzing Data with Hive and Pig.

Hadoop Essentials For Managers: http://www.eventbrite.com/event/762237874

This one-day course will give decision-makers the information they need to know about Apache Hadoop, answering questions such as:

• When is Hadoop appropriate?
• What are people using Hadoop for?
• How does Hadoop fit into our existing environment?
• What do I need to know about choosing Hadoop?

Cloudera HUE SDK Training: http://www.eventbrite.com/event/764021208

Cloudera Hue provides developers with back end APIs to simplify interacting with Hadoop and front end APIs to deliver rich, web based, graphical user experiences. For this training, developers should have experience building web apps using modern MVC frameworks and Ajax. Experience with Python and Django is a strong plus. In this session we spend half the day covering the following topics, and the other half of the day interactively building applications with the Cloudera Hue team.

Oct 13 & 14:

Developer Training & Certification: http://www.eventbrite.com/event/762320120

In this two-day hands-on session, developers learn the MapReduce framework and how to write programs against its API. In addition to learning how to write individual MapReduce jobs, we discuss design techniques for larger workflows. This course also covers advanced skills for debugging MapReduce programs and optimizing their performance. At the end of the course, attendees have the option to take a certification exam documenting their understanding of the concepts taught during the training session.

Administrator Training & Certification: http://www.eventbrite.com/event/762677188

This two-day hands-on session covers the system administration aspects of Hadoop from installation and configuration to load balancing and tuning including diagnosing and solving problems in your deployment. At the end of the course, attendees have the option of taking a certification exam documenting their understanding of the concepts taught at the training session.

Analyzing Data with Hive and Pig: http://www.eventbrite.com/event/762318114

Cloudera’s two-day hands-on course on Hive and Pig is designed for people who have a basic understanding of how Hadoop works and want to utilize these languages for analysis of their data. Hive makes Hadoop accessible to users who already know SQL; Pig is similar to popular scripting languages. This course teachs you how to process data by using filters, joins, user-defined functions and more.

Oct 15:

HBase Training: http://www.eventbrite.com/event/762317111

This one-day hands-on course gives you the necessary knowledge for using HBase as a distributed data store to achieve low-latency queries and highly scalable throughput. This class covers the HBase architecture, data model, and Java API as well as advanced topics and best practices. This course is for developers who already have a basic understanding of Hadoop (Java experience is recommended).

Network

Memory

Disk

CPU

Summary