Flume v0.9.1

Flume v0.9.1 is now available both in tarball and packaged forms. This version resolves 63 issues and contains several key improvements and bugs fixes. Much of this release is focused on improving the stability of Flume’s internals to help users quickly get Flume up and running and to help developers build extensions to Flume.

You can download the new release as an update to your Redhat RPM or Debian DEB based package managers. Or, you can download it in tarball form from Cloudera’s archive, or as always from the Cloudera’s github repository .

The key functional highlights include:

• Support for gzip compressed output files.
• New and improved sources: scribe, syslog, tailDir (tail all files in a directory)
• Significant robustness improvements when using in the disk fail-over and end-to-end reliability modes.
• Significant robustness improvements when reconfiguring, commissioning, and decomissioning logical nodes.

To improve the documentation and enhance debugging support, we have added:

• A new section of the manual that explains how to build your own flume source, sink, and decorator plugins by example.
• An ‘ant eclipse’ option to automatically build project files for developing in the Eclipse IDE.
• Improved error messages in logs, exposed Flume internals such as current configuration properties, and source/sink catalogs to ease operator and developer debugging and verification.

For more details, read the full release notes.

Up and coming Flume features

One of Flume’s key design principles is extensibility. We are happy people are taking advantage of this to integrate Flume with other systems. Some new features currently being developed will enable the next release of Flume to have greater integration with CDH’s core components as well as other systems in the Hadoop ecosystem.

Here are some of the new major contributions near completion or actively in the works:

• Flume + Hive integration plugin. Mozilla’s Anurag Phadke has been working with Cloudera’s Carl Steinbach to automatically import data ingested by Flume into Hive warehouses.
• Flume + HBase integration plugin. Several guests at the recent Cloudera Hackathon improved upon our initial Flume/HBase connector and posted it so the community could continue improving it. Since then, a more generic design was proposed and Cloudera’s new intern, Dani Rayan, has volunteered to implement it.
• Flume + Cassandra integration plugin. Tyler Hobbs contributed a first version of this plugin.  It is blocked by some Thrift compatibility and dependency issues.
• Secured data transport via TLS. Kim Vogt and Ben Standefer from SimpleGeo, with some feedback from David Zuelke of Bitextender have been working on adding TLS-based wire encryption to the RPC sources and sinks to provide secure data center communications.
• Flume + Kerberized HDFS integration. Flume takes its first steps to support the newer versions of HDFS that require Kerberos authentication in order to read from and write to HDFS.
• Generic compression codec support for output files. This enables users to choose from all of the codecs Hadoop supports: gzip, bzip2, and deflate.  It should also enable the LZO codec with a little extra work.
• Documentation improvements galore. Currently in the works are a semantics specification for sources and sinks, and step-by-step instructions for connecting Flume to common sources such as  Apache web servers, syslog, and existing scribe loggers.

Community

We are really grateful to the folks who have been exploring and talking about the project!  The guests (Dustin Sallings of NorthScale and Ron Bodkin among others…) who tried out Flume at the Cloudera’s Hackathon day gave us valuable feedback.  In the past month, Cloudera’s Henry Robinson presented “Inside Flume” at Hadoop Day in Seattle.   It is also great to see that some folks are slated to present at Hadoop World 2010 about integrating and using Flume.  Otis Gospodnetic of Sematext will be talking about analytics with Flume and HBase. Also, Anurag Phadke from Mozilla will be presenting a talk about of his experiences integrating Flume-collected data automatically into Hive. He recently posted some details in his blog.

It is great to see the community growing and we love hearing from all of you as well! It has been two months since Flume was open sourced, and our main github repository now has 136 watchers and 24 forks.  Our user mailing list has 102 members and our developers mailing list has 41 members. Please join us! If you are using Flume and want to keep up with where it is going, join the mailing lists and follow us on Twitter at @cloudera and #flume.  If you need help, just send questions to the mailing lists or chat with us directly in IRC on channel #flume at irc.freenode.net.  To meet the Flume Team and contributors in person, you should join us in New York City at Hadoop World on October 12th!

Hadoop World 2010 presenter Saptarshi Guha works in the Department of Statistics at Purdue University. His presentation for Hadoop World is titled “Using R and Hadoop to Analyze VoIP Network Data for QoS.” Guha has been developing with Hadoop and R for over a year.

Q: What can attendees expect learn about Hadoop from your presentation at Hadoop World?

The quality of VoIP calls are suspect to the queuing effects introduced by the network gateways. The jitter between two consecutive packets is the deviation of the real inter-arrival time from theoretical.

We use the R environment for the statistical analysis of data to show jitter follows desired properties and is negligible, which demonstrates that the measured traffic is close to the offered traffic.  Data sets used to study the departure from offered load can be massive and require detailed study of several complex data structures. Using an environment that integrates R and Hadoop, we hope to demonstrate the effectiveness of R and Hadoop for the comprehensive statistical analyses of massive data sets.

Q: Describe use cases for Hadoop at Purdue.

Our team works with large amounts of network traffic data collected for VoIP and network security projects. Our language of analysis is almost exclusively R and we need a way to store the 190 gigabytes of VoIP related data, create data structures for analysis and compute across these. The R and Hadoop combination allows us to do all of this in a manner that scales with the size of the data and returns results within acceptable time frames. Despite not having HBase installed, we use Hadoop map files and R to query data structures from a database of 14 million objects spanning 21GB within seconds.

Q: What benefits do you see from Hadoop?

The biggest win is the reduction in computing time, the ease of programming in the R and Hadoop environment and the Hadoop Distributed Filesystem. We have stopped worrying about disk space and freely store as many databases of objects as required. It must be mentioned, that Hadoop DFS and MapReduce are both very easy to setup and return very impressive results. For our approach to analysis, the Hadoop MapReduce paradigm fits very well. We partition the data into many subsets (usually by the levels of categorical variables), compute across these and recombine the results. We also visualize a subset of these and recombine the results in to multi panel multi page displays, which are viewed across large 30″ monitors.

Q: What did you use before Hadoop?

Some of the things we have done were impossible without Hadoop. Before this we used a tree hierarchy of directories of flat files containing R objects and index files to locate objects with in these flat files. Distributing computation across our cluster was a laborious, manual and very project specific affair but now using the R and Hadoop system the we have sufficiently abstracted the workflow to span a multitude of data sets.

Q: How has Hadoop improved your work at Purdue?

Hadoop has certainly improved our workflow, allowing the researchers to think about studying the data rather than how to distribute code and data, how to maintain a cluster, or how to tackle tedious but vital things such as computer failure. Because the time to compute is substantially less the researchers have the flexibility to implement their ideas and interactively analyze the data. We hope to increase our cluster size and bring more people into the fold.

Q: What are you hoping to get out of your time at Hadoop World?

To demonstrate that it is indeed possible to comprehensively analyze gigabytes of data with a level of detail that was only possible with small data sets and to learn of new Hadoop related technologies that might benefit our workflow.

Hear more from Guha at Hadoop World in New York! For more information regarding the conference its-self click here.

You will learn everything you need to know about migrating with CDH3b2 ranging from why migrate to testing.

Flume community update - August 3

In this blog we address Flume issues, talk about new features, and the improvement of the platform.

Hadoop World: early-bird rate ends on August 11 – August 9

The early-bird registration window may have passed, however, it is not too late to register for Hadoop World. Register Now!

Cloudera’s Henry Robinson to speak at Hadoop Day in Seattle - August 10

At Seattle’s Hadoop Day Cloudera’s Henry Robinson gave a speech entitled “Inside Flume.” Click the above title to read a short abstract, and the slides of his presentation are available by clicking here.

CDH3b2 Release Recap – August 11

This post briefly explains CDH3b2 and contains links to the components of this package.

Avoiding Common Hadoop Administration Issues – August 12

Cloudera’s support services see many issues on a regular basis. This post runs through some common administration issues we have come across and how to avoid them.

Hadoop/HBase Capacity Planning – August 17

This blog provides guidance in sizing your first Hadoop/HBase cluster.

Hadoop World: NYC – Training – August 19

This post contains brief summaries of the trainings available surrounding Hadoop World. Space is still available, sign up now!

Improving Hotel Search: Hadoop @ Orbitz Worldwide – August 23

Jonathan Seidman provides a use-case from Orbitz, which will be further covered at Hadoop World on October 12th.

Hadoop Administrator Training Comes to London – August 24

There will be Hadoop Training in London for Administrators and Developers alike. These sessions are September 9th, so sign up ASAP, use the promotion code in this post for a discount!

Using Hadoop for Fraud Detection and Prevention – August 24

Hadoop has been proven very useful in fraud detection and prevention. This post covers how Hadoop can help solve these problems.

What’s New in Apache Hadoop 0.21 – August 26

Tom White, author of Hadoop: The Definitive Guide gives us an overview of the changes and improvements with Apache Hadoop 0.21.

Hadoop World 2010: Speaker Highlights – August 30

A brief glimpse into three of the thirty-six presentations that will be given at Hadoop World. This blog has presentation abstracts for General Electric, Bank of America, and eBay.

This concludes the summary of Cloudera blog posts for August 2010. Be sure to follow Cloudera as we will continue to provide updates on Apache Hadoop and Hadoop-related projects.

Be sure to attend Hadoop World October 12th in New York City!

In my summer internship project at Cloudera, I added RPC tracing as a first-order feature of Apache Avro. Avro is a platform for data storage and exchange that caters to data-intensive, dynamic applications. My project focused on Avro’s RPC functionality.

It is common knowledge that tracing in distributed systems can be difficult. In user-facing web services, a front-end function may recursively trigger several function calls to mid and back-tier services. In offline processing, data-center storage layers may distribute data across several hosts, querying one or many of them when a client requests a file. In either case, the inter-dependency of components makes it difficult to pinpoint the source of a slowdown or hang-up when they inevitably occur.

What can attendees expect learn about Hadoop from your presentation at Hadoop World?

We’re now able to quantify aspects of human behavior never before accessible. Twitter, the News stream, the Smart Grid, are exquisite lab instruments for measuring ‘Conversation’, ‘Interest’, ‘Activity’. What’s more, with enough data machine-learning algorithms and big data tools let us expose insight using only the *structure*, not the content of the data. The massive quantity and connectivity required demands industrial-strength tools such as Hadoop.

We do *all* our data processing in high level tools (chiefly Pig and Wukong) — “black boxes with flexible glue”. We use ‘programmer fun’ + ‘programmer time’ as our primary development  metrics. Together, writing simple loosely coupled scripts lets us run the fast experiment-driven design cycles that a lean startup demands. It has also let us grow our own talent and recruit outside CS (physicists, in particular, dream in map reduce). I think this approach should have strong appeal to small- and medium-sized businesses, or anyone looking for low barrier-to-adoption of Hadoop.

Do you have Hadoop in production use today?

We have Hadoop in heavy production use for ad-hoc analysis and for automated processes digesting terabytes of data.

Can you describe some use cases for Hadoop in your business?

We have scraped data from around the web, principally Social Networks. We use Hadoop for processing it on its own and to mash it up with other open & commercial datasets.

Examples:

• We have a collection of 3 billion tweets (twitter messages) from 60+million users that we tokenize into 16B+ usages of 65M terms — more than a terabyte of data on its own. Using Pig and Wukong we can identify whom to follow, to understand how events and news stories resonate, and even to find dates.
• MLB has released a dataset describing the trajectory and full game state for every pitch of every game for the past several seasons.  Smashing this against the hourly weather data produces a laboratory able with the potential to describe the physics of a knuckleball or the performance for pitcher’s age vs. game-time temperature.

How do you support Hadoop?

Operationally,  we use the Amazon cloud and a collection of Chef recipes (that we’ve open-sourced). These let us spin up, use, and spin down clusters of one to hundreds of machines, using either local (persistent) HDFS or just push/pull from Amazon S3.

We have also been supporting Hadoop by giving back to the Hadoop open-source community.

• Wukong (our Ruby-language toolkit for Hadoop), which we believe is the easiest and most fun way to write map-reduce programs.
• At Hadoop World we’ll be announcing Chimpmark, a target benchmark for implementers and users of big data tools. It’s a collection of large scale datasets, accompanying challenges, and reference implementations that let you profile, tune and more deeply understand your hadoop system.
• ClusterChef, the cluster management toolkit I described above.

How has Hadoop improved your business?

Most of the stuff we use Hadoop for would be otherwise impossible.

What are you hoping to get out of your time at Hadoop World?

• Learn Ideas.
• Popularize and receive feedback on the development of a data marketplace.
• Hear where the world of Big Data is going.

At Hadoop World you can hear more from Philip Kromer as well as any of the thirty-five other presenters! Click here to register right away!

If you are a manager trying to decide whether Hadoop is an appropriate technology for your organization, Hadoop Essentials for Managers will answer your questions. We will show you when using Hadoop is appropriate, what Hadoop is being used for in a range of industries, how Hadoop fits into your existing environment and what you need to know in order to deploy it within your organization.

Why not turn your Hadoop World trip into a multiple day Hadoop learning extravaganza by attending one of our two-day sessions? Both the developer and administrator training courses culminate in an exam which, when passed, confers Cloudera Certified Hadoop Developer or Administrator status.

For the developer with an existing understanding of Hadoop and ready to utilize Hive and Pig for their data analysis, there is a two-day class teaching you how to process data using filters, joins, user-defined functions and more.

For those looking to deploy HBase, consider our one-day HBase training session. Learn how to use HBase as a distributed data store to achieve low-latency queries and highly scalable throughput. This class covers HBase architecture, data modeling, and the Java API as well as some advanced topics and best practices.

If you’re a developer who is completely new to Hadoop, we have put together a course that will provide you with a solid foundation in large scale data processing using MapReduce and Hadoop. This course is purposely offered the day before Hadoop World, so that while in attendance you will be able to better grasp the topics at the conference with your fresh Hadoop knowledge. Once you have taken this course and are comfortable with Hadoop, feel free to also enroll in a training course followed by Certification to document your new-found Hadoop knowledge.

For developers who wish to simplify interacting with Hadoop, Cloudera HUE provides back- and front-end APIs to deliver a rich, web-based, graphical user experience. This class covers using the HUE APIs to develop your own rich, graphical applications built on top of the HUE platform.

Once again, you will receive free entry to Hadoop World if you are registered in any of the training sessions surrounding the event! Don’t miss out on this opportunity to broaden your knowledge, and we hope to see you there!

GE, Product Manager, Linden Hillenbrand, will be talking about how Hadoop has improved GE’s Marketing & Communications functions.  One capability GE has implemented is assessing the external perception of GE–positive, neutral, or negative–through various marketing campaigns.

Managing Director, Big Data & Analytics at Bank of America, Abhishek Mehta, will  present “The Business of Big Data.” This presentation will discuss how an organization with established and legacy infrastructure, technology and business processes can adopt Hadoop technologies and processes to find groundbreaking solutions to known problems.

eBay Engineering director of Analytical Platform Development, Anil Madan, is presenting “Hadoop at eBay.” One of eBay’s largest assets is the large amount of user data they have collected. By sourcing huge volumes of this data into the HDFS cluster and running click stream and transactional data analysis eBay gets a better understanding of user behavior as well as search quality.

Hadoop World is a great way to learn how Hadoop is being used to power today’s modern enterprises. These presentations will help you understand how Hadoop improves your data storage and processing environment and directly impacts your business.

Don’t miss out! Register now

With such a large delta from the last release, it is difficult to grasp the important new features and changes. This post is intended to give a high-level view of some of the more significant features introduced in the 0.21.0 release. Of course, it can’t hope to cover everything, so please consult the release notes (Common, HDFS, MapReduce) and the change logs (Common, HDFS, MapReduce) for the full details. Also, please let us know in the comments of any features, improvements, or bug fixes that you are excited about.

You can download Hadoop 0.21.0 from an Apache Mirror. Thanks to everyone who contributed to this release!

Project Split

Organizationally, a significant chunk of work has arisen from the project split, which transformed a single Hadoop project (called Core) into three constituents: Common, HDFS, and MapReduce. HDFS and MapReduce both have dependencies on Common, but (other than for running tests) MapReduce has no dependency on HDFS. This separation emphasizes the fact that MapReduce can run on alternative distributed file systems (although HDFS is still the best choice for sheer throughput and scalability), and it has made following development easier since there are now separate lists for each subproject. There is one release tarball still, however, although it is laid out a little differently from previous releases, since it has a subdirectory containing each of the subproject source files.

From a user’s point of view little has changed as a result of the split. The configuration files are divided into core-site.xml, hdfs-site.xml, and mapred-site.xml (this was supported in 0.20 too), and the control scripts are now broken into three (HADOOP-4868): in addition to the bin/hadoop script, there is a bin/hdfs script and a bin/mapreduce script for running HDFS and MapReduce daemons and commands, respectively. The bin/hadoop script still works as before, but issues a deprecation warning. Finally, you will need to set the HADOOP_HOME environment variable to have the scripts work smoothly.

Common

The 0.21.0 release is technically a minor release (traditionally Hadoop 0.x releases have been major, and have been allowed to break compatibility with the previous 0.x-1 release) so it is API compatible with 0.20.2. To make the intended stability and audience of a particular API in Hadoop clear to users, all Java members with public visibility have been marked with classification annotations to say whether they are Public, or Private (there is also LimitedPrivate which signifies another, named, project may use it), and whether they are Stable, Evolving, or Unstable (HADOOP-5073). Only elements marked as Public appear in the user Javadoc (Common, MapReduce; note that HDFS is all marked as private since it is accessed through the FileSystem interface in Common). The classification interface is descibed in detail in Towards Enterprise-Class Compatibility for Apache Hadoop by Sanjay Radia.

This release has seen some significant improvements to testing. The Large-Scale Automated Test Framework, known as Herriot (HADOOP-6332), allows developers to write tests that run against a real (possibly large) cluster. While there are only a dozen or so tests at the moment, the intention is that more tests will be written over time so that regression tests can be shared and run against new Hadoop release candidates, thereby making Hadoop upgrades more predictable for users.

Hadoop 0.21 also introduces a fault injection framework, which uses AOP to inject faults into a part of the system that is running under test (e.g. a datanode), and asserts that the system reacts to the fault in the expected manner. Complementing fault injection is mock object testing, which tests code “in the small”, at the class-level rather than the system-level. Hadoop has a growing number of Mockito-based tests for this purpose (MAPREDUCE-1050).

Among the many other improvements and new features, a couple of small ones stand out: the ability to retrieve metrics and configuration from Hadoop daemons by accessing the URLs /metrics and /conf in a browser (HADOOP-5469, HADOOP-6408).

HDFS

Support for appends in HDFS has had a rocky history. The feature was introduced in the 0.19.0 release, and then disabled in 0.19.1 due to stability issues. The good news is that the append call is back in 0.21.0 with a brand new implementation (HDFS-265), and may be accessed via FileSystem’s append() method. Closely related—and more interesting for many applications, such as HBase—is the Syncable interface that FSDataOutputStream now implements, which brings sync semantics to HDFS (HADOOP-6313).

Hadoop 0.21 has a new filesystem API, called FileContext, which makes it easier for applications to work with multiple filesystems (HADOOP-4952). The API is not in widespread use yet (e.g. it is not integrated with MapReduce), but it has some features that the old FileSystem interface doesn’t, notably support for symbolic links (HADOOP-6421, HDFS-245).

The secondary namenode has been deprecated in 0.21. Instead you should consider running a checkpoint node (which essentially acts like a secondary namenode) or a backup node (HADOOP-4539). By using a backup node you no longer need an NFS-mount for namenode metadata, since it accepts a stream of filesystem edits from the namenode, which it writes to disk.

New in 0.21 is the offline image viewer (oiv) for HDFS image files (HADOOP-5467). This tool allows admins to analyze HDFS metadata without impacting the namenode (it also works with older versions of HDFS). There is also a block forensics tool for finding corrupt and missing blocks from the HDFS logs (HDFS-567).

Modularization continues in the platform with the introduction of pluggable block placement (HDFS-385), an expert-level interface for developers who want to try out new placement algorithms for HDFS.

Other notable new features include:

• Support for efficient file concatenation in HDFS (HDFS-222)
• Distributed RAID filesystem (HDFS-503) – an erasure coding filesystem running on HDFS, designed for archival storage since the replication factor is reduced from 3 to 2, while keeping the likelihood of data loss about the same. (Note that the RAID code is a MapReduce contrib module since it has a dependency on MapReduce for generating parity blocks.)

MapReduce

The biggest user-facing change in MapReduce is the status of the new API, sometimes called “context objects”. The new API is now more broadly supported since the MapReduce libraries (in org.apache.hadoop.mapreduce.lib) have been ported to use it (MAPREDUCE-334). The examples all use the new API too (MAPREDUCE-271). Nevertheless, to give users more time to migrate to the new API, the old API has been un-deprecated in this release (MAPREDUCE-1735), which means that existing programs will compile without deprecation warnings.

The LocalJobRunner (for trying out MapReduce programs on small local datasets) has been enhanced to make it more like running MapReduce on a cluster. It now supports the distributed cache (MAPREDUCE-476), and can run mappers in parallel (MAPREDUCE-1367).

Distcp has seen a number of small improvements too, such as preserving file modification times (HADOOP-5620), input file globbing (HADOOP-5472), and preserving the source path (MAPREDUCE-642).

Continuing the testing theme, this release is the first to feature MRUnit, a contrib module that helps users write unit tests for their MapReduce jobs (HADOOP-5518).

Other new contrib modules include Rumen (MAPREDUCE-751) and Mumak (MAPREDUCE-728), tools for modelling MapReduce. The two are designed to work together: Rumen extracts job data from historical logs, which Mumak then uses to simulate MapReduce applications and clusters on a cluster. Gridmix3 is also designed to work with Rumen traces. The job history log analyzer is another tool that gives information about MapReduce cluster utilization (HDFS-459).

On the job scheduling front there have been updates to the Fair Scheduler, including global scheduling (MAPREDUCE-548), preemption (MAPREDUCE-551), and support for FIFO pools (MAPREDUCE-706). Similarly, the Capacity Scheduler now supports hierarchical queues (MAPREDUCE-824), and admin-defined hard limits (MAPREDUCE-532). There is also a brand new scheduler, the Dynamic Priority Scheduler, which dynamically changes queue shares using a pricing model (HADOOP-4768).

Smarter speculative execution has been added to all schedulers using a more robust algorithm, called Longest Approximate Time to End (LATE) (HADOOP-2141).

Finally, a couple of smaller changes:

• Streaming combiners are now supported, so that the -combiner option may specify any streaming script or executable, not just a Java class. (HADOOP-4842)
• On the successful completion of a job, the MapReduce runtime creates a _SUCCESS file in the output directory. This may be useful for applications that need to see if a result set is complete just by inspecting HDFS. (MAPREDUCE-947)

What’s Not In

Finally, it bears mentioning what didn’t make it into 0.21.0. The biggest omission is the new Kerberos authentication work from Yahoo! While a majority of the patches are included, security is turned off by default, and is unlikely to work if enabled (certainly there is no guarantee that it will provide any level of security, since it is incomplete). A full working security implementation will be available in 0.22, and also the next version of CDH.

Also, Sqoop, which was initially developed as a Hadoop contrib module, is not in 0.21.0, since it was moved out to become a standalone open source project hosted on github.

From the statistical or technical perspective, fraud is a rare event that results in a significant financial impact to the organization.

Both definitions emphasize that the event is rare (assuming that most of the population is law-abiding citizens), is intentional (there is no “accidental” fraud), as well as imply a significant damage caused to the defrauded party (otherwise why bother).  Fraud detection is difficult from statistical point of view for exactly these reasons: (a) the events are rare and it is difficult to build a predictive model and (b) fraud assumes a real human being behind it and incorporates elements of game theory since the fraudster is often an insider who knows how to game the system.

Fraud and Rare Events

By definition, fraud is an unexpected or rare event with significant financial or other damage.  Fraud assumes that the fraudster has some prior information how the current system works including previous successful and unsuccessful fraud cases and possibly the fraud detection mechanisms.  The above breaks the standard statistical modeling assumption, the variable independence or i.i.d. assumption, making building a reliable statistical model difficult.  Often the fraudster is working in the same industry that the fraud detection is supposed to protect, is intimately familiar with the fraud detection methods, and is actively trying to avoid detection by masquerading.

Rare event detection problem is also applicable to online advertising and marketing, particularly with predicting “long tail” events and terrorism detection.

One common example of fraud is associated with Taleb distribution where a seemingly high probability of a small gain shadows a small probability of a large loss that more than outweighs the gains.  Relatively long periods of slightly better than moderate gains are interrupted by a rare event of large losses.  It is easy to defraud investors by presenting the results of partial analysis excluding the “rare events”.

Fraud Prevention

Since fraud is so hard to prove in courts, most organizations and individuals try to prevent fraud from happening by blanket measures.  This includes limiting the amount of damage the fraudster can impact on the organization as well as early detection of fraud patterns.  For example, credit card companies can cut the credit card limit across the board in anticipation of a few negative fraud cases.  Advertisers can prevent advertising campaigns with low number of qualifying events.  And anti-terrorism agencies can prevent people with bottles of pure water from boarding the planes.  These actions are often in contrast with the company efforts to attract more customers and result in general dissatisfaction.  To the rescue are new technologies like Hadoop, Influence Diagrams and Bayesian Networks which are computationally expensive (these are NP-hard in computer science terminology) but are more accurate and predictive.

Why Hadoop?

Hadoop is a distributed system for processing large amounts of data.  In a recent Hadoop Summit 2010 Yahoo, Facebook, and other companies announced that they currently process a few TBs of data per day and the volumes are growing at exponential rates.  Hadoop can be vital for solving the fraud detection problem because:

1. Sampling does not work for rare events since the chance of missing a positive fraud case leads to significant deterioration of model quality.
2. Hadoop can solve much harder problems by leveraging multiple cores across thousands of machines and search through much larger problem domains.
3. Hadoop can be combined with other tools to manage moderate to low response latency requirements.

Let’s go through these reasons one by one.  Sampling is a common technique for modeling rare events.  One of the problems with sampling is that we cannot afford to throw away rare positive cases.  Even in a stratified or proportional sampling scheme one has to retain all positive cases since the model accuracy heavily depends on them (one can usually discard some negative cases though).  Given the above, the system still has to go through the whole dataset to sieve through the positive and negative cases.

Hadoop is known for its gnawing power.  Nothing can compare with the throughput power of thousands of machines each of which has multiple cores.  As was reported recently at the Hadoop Summit 2010, the largest installations of Hadoop have 2,000 to 4,000 computers with 8 to 12 cores each, amounting to up to 48,000 active threads looking for a pattern at the same time.  This allows either (a) looking through larger periods of time to incorporate events across a larger time frame or (b) taking more sources of information into account.  It is quite common among social network companies to comb through twitter blogs in search of relevant data.

Finally, one of the fraud prevention problems is latency.  The agencies want to react to an event as soon as possible, often within a few minutes of the event.  Yahoo recently reported that it can adjust its behavioral model in a response to a user click event within 5-7 minutes across several hundred of millions of customers and billions of events per day.  Cloudera has developed a tool, Flume, that can load billions of events into HDFS within a few seconds and analyze them using MapReduce.

Often fraud detection is akin to “finding a needle in a haystack”.  One has to go through mountains of relevant and seemingly irrelevant information, build dependency models, evaluate the impact and thwart the fraudster actions.  Hadoop helps with finding patterns by processing mountains of information on thousands of cores in a relatively short amount of time.

Where to look next?

Techniques for fraud detection are industry-specific as a rule and often are guarded since they obviously represent valuable information for potential fraudsters.  They are often kept confidential for this reason.  Moreover, the fraud detection techniques are usually a moving target since the fraudsters quickly adjust to the new fraud detection mechanisms.

One of the most publicized technical frauds is click fraud in on-line advertising.  Since advertisers are often charged on the per-click basis — so called PPC campaigns; there is a way to charge advertisers on a per-conversion basis, which we will cover shortly, but a different type of fraud emerges there where the advertiser tries to conceal the conversions — the traffic provider like a search web site has a clear incentive to inflate the number.  Additionally, an advertiser competitor may be incentivized to inflate the number to skew the original advertiser margin.  This can be achieved by a human or software agent that generates extra traffic and clicks on the competitor site.  Fraud management companies like Anchor Intelligence and Click Forensics estimate that approximately 20% to 30% of all clicks are fraud.  How do we know that a click is a fraud?

Decline in the number of conversions — first and most important, if your conversion rate is normally positive (that is, you are making a profit on your ad), and all of a sudden, conversion dives into negative numbers, this could be a sign of click fraud in action.  Click fraud causes extra clicks on your ad with no actual purchases, and your conversion rate will fall accordingly.

An abnormal number of clicks from the same IP address or a pattern in the access times — although this is the most obvious and easily identified form of click fraud, it is amazing how many fraudsters still use this method, particularly for quick attacks.  They may choose a to strike over a long weekend when they figure you may not be watching your log files carefully, clicking on your ad repeatedly so that when you return to work on Tuesday, your account is significantly depleted.  Part of this fraud might be unintentional when a user tries to reload a page.

Large “abandonment rate”, or numbers of visitors who leave your site quickly — another indication of click fraud can be a pattern of visitors clicking on your ad, spending the minimum amount of time on your site required by your PPC search engine to establish it as a valid click (usually 30 seconds or more), and then leaving without having left the landing page at all.

A large number of impressions, without the follow-through clicks or click on your ad — if you notice that there are a lot more impressions (views) of your website; this could indicate the impression fraud we discussed earlier. Artificial inflation of your ad impressions may cause your clickthrough rates to drop below the Google minimum, and your ad will be disabled.  Until you realize this, your competitors have free reign to use your keywords, sometimes at bargain prices.  As well, your relevancy ratings for search engines may drop as they record numerous impressions, but no interest shown via visits to other parts of your website, which could lead to a shutdown of your campaign.

Abnormally high clicks and impressions on affiliate websites — although affiliates themselves are sometimes involved in conducting click fraud schemes, they can be victims of click fraud themselves.  If one of their competitors uses this same method of excessive clicks and impressions on an affiliate’s site, the PPC search engine will soon notice an abnormally high payment to a certain affiliate and perhaps go as far as canceling that affiliate’s account, even though he or she was not engaging in any form of click fraud.

A large number of clicks coming from countries outside of your normal market area — using IP geo-location services, you can identify which country an IP address is probably coming from.

In the case of performance-based advertising, the advertiser himself is interested in concealing some of the traffic, not inflating it.  Since most of the performance-based measurements is based in beacons or pixels placed on the advertiser conversion page, advertiser has an incentive to (temporarily) block the traffic from the beacon or to completely remove it from their web-site.

Fraud is prevalent in telecom industry.  One of the leading commercially available fraud detection products is HP FMS system on which the author had a pleasure to work personally.  The types of telecom fraud include:

Subscription fraud — involves the acquisition of telecommunications services using stolen or false credentials and/or identity with no intention of paying. With subscription fraud, not only do service providers lose revenue, but also individual consumers are vulnerable to having their identity stolen and credit rating tarnished.

Technical/network fraud — occurs when someone uses equipment or technology to gain access to a service without paying. Fraudulent calls are typically billed to the legitimate owner of the line or service.  Wireless examples include cloning of cell phones or subscriber identity module (SIM) cards. Fixed line examples include clip on or line tapping, private branch exchange (PBX) hacking and calling card fraud. Prepaid services also have a large exposure to fraud with terminal tampering via magnetic strips or SIM chips, or recharging with stolen credit card numbers.

Insider fraud — occurs when individuals inside the operator provide fraudulent access to networks or otherwise thwart the ability of the operator to be paid for services used.

Handset abuse — is what takes place when stolen or lost handsets are used to consume telecommunications services that are in turn paid for by the service provider.  This is an expensive liability for carriers who absorb the costs.

Social engineering — is an effective fraud technique in which people unwittingly help perpetrators by providing sensitive data, illicit access or simply forwarding their calls without ever knowing they have done anything wrong.

All these patterns can be detected with special MapReduce pattern detection techniques. Flume offers low-latency stream processing capabilities.

Needless to say, the fraudsters also explore the potential market and invent new innovative ways to generate fraud.  One of them is deployed by Click Monkeys which deploys a vessel with animals next to the coast of California to generate seemingly random traffic.

Enter the code “london_10pct” when registering and receive a 10% discount!

Hadoop is a rapidly growing field. Prove your expertise by attaining certification from the world’s foremost Hadoop training and consulting company.

.