Three years ago I joined Treato, a social healthcare analysis firm to help treato.com scale up to its present capability. Treato is a new source for healthcare information where health-related user generated content (UGC) from the Internet is aggregated and organized into usable insights for patients, physicians and other healthcare professionals. With oceans of patient-written health-related information available on the Web, and more being published each day, Treato needs to be able to collect and process vast amounts of data – Treato is Big Data par excellence, and my job has been to bring Treato to this stage.

Before the Hadoop era

When I arrived at Treato, the team had already developed a Microsoft-based prototype that could organize a limited amount of health-related UGC into relevant insights, as a proof of concept. The system would:

• Crawl the Web and fetch raw HTML sources,
• Extract the user-generated content (i.e. user’s posts) out of the raw sources,
• Extract concepts from the posts and index them,
• Execute semantic analysis on the posts using natural language processing (NLP) algorithms
• And calculate statistics.

The prototype was able to prove the initial hypothesis that relevant medical insights can be found in social media, you just have to know how to analyze it. We collected data from dozens of websites and individual social media posts in the tens of millions. We had a handful of text analysis algorithms and could only process a couple million posts per day, but the results were impressive. We found that we were able to identify side effects through social media long before initial FDA or pharmaceutical companies issued warnings about them. For example, when we looked at the discussions about Singulair – an asthma medication – we found that almost half of the user generated content discussed mental disorders. When we looked back through the historical data, we learned that this would have been identifiable in our data four years before the official warning.

In order to gain even more health-related insights, we knew we needed a solution that could crawl and process a larger quantity of data – larger by an order of magnitude. That was the point at which Web scale joined the game. In order to collect massive amounts of posts, we needed to add thousands of data sources. And, of course, all the data we collected would need to be analyzed.

Dealing with a few dozen websites was difficult and costly. But we were able to scale up our Microsoft code to handle collection from a several hundred sites, and could process around 250 million posts. We were running a few old IBM boxes that did the collection work and had developed a job manager that administered crawling and fetching tasks. Different servers ran the indexing and the stats calculations, and we had developed a distributed job manager to direct task executions. Different servers were used for serving the data. We didn’t have any storage solution, and all of the boxes worked with local drives.

Besides the fact that administering the process was hell, it was expensive in terms of CPU, network and input/output (I/O); e.g., after each stage, the data needed to be moved to a different server for the next stage. In addition, our job manager didn’t deal with failures; every time a task failed we needed to handle it manually. Needless to say, supporting collection and analysis of thousands of websites would have been impossible using this approach.

Looking at scale

In the beginning of 2010, we started searching for solutions that could support the capabilities we wanted. The requirements included:

1. Reliable and scalable storage.
2. Reliable and scalable processing infrastructure.
3. Search engine capabilities (for retrieving posts) with high availability (HA).
4. Scalable real-time store for retrieving stats, with HA.

We wanted the ability to periodically reprocess the data in a timely manner, so new algorithms or other analysis improvements would take effect on all historical data.

We wanted to know how much it costs to deal with X number of posts, and to be able to scale according to this formula.

We wanted a technology and architecture that would scale with the business.

We searched for answers to questions such as: “How does Google do it?” and it didn’t take too long to find Google’s papers, documentation on Hadoop and MapReduce, and so on.

We started digging deeper in these areas. After a short investigation, it was clear that the Hadoop Distributed File System (HDFS) would support our storage demands, and MapReduce would be a good fit for the processing infrastructure.

First Hadoop cluster in the lab

While looking for Hadoop distributions, I encountered Cloudera’s Distribution including Apache Hadoop (CDH), however, I decided to start with a manual installation since this usually helps me better understand how things work. We started a pilot, setting up a 2 node cluster on Linux boxes. As mentioned, the first installation was done totally manually using the binaries downloaded from Apache, and gently configuring the system. This process was ugly: I needed to download all sorts of binaries from different sources, deal with networking issues, exchange of SSH keys between the nodes, formatting the FS and all sorts of OS tweaks.

We started testing the behavior of the new technology, first with some simple WordCount and pi calculations, and then we quickly wrote MapReduce (Java) code that did parts of our processing and tested it on real HTML sources. The little cluster just worked: I was able to submit jobs & monitor them; I tested recovery from task failures, crash of a node, etc.

Next, I wanted to see how this Hadoop solution scaled. To do this, I installed an additional box and added it to our little Hadoop cluster. It was awesome: after adding the new slave to the cluster, everything was transparent. Suddenly we had more capacity on the file system and more horsepower for processing. The job submission was the same as before; the job submitter (Hadoop client) didn’t even know that the cluster had changed, it simply got the results quicker. We were able to crunch some numbers and got a dollar-per-post cost.

So, the evaluation was great, but still there was the awful installation and maintenance process. That’s when we started to test Cloudera’s Distribution including Apache Hadoop; I think it was version 2 of CDH back then. We re-installed our little cluster from scratch using this Hadoop distribution. The installation process was much easier, and the documentation helped. The setup took only a couple of hours. (CDH3 takes less than an hour). 

After we found a good package, we wanted to set up a bigger cluster for prototyping, and deeper tests and evaluations. Amazon seemed to be the perfect place for that. Using CDH we set up a 10 node (small instances) cluster on EC2. This was used for performance evaluation and the processing rate was about 10M-20M posts per day — approximately 6 times higher than the performance from our pre-Hadoop solution.

We decided to go with Hadoop. This was a dramatic decision, as we took a company with a Microsoft-oriented development team, ported all the code into Java, all the while adopting a new and very complicated technology stack. This actually meant starting implementation from the beginning, opening a new integrated development environment (IDE) and starting to code from scratch. 

In order to reduce risks and avoid critical mistakes, we searched for someone who has “been there, done that” so we could learn from them and validate our overall planned new architecture. Cloudera was our first choice; it made sense to go with a company that has multiple setups behind them, some of which are very large clusters. Cloudera sent Solutions Architect, Lars George, to our offices for two days, and we gave him our suggested design in advance. We felt lucky to have Lars, an HBase committer and author of HBase: The Definitive Guide, as our consultant since HBase was one of the core technologies we were using.

For the first implementation phase, we decided to go with HDFS, MapReduce & HBase. Our in-house-developed crawlers were using HBase as the store for the list of URLs to be fetched. This table should be able to scale to billions of rows. The fetcher (the component in charge of fetching the raw HTML sources) gets the URL queues out of HBase, runs HTTP requests, and stores the raw HTML sources in large files on top of HDFS (few gigs per file). Both the crawler and fetcher don’t use any relational database or any other kind of store except HDFS & HBase. These two components are network and I/O intensive, but CPU is not much of an issue.

Next comes the processing. Each line in the HDFS files contains an HTML source and metadata related to this source. For each directory of files in HDFS, the following processing jobs need to be executed:

1. Turn the unstructured HTML into a list of post entities (content, timestamp, etc.)
2. Each post needs to be processed as follows:
• Index key terms – extract medical concepts out of the post content, using Treato’s extensive knowledge base
• Execute text analysis algorithms
3. Calculate all statistics and update the HBase stats tables.
4. Post all documents (user’s posts) into our search engine (Solr).

During this process, many database queries and updates are needed. For example, each post retrieved may potentially already exist in our system, and of course we don’t want to add a duplicate post to our system, nor invest processing power on documents we already have. In order to accomplish this, we need to calculate a hash for each post, and then check it against a database containing all of the existing hashes. For this purpose HBase works perfectly in terms of both latency and load.

After the design phase, we started implementation. All R&D teams worked on porting their code into Java, and our Ops team worked on planning the data center (we decided on co-location data center setup).

For the initial setup, we had 11 boxes that comprised our Hadoop cluster, two of which were name nodes in an active / passive mode (one was in standby for manual failover in case the active NameNode failed). Nine nodes were slaves hosting DataNodes, TaskTrackers and Region-Servers daemons. In addition to this we had three boxes running Zookeeper services.

The new system was capable of analyzing 50M posts per day. This was a significant performance improvement. In addition, it was reasonable to maintain, reliable and worked quite smoothly. Of course, there were bumps in the road, but in the end we managed to overcome them all.

We have continued to improve and expand the solution, and today we can process 150 – 200 million user posts per day. In subsequent blog posts, I will share in greater detail our system design, use of HBase, and cluster architecture.

A year ago I rolled my first Hadoop system into production. Since then, I’ve spoken to quite a few people who are eager to try Hadoop themselves in order to solve their own big data problems. Despite having similar backgrounds and data problems, few of these people have sunk their teeth into Hadoop. When I go to Hadoop Meetups in San Francisco, I often meet new people who are evaluating Hadoop and have yet to launch a cluster. Based on my own background and experience, I have some ideas on why this is the case.

I studied computer science in school and have worked on a wide variety of computer systems in my career, with a lot of focus on server-side Java. I learned a bit about building distributed systems and working with large amounts of data when I built a pay-per-click (PPC) ad network in 2004. The system is still in operation and at one point was handling several thousand searches per second. As the sole technical resource on the system, I had to educate myself very quickly about how to scale up.

As I contemplated how doomed I would be should traffic levels increase much more, I remember wondering to myself, “How does Google deal with all that data?” The answer came to me in the form of the Google File System (GFS) paper and later the MapReduce paper, both from Google. It dawned on me that because Google was forced to solve a much larger problem, they had come up with an elegant solution for a whole range of more modest data problems running on commodity hardware. But it wouldn’t be until 2010 that I would get to work with this technology firsthand.

As I wrote in an earlier article, I started re-architecting USASearch, the U.S. government’s search system, in 2009 based on a solution stack of free, open source software including Ruby on Rails, Solr, and MySQL. A wave of déja vu hit me as I started worrying about what to do with the growing mountain of data piling up in MySQL and our increasing need to analyze it in different ways. I had heard that a new company called Cloudera, founded by some big data people from Yahoo!, Google, and Facebook, was making Hadoop available for the masses in a reliable distribution, much in the same way that RedHat did for Linux. Curiosity got the best of me and I bought the newly minted Hadoop: The Definitive Guide from O’Reilly. The most insightful part of the book to me was the very first sentence. It’s a quote from Grace Hopper: “In pioneer days, they used oxen for heavy pulling, and when one ox couldn’t budge a log, they didn’t try to grow a bigger ox.” I didn’t want to grow a bigger server; I wanted to harness a bunch of small servers together to work in unison. The more I learned the more curious I got, so I started reading more. And that’s when I hit my first roadblock.

I think people who have been working with Hadoop technologies for years and years sometimes forget just how rich and diverse the big data software ecosystem has become, and how daunting it can be to folks approaching it for the first time. When people at the Meetups say they are evaluating solutions to their data scaling problem, the answers they hear sound something like this: “Just use Hadoop Hive Pig Mahout Avro HBase Cassandra Oozie Sqoop Flume ZooKeeper Cascading NoSQL RCFile. Oh, almost forgot…cloud.”

The thought of wading through all of that just to learn about what I needed to learn about was a bit too overwhelming for me, so I put the whole matter aside for a few months. Over time, I started to dive into each of these projects to understand the primary use case, how active the developer community was and which organizations were using it in production. I converged on the idea of using Hive as a warehouse for our data. I opted for Cloudera’s distribution since I wanted to reduce the risk of running into compatibility issues between all the various subsystems. Having tracked down anomalies in a highly multi-threaded and contentious distributed Java system before, I liked the idea of someone else taking on that problem for me.

At some point, I had read everything I could read and grew impatient to get my hands dirty, so I decided to just download CDH3 on my laptop and give it a try. The tutorial instructions for the standalone version worked, and I successfully computed more digits of pi than I ever thought I’d need. After creating some sample data in Hive and running a few queries, I felt pretty confident that Hive would be the right tool for the job. I just needed to find somewhere to install and run HDFS (namenode, secondary namenode, and data nodes), Hadoop (jobtracker and tasktracker nodes), Hive, and Hue for a nice front end to it all.

I knew from my past experience how to stretch the limits of CPU, disk, IO, and memory on commodity servers, and I identified a few potential servers at our primary datacenter with resources I figured I could leverage. Once again I followed the tutorial instructions, this time for the fully distributed version of CDH3, and once again I started to compute pi. And that’s when I hit my second roadblock. It took me a few days to figure out that I had a problem with DNS. Each machine needs to be able to resolve every other machine’s name and IP in the cluster. Whether you do that via /etc/hosts or a local DNS server is up to you, but it needs to happen or the whole thing gets wedged. Once I got that sorted out, everything just started falling into place and I had Hive working in production within a few days. A week later, I started pulling out the MySQL jobs and deleting big tables, and that’s been the trend ever since.

Over time, I’ve gone on to learn about using custom Ruby mappers in Hive, moving data back and forth between MySQL and Hive with Sqoop, and getting the data into HDFS in real-time with Flume. All of these components from the Cloudera distribution are working nicely in our production environment now, and I sleep well at night knowing I have such a solid, deliberate plan for growth. My initial investment in learning about the Hadoop ecosystem is really paying dividends, but when I think about all those people at the Meetups stuck in evaluation mode, I feel their pain. Does it have to be such a struggle?

The big challenge in my opinion is not that any one piece of the puzzle is too difficult. Any reasonably smart (or in my case stubborn) engineer can set themselves on the task of learning about a new technology once they know that it needs to be learned. The challenge with the Hadoop ecosystem is that it presents the newbie with the meta-problem of figuring out which of these tools are appropriate for their use case at all, and whether or not to even consider the problem today versus deferring it until later. In a way Facebook has it easy, because when you are adding 15TB of data per day, that decision is pretty much made for you.

For all the companies sitting in the twilight between the gigabyte and the petabyte who don’t have Hadoop expertise in-house, there is a collection of free information to help guide people to the right solution space (Hadoop Tutorial, White Papers). These days, when I talk to people who are evaluating solutions to their big data problems, my advice to them is to break down their problems into a few discrete use cases and then work on ferreting out the technologies that are designed for that use case. Get a proof of concept to demonstrate that the technology can address your use case and convince yourself and others that you’re on the right track. Work toward putting something simple into production. Lather, rinse, and repeat. I am still in that cycle myself, as these days I’m exploring HBase and OpenTSDB to give me low-latency access to time series data and Mahout to do frequent item set mining, but that’s another article for another day.

In Early 2010 at RichRelevance, we were searching for a new way to store our long lived data that was compact, efficient, and maintainable over time. We had been using Hadoop for about a year, and started with the basics – text formats and SequenceFiles. Neither of these were sufficient. Text formats are not compact enough, and can be painful to maintain over time. A basic binary format may be more compact, but it has the same maintenance issues as text. Furthermore, we needed rich data types including lists and nested records.

After analysis similar to Doug Cutting’s blog post, we chose Apache Avro. As a result we were able to eliminate manual version management, reduce joins during data processing, and adopt a new vision for what data belongs in our event logs. On Cyber Monday 2011, we logged 343 million page view events, and nearly 100 million other events into Avro data files.

Avoiding Version Management Baggage

Have you ever seen code for manual serialization version management like the below?

int version = input.readInt();
this.name = input.readName();
this.age = input.readInt();
if (version >= 2) {
  this.favoriteColor = input.readString();
} else {
  this.favoriteColor = "";
}

Manual version management is painful. If you evolve what data you store continuously, it does not take long to end up with dozens of versions. In order to read every version that has been written and stored, your code has to carry a lot of baggage.

With Avro, you can avoid writing code like the above. The concept is simple. Store the schema used to write your data along with your data, and use it to make the written data conform to the schema that the reader expects. If a field is missing, use the default. If has been removed or moved, handle it.

Over the last two years, we have doubled the complexity of our our page view schema across about 15 schema versions. There is not one line of code that deals with version management, and the current code can read any of the data written over that time. What may be a surprise to some is that our old code can read newly written data as well. The data is both forward and backward compatible, within the rules in the Avro Specification.

Leveraging Complex Data Types

Avro supports complex data types such as arrays, maps, enumerations, and nested records. The Avro data model makes it possible to serialize any non-recursive data structure, including trees and heterogeneous lists. We use this property to describe our events using Avro Schemas that map to natural object representations on our front end servers. For example, one of the elements in a page view is an array of product recommendation sets, each set containing a list of products displayed. Another element in a page view is what we call a page context – each type of page on a merchant’s site has a unique context that differs from other page types. A product page context is the product being displayed. A search page context is the search terms in the search query. There are about 30 page context types, and we represent the range of page context possibilities using an Avro union, so that all of these different event variations can be written in the same format and to the same log.

With a simpler data model, one might have had to log each context type separately, making it harder to get a full picture of what happened in a single request during analysis.

A New Vision for an Event Log

With the above properties of Avro, we were able to formulate a new vision for what an event log should be. The new model has the following properties:

• A singe HTTP request creates a single, atomic log event defined by an Avro schema
• The event contains all of the resolved request inputs
• The event contains the result of any decisions made during the request

Together, these imply that it is never necessary to join different sets of data together to reconstruct what happened in an individual request during analysis. This also significantly reduces the value of data contained in raw HTTP logs, since the Avro based logs become the origin for all major processing. Since raw HTTP logs are significantly larger than compressed binary structured data, this significantly reduces the size of data we must keep for long periods of time.

More Avro at RichRelevance

We have built Hive and Pig adapters to map our Avro data into these tools for ad-hoc queries and automated tasks. Additionally, we leverage the same Avro schemas from our log files to store click streams in HBase. We also use Avro to store data compactly in key-value stores.

The log file example is what I call a schema first use case of Avro, where we define a schema for log events that can be used across different systems over a long period of time. An alternative usage style is what I call code first, where you start with code and bind that to a serialization with a less schema-centric view. I feel that the code first usage style is more applicable for data that lives for short or medium time scales, such as with RPC or MapReduce intermediates. We will be deepening our investment in Avro and using it with code first use cases in the future, in the process working with the community to improve the developer experience for those use cases.

Avro is a growing, evolving project that I see as more broad than a serialization framework. At heart Avro is about applying a schema to data, in order to manipulate that data in well defined ways. Serialization, validation, and transformation are only some of the operations you can apply to data that conforms to a known schema. Over time the project will grow to have more and more functionality centered around operations you can apply to data that conforms to an Avro Schema. I look forward to working with the Avro community as the project continues to evolve!

The missions and data of governments make the government sector one of particular importance for Big Data solutions. Federal, State and Local governments have special abilities to focus research in areas like Health Sciences, Economics, Law Enforcement, Defense, Geographic Studies, Environmental Studies, Bioinformatics, Information Search/Discovery, and Computer Security. Government-Industry teams are working to field Big Data solutions in all these fields.

The Government Big Data Solutions Award was established by the technology blog CTOvision.com to help facilitate the exchange of best practices, lessons learned and creative ideas for solutions to hard data challenges in the government sector. Nominations are being sought and your input would be most appreciated. Please nominate capabilities and solutions you know hold great potential for mission impact in the government sector.

Award winners will be written up on CTOvision.com, and a presentation of awards will also be made at Hadoop World 2011.

You may nominate industry solution providers with proven capabilities, government organizations who have built or implemented Big Data solutions or individuals who have played a direct role in establishing capability. The key criteria we are evaluating is the utility of the solution to serve government missions.

Our judges include: Doug Cutting (creator of Lucene and Hadoop), Alan Wade (former CIA and IC CIO), Ryan Lasalle (Accenture Cyber R&D), Ed Granstedt (QinetiQ Strategic Solution Center) and Chris Dorobek (Founder, editor and publisher of DorobekInsider.com).

To nominate, please use our online survey form at: https://www.surveymonkey.com/s/CNHB5QZ

A critical premise for success of online advertising networks is to successfully collect, organize, analyze and use large volumes of data for decision making. Given the nature of their online orientation and dynamics, it is critical that these processes be automated to the largest extent possible.

Specifically, the success of advertising technology and its impact on revenue are directly proportional to its capability to use large amounts of data in order to compute proper impression value given the unique circumstances of ad serving events such as the characteristics of the impression, the ad, and the user as well as the content and context. As a general rule, more data results in more accurate predictions.

In addition, to Optimization, Reporting and Analytics provide indispensable feedback to our internal Business and Sales teams helping us acquire new, and expand current, commitments from external customers.

At AOL, we started large-scale data collection more than 4 years ago and went from using heavily sampled data sets to being able to process full serving logs. We have been using Apache Hadoop since version 0.14 as a part of an R&D effort and recently moved to Cloudera CDH3 distribution. Gradually, we introduced more systems and technologies to our ecosystem around Hadoop.

We chose Hadoop for several reasons:

Figure 1. Growth of Hadoop cluster

Figure 2. Growth in sampling rate

We show growth of our Hadoop clusters in Figure 1, and increase in the sampling rate in Figure 2. Between the 3^rd and 4^th iteration we switched to disks that are 4 times larger and we used 4-8 times more cores per node. The increase in the total number of CPUs was even more pronounced as we found we needed more processing power for newly developed processing flows. During the initial stages growing the sampling rate was the primary goal. As the number of processing pipelines increased, the output data volume increased. We’ve also added more external data flows. These two trends drove the increase in total storage space and processing power beyond full log samples between stages 4 and 5. Note that the impact of important factors like the business environment and team growth had significant impact on the pace of cluster upgrades.

At the same time, we grew the ecosystem around Hadoop to encompass other infrastructure and computational components such as databases, caching and high-performance computing clusters. As our Hadoop clusters increased in size, these clusters correspondingly increased to store and process larger data sets.

The main reason for the qualitative change in shifting between the 3^rd and 4^th iteration was the move from R&D to a production environment. With involvement of additional teams we faced several challenges that Cloudera helped us with:

The last point is especially important as we see Hadoop as an ever-evolving data processing platform. We see ourselves as a contributor and partner in this process – through the recently introduced Cloudera Customer Council we participate in discussions and working groups. For us, this is a great learning experience which simultaneously provides ample opportunities for us to contribute to an important technology that is changing the way we do business.

This is the first of a two part series about moving away from Amazon’s EMR service to an in-house Hadoop cluster.

When we first started using Hadoop, we went down the path of Amazon’s EMR service.  We had limited operational resources and wanted to get up and running quickly.  After a while, we starting hitting the limitations of EMR and had to migrate towards managing our own cluster.  In doing so we did not want to lose the features of EMR we found useful – mainly the ease of cluster setup.

This first part of the series discusses our motivation for choosing and then moving away from EMR, while the second part deals with how we maintained ease of cluster setup using Puppet.

Many of our systems use Amazon’s S3 as a backup repository for log data.  Our data became too large to process by traditional techniques, so we started using Amazon’s Elastic MapReduce (EMR) to do more expensive queries on our data stored in S3.  The major advantage of EMR for us was the lack of operational overhead.  With a simple API call, we could have a 20 or 40 node cluster running to crunch our data, which we shutdown at the conclusion of the run.

We had two systems interacting with EMR.  The first consisted of shell scripts to start an EMR cluster, run a pig script, and load the output data from S3 into our data warehousing system.  The second was a Java application that launched pig jobs on an EMR cluster via the Java API and consumed the data in S3 produced by EMR.

The magic of spinning up and configuring a Hadoop cluster in EC2 was spectacular, but there were a few areas that we saw room for improvement.  In particular:

Performance & Tuning. We were hit by the small-files problem, lack of data locality (data stored in S3 but processed on nodes of the EMR cluster), decompression (bz2) performance issues, and virtualization penalties.  To solve these problems, we decided that we needed a non-transient cluster (to satisfy data locality), and a process to aggregate our logfiles into a Hadoop-friendly size and data format (we ultimately chose avro). After crunching the numbers, it was evident that storing large amounts of data on an EC2 cluster quickly becomes expensive, and one still suffers from virtualization penalties (particularly since Hadoop is so I/O intensive), so we decided to build-out a cluster using CDH3.

Monitoring. Typically for us, a pig script running on EMR was one step in a workflow, so we needed to monitor the status of the job to determine when it finished and the next steps could continue.  While Amazon exposes a rich API for monitoring a job, we really wanted a more generic mechanism for monitoring all steps in a workflow, not just those on an EMR cluster.  After considering a number of solutions, we ultimately chose to use Azkaban as our workflow engine for managing dependencies, alerting, and monitoring (which we added atop Azkaban ourselves).

API Access. Interacting with a cluster only over an API is both a blessing and a curse.  The API takes care of otherwise complicated mechanics, such as starting, configuring, and stopping the cluster.  With that said, the calls to the EMR service are rate-limited, so it doesn’t scale very well for monitoring a number of clusters.  Also, we found that we could continuously keep a cluster busy, and thus the EMR limitation of 100 or so jobs on a cluster meant that we had to build wrappers to periodically shutdown and startup clusters.

Lack of latest features. We were using Hadoop 0.18 and Pig 0.3 on EMR, which were missing many features that we wanted to try (e.g. JVM reuse, CombineInputFormats, and improved pig optimization plans).  Eventually, Amazon upgraded to Hadoop 0.20 and Pig 0.6, but even at that point Cloudera’s Distribution including Apache Hadoop had backported many useful features such as performance improvements, monitoring enhancements, and additional APIs.  In addition, CDH provides a full-suite of solutions including Pig, Hive, Flume, and Sqoop, that we’re either actively using or planning to use.

For us, the major drawback to moving away from EMR was new operational overhead.  Starting a cluster with an API call is incredibly useful, and we soon discovered that CDH provided scripts for doing so (now there’s something even better, Apache Whirr).  Eventually, we decided to move out of the cloud, though, so we wanted to build an infrastructure for maintaining a cluster that worked regardless of the hardware configurations.  The RPMs for CDH3 and the great documentation on installing and configuring CDH from Cloudera helped to make this project much-less intimidating.  Ultimately, we built puppet modules for configuring our cluster, which we’ll talk much more about in part two of this post.

This post was contributed by The Global Biodiversity Information Facility development team.

The Global Biodiversity Information Facility is an international organization, whose mission is to promote and enable free and open access to biodiversity data worldwide. Part of this includes operating a search, discovery and access system, known as the Data Portal; a sophisticated index to the content shared through GBIF. This content includes both complex taxonomies and occurrence data such as the recording of specimen collection events or species observations. While the taxonomic content requires careful data modeling and has its own challenges, it is the growing volume of occurrence data that attracts us to the Hadoop stack.

The Data Portal was launched in 2007. It consists of crawling components and a web application, implemented in a typical Java solution consisting of Spring, Hibernate and SpringMVC, operating against a MySQL database. In the early days the MySQL database had a very normalized structure, but as content and throughput grew, we adopted the typical pattern of denormalisation and scaling up with more powerful hardware. By the time we reached 100 million records, the occurrence content was modeled as a single fixed-width table. Allowing for complex searches containing combinations of species identifications, higher-level groupings, locality, bounding box and temporal filters required carefully selected indexes on the table. As content grew it became clear that real time indexing was no longer an option, and the Portal became a snapshot index, refreshed on a monthly basis, using complex batch procedures against the MySQL database. During this growth pattern we found we were moving more and more operations off the database to avoid locking, and instead partitioned data into delimited files, iterating over those and even performing joins using text files by synthesizing keys, sorting and managing multiple file cursors. Clearly we needed a better solution, so we began researching Hadoop.  Today we are preparing to put our first Hadoop process into production.

Our first objective is to address the monthly processing we perform. This area of work does not increase functionality offered through the portal (that will be addressed following this infrastructural work) but rather aims to:

• Reduce the latency between a record changing on the publisher side, and being reflected in the index
• Reduce the amount of (wo)man-hours needed to coax through a successful processing run
• Improve the quality assurance by inclusion of
  • Checking that terrestrial point locations fall within the stated country using shapefiles
  • Checking coastal waters using Exclusive Economic Zones
  • Rework all the date and time handling
  • Use dictionaries (vocabularies) for interpretation of fields such as Basis of Record
  • Integrate checklists (taxonomic, nomenclatural and thematic) shared through the GBIF ECAT Programme to improve the taxonomic services, and the backbone (“nub”) taxonomy.
  • Provide a robust framework for future development
  • Allow the infrastructure to grow predictably with content and demand growth

Things have progressed significantly since the early Hadoop investigations which included hand crafting MapReduce jobs, and GBIF are now developing using the following technologies:

• Apache Hadoop: A distributed file system and cluster processing using the Map Reduce framework
  • GBIF are using the Cloudera’s Distribution including Apache Hadoop
  • Sqoop: A utility to synchronize between relational databases and Hadoop
  • Hive: A data warehouse infrastructure built on top of Hadoop, and developed and open-sourced by Facebook. Hive gives SQL capabilities on Hadoop, which is particularly attractive to a development team fluent in SQL. [Full table scans on GBIF occurrence records reduce from hours to minutes]
  • Oozie: An open-source workflow/coordination service to manage data processing jobs for Hadoop, developed and then open-sourced by Yahoo!

The processing architecture is depicted:

Following this processing work, we expect to modify our crawling to harvest directly into HBase. The flexibility HBase offers will allow us to grow incrementally the richness of the terms indexed in the Portal, while integrating nicely into Hadoop based workflows. The addition of coprocessors to HBase is of particular interest to further reduce the latency involved in processing, by eliminating batch processing altogether.

The combination of Hadoop, Oozie and Hive offer a framework that we anticipate will fit nicely with many of our data transformation tasks, and Sqoop and Hive have made the technologies far more accessible to our development team than was  previously possible.

All GBIF source code is available under open source licensing, and this work is regularly blogged on the GBIF Developer Blog.

This procedure has been tested on the following configuration:

Assumptions:

Step 1: Download the Cloudera demo virtual machine from the ?Downloads? area of the Cloudera web site.

Click on the ?Download? link beside the ?Virtual Machine? section.

The file should automatically start to download. It will be saved to your ?Downloads? folder.

Step 2: Once the file has downloaded you now need to decompress it. The file that the virtual machine is contained within is a bz2 archive (specifically a bzip2 compressed archive) and can be decompressed with the finder.

Navigate to the ?Downloads? section, and either double-click on the file or right click on the file and then select ‘Open With/Archive Utility (Default). This will start the decompression/unarchiving process.

Step 3: When the archive utility has finished, there will be a folder named cloudera-demo-0.3.7. The contents of which are shown below:

Step 4: Start VirtualBox and create a new VM by clicking the ?New? button. The following is the first dialog for the new virtual machine.

Step 5: Once you click on the ?Continue? button you are presented with the following dialog.

Step 6: Give the new virtual machine a name, in this example we?ll be using ?Cloudera-CDH3?.

For Operating System and Version select ?Linux? and ?Ubuntu?.

Step 7: Increase the base memory to 1024 MB (if possible) for better performance.

Step 8: In this step we need to select the training VM file we just downloaded and extracted. Click on the second radio button ?Use existing hard disk? and then click on the file folder icon with the green ^ on the right hand side of the drop down.

Please Note: The contents of the drop down on your system will be different than that displayed here.

Navigate to the folder where the demo VM is located, in this example it should be in the ?Downloads? folder of the user and called ?cloudera-demo-0.3.7?.

Step 9: Click the ?Continue? button and the following dialog is now displayed and it shows a summary of the choices made so far.

Step 10: Click the ?Done? button to finish the creation of the virtual machine in VirtualBox 4. The following shows the newly created cloudera-CDH3 VM in the VirtualBox manager screen.

Step 11: Start the virtual machine. When the Demo VM launches you should be presented with the following login.

Step 12: Finally, once you have successfully logged into the Demo VM, the image below is the initial view that you should see.

About the author:

John Zanchetta heads up an integration test team in the mobile telecommunications space and is experimenting with CDH from a number of different perspectives. He has been hacking in various technology pools for the past ~20+ years. John can be contacted at johnzan at gmail dot com.

Klout’s goal is to be the standard for influence. The advent of social media has created a huge number of measurable relationships. On Facebook, people have an average of 130 friends. On Twitter, the average number of followers range from 300+ to 1000+. With each relationship comes a different source of data. This has created A LOT of noise and an attention economy. Influence has the power to drive this attention.

When a company, brand, or person creates content, our goal is to measure the actions on that content. We want to measure every view, click, like, share, comment, retweet, mention, vote, check-in, recommendation, and so on. We want to know how influential the person who *acted* on that content is. We want to know the actual meaning of that content. And we want to know all of this, over time.

Measuring influence is a bit like trying to measure an emotion like hate or jealousy. It’s really hard and takes a boatload of data.

A huge part of what we do is develop machine learning models that make sense of this data. On top of that, there’s an endless amount of this data and we need a platform to ingest, prepare, and analyze it.

The two biggest platforms are Facebook and Twitter, but it hardly ends there when it comes to social media. There’s LinkedIn, Foursquare, Path, Youtube, Quora, and many others. This presents the challenge of creating models for each platform and building data analysis platforms that can handle unstructured data.

To handle this at Klout, we’ve turned to open source technologies such as Hadoop. Specifically, we turned to Cloudera’s CDH3 distribution due to ease of installation and availability of enterprise support.

Twitter Influence

Twitter was the natural selection for our first network to analyze due to the open nature of the data as well as the simplistic nature of actions you can take on Twitter, such as a mention or a retweet.

However, as our models matured, the growth of Twitter increased. As of this post, our Twitter cluster has the following stats:

Twitter Analytics Overview

From the twitter firehose, data is written to disk in buffered chunks. A mapreduce job handles the task of preparing the firehose data into different buckets needed for each of the workflows. These different workflows serve different products from performing bot and spam detection to scoring to topic extraction.

Many of our mapreduce jobs are written in java, but we also rely on Pig Latin for some purposes such as performing simple joins are population aggregates and statistics.

Oozie is used to coordinate the different workflow components. To serve out data both internally and externally, we dump out raw csv files or load this data into HBase which interfaces with load balanced API servers.

Twitter Scoring Workflow

We use a machine learning and statistical based approach to perform our scoring. This model currently has over 35 features. The scoring workflow consists of different Oozie jobs, many of which perform feature extraction. In the final jobs of this workflow, all the features are fed into the scoring model, which produces scores.

We’ve experimented with Mahout in the past and we will be using more of it in the future.

Challenges

Having a highly available API is one of our key goals. However, when we refresh 75 million scores + meta data daily, it becomes challenging to flip a switch to make all the new data available. This led to us having multiple clusters. When one cluster is loading data, the load balanced API servers are aware of each cluster’s status, and switches to the non-loading cluster. This also mitigates any performance issues due to splits, minor and major compactions on the clusters. This also allowed us to cope with instabilities caused by cloud instances in unpredictable states.

That said, we are in the process of building out our own servers and racks at a nearby facility. We’ve also had issues where our edits logs for namenodes get corrupted due to server instabilities. This is where Cloudera has come to our rescue. We initially had to manually apply patches and build hadoop-core jars ourselves to resolve such problems, but with Cloudera’s Distribution including Apache Hadoop and their expert Solution Architects help this is no longer an issue. We now are able to focus our resources on our products.

I recently evaluated several serialization frameworks including Thrift, Protocol Buffersand Avro for a solution to address our needs as a demand side platform, but also for a protocol framework to use for the OpenRTB marketplace as well. The working draft of OpenRTB 2.0 uses simple JSON encoding, which has many advantages including simplicity and ubiquity of support. Many OpenRTB contributors requested we support at least one binary standard as well, to improve bandwidth usage and CPU processing time for real-time bidding at scale.

After reviewing many candidates, Apache Avro proved to be the best solution.

To demonstrate what differentiates Avro from the other frameworks (the link to my source code is at the end of this post), I put together a quick test of key features. The following are the key advantages of Avro 1.5:

* Schema evolution – Avro requires schemas when data is written or read. Most interesting is that you can use different schemas for serialization and deserialization, and Avro will handle the missing/extra/modified fields.

* Untagged data – Providing a schema with binary data allows each datum be written without overhead. The result is more compact data encoding, and faster data processing.

* Dynamic typing – This refers to serialization and deserialization without code generation. It complements the code generation, which is available in Avro for statically typed languages as an optional optimization.

Schema Evolution

This is the most exciting feature! It allows for building less decoupled and more robust systems. Below, I made significant changes to the schema, and things still work fine. This flexibility is a very interesting feature for rapidly evolving protocols like OpenRTB.

The following example demonstrates how this works. First, I created a new (example) schema. (Avro schemas are defined in JSON):

{
    "type": "record",
    "name": "Employee",
    "fields": [
        {"name": "name", "type": "string"},
        {"name": "age", "type": "int"},
        {"name": "emails", "type": {"type": "array", "items": "string"}},
        {"name": "boss", "type": ["Employee","null"]}
    ]
}

Next, I serialized a few records into a binary file using that schema. After that, I evolved my schema to the following:

{
    "type": "record",
    "name": "Employee",
    "fields": [
        {"name": "name", "type": "string"},
        {"name": "yrs", "type": "int", "aliases": ["age"]},
        {"name": "gender", "type": "string", "default":"unknown"},
        {"name": "emails", "type": {"type": "array", "items": "string"}}
    ]
}

This is a snapshot of the changes I made to the schema:

1) Renamed the field ‘age’ to ‘yrs’. Thanks to the alias feature, I can retrieve the value of ‘age’ by using the field name ‘yrs’.

2) Added a new ‘gender’ field, and defined a default value for it. This can be used to set values during deserialization as this field isn’t present in the original schema records.

3) Removed the ‘boss’ field.

Finally, I deserialized the binary data file with this new schema, and print it out. Success!

Untagged Data

There are two ways to encode data when serializing with Avro: binary or JSON. In the binary file, the schema is included at the beginning of file. I verified that the binary data was serialized untagged, which resulted in a smaller footprint. Another interesting point is that the schema can be defined, and then the data can be encoded/decoded in JSON; allowing you to define a schema for JSON rich data structures. Anyone needing to implement validation for a JSON protocol (like we did for OpenRTB) will appreciate this feature. And switching between binary and JSON encoding is simply a one-line code change. Switching JSON protocol to a binary format in order to achieve better performance is pretty straightforward with Avro.

Dynamic Typing

The key abstraction is GenericData.Record. This is essentially a set of name-value pairs where name is the field name, and value is one of the Avro supported value types. I found the dynamic typing to be very easy to use. When a generic record is instantiated, you have to provide a JSON-encoded schema definition. To access the fields, just use put/get methods like you would with any map. This approach is referred to as “generic” in Avro, in contrast to the “static” code generation approach also supported by Avro. The extra flexibility of the generic data handling has performance implications. But, this excellent benchmark – https://github.com/eishay/jvm-serializers/wiki/ – shows the penalty is minor, and the benefit is a simplified code base.

In conclusion, Avro is a unique serialization framework that works, although it took a bit of experimentation to get the code working. If you are interested in my Java code for an example of how Avro can be used, you can find it here: https://github.com/rfoldes/Avro-Test.

Robert Foldes

Senior Architect, DataXu