Background

The amount of memory available on a commodity server has increased drastically in tune with Moore’s law. Today, its very feasible to have up to 96 gigabytes of RAM on a mid-end, commodity server. This extra memory is good for databases such as HBase which rely on in memory caching to boost read performance.

However, despite the availability of high memory servers, the garbage collection algorithms available on production quality JDK’s have not caught up. Attempting to use large amounts of heap will result in the occasional stop-the-world pause that is long enough to cause stalled requests and timeouts, thus noticeably disrupting latency sensitive user applications.

Garbage Collection

The below is meant to be a quick summary of an immensely complex topic, if you would like a more detailed explanation of garbage collection, check out this post.

HBase, along with the rest of the Apache Hadoop ecosystem, is built in Java. This gives us access to an incredibly well-optimized virtual machine and an excellent mostly-concurrent garbage collector in the form of Concurrent-Mark-Sweep (CMS). However, large heaps remain a weakness, as CMS collects garbage without moving it around, potentially causing the free space to be spread throughout the heap instead of in a large contiguous chunk. Given enough time, fragmentation will require a full, stop the world, garbage collection with a copying collector capable of relocating objects. This results in a potentially long stop-the-world pause, and acts as a practical limit to the size of our heap.

Garbage collectors which do not require massive stop the world compactions do exist, but are not presently suitable for use with HBase at the moment. The Garbage-First (G1) collector included in recent versions of the JVM, is one promising example, but early testing still indicates that it exhibits some flaws. JVMs from other (non-Oracle) vendors which offer low-pause concurrent garbage collectors also exist, but they are not in widespread use by the HBase and Hadoop communities.

The Status Quo

Currently, in order to utilize all available memory, we allocate a smaller heap and let the OS utilize the rest of the memory. In this case, the memory isn’t wasted – it’s used by the filesystem cache. While this does give a noticable performance improvement, it has its drawbacks. Data in the FS cache is also treated as a file, requiring us to checksum, and verify the file.  We also have no guarantee what the FileSystem cache will do and have only limited control over the eviction policy of this cache. While the Linux FS cache is nominally a LRU cache, other processes or jobs running on the system may flush our cache, adversely impacting performance. The FS cache is better than putting the memory to waste, but it’s neither the most efficient, nor the most consistent solution.

Enter SlabCache

Another option would be to manually manage the cache within Java via Slab Allocation – opting to avoid garbage collection all together. This is the approach I implemented in HBASE-4027.

SlabCache operates by allocating a large quantity of contiguous memory, and then performing Slab Allocation within that block of memory. Buffers of likely sizes of cached objects are first allocated in advance – objects are fit into the smallest buffer available that can contain them upon caching.  Effectively, any fragmentation issues are internalized by the cache, trading off some space in order to avoid any external fragmentation issues. As blocks generally converge around a single size, this method can still be quite space efficient.

Implementation

While slab allocation does not create fragmentation, other parts of HBase still can. With Slab Allocation, the frequency of stop-the-world(STW) pauses may be reduced, but the worst case maximum pause time isn’t – The JVM can still decide to move our entire slab around if we happen to be really unlucky, contributing again to significant pauses. In order to prevent this, SlabCache allocates its memory using direct ByteBuffers.

Direct ByteByffers, available in the java.nio package, are allocated outside of the normal Java heap — just like using malloc() in a C program. The garbage collector will not move memory allocated in this fashion – guaranteeing that a direct ByteBuffer will never contribute to the maximum garbage collection time. The ByteBuffer “wrapper” is then registered as an object, which when collected, is released back into the system using free.

Reads are performed using a copy-on-read approach. Every time HBase does a read from SlabCache, the data is copied out of the SlabCache and onto the heap. While passing by reference would have been the more performant solution, that would have required some way of carefully tracking references to these objects. I decided against reference counting, as reference counting opens up the potential for an entirely new class of bugs, making continuing work on HBase more difficult.  Solutions involving finalizers or reference queues, were also discarded, as neither of them guarantee timely operation. In the future, I may decide to revisit reference counting if necessary to increase read speed.

SlabCache operates as an L2 cache, replacing the FS cache in this role. The on-heap cache is maintained as the L1 cache. This solution allows us to use large amounts of memory with a substantial speed and consistency performance over the status quo, while at the same time ameliorating the downsides of the copy-on-read approach. Because the vast majority of our hits will come from the on-heap L1 cache, we do a minimum of copying data and creating new objects.

Performance

SlabCache operates at around 3-4x the performance of the file system cache, and also provides more consistent performance.

Performance comparisons of the 3 caches as followed. In each test, each cache was configured so that it was the primary (L1), and only cache of HBase. YCSB was then run against HBase-trunk.

HBase in all cases was running in Standalone mode, compiled against 0.20-append branch. As HDFS has gotten faster since the last release, I’ve also provided tests with the RawLocalFS, in order to isolate the difference between accessing the Slab cache and accessing the FS cache by removing HDFS from the equation. In this mode, CRC is turned off, and the local filesystem (ext3) is used directly. Even given these optimal conditions, SlabCache still nets a considerable performance gain.

If you’d like to try out this code in trunk, simply set MaxDirectMemorySize in hbase-env.sh. This will automatically configure configure the cache to use 95% of the MaxDirectMemorySize, and set reasonable defaults for the Slab Allocator. If finer control is desired, you are free to change the SlabCache settings in hbase-site.xml, which will allow you to have finer control over off-heap memory usage and slab allocation sizing.?

Conclusion

If you’re running into read performance walls with HBase, and have extra memory to spare, then please give this feature a try! This is due to be released in HBase-0.92 as an experimental feature, and will hopefully enable the more efficient usage of memory.

I had an amazing summer working on this project, and as an intern, I’m awed to see this feature work and be released publicly. If you found this post interesting, and would like to work on problems like this, check out the careers page.

A year ago I rolled my first Hadoop system into production. Since then, I’ve spoken to quite a few people who are eager to try Hadoop themselves in order to solve their own big data problems. Despite having similar backgrounds and data problems, few of these people have sunk their teeth into Hadoop. When I go to Hadoop Meetups in San Francisco, I often meet new people who are evaluating Hadoop and have yet to launch a cluster. Based on my own background and experience, I have some ideas on why this is the case.

I studied computer science in school and have worked on a wide variety of computer systems in my career, with a lot of focus on server-side Java. I learned a bit about building distributed systems and working with large amounts of data when I built a pay-per-click (PPC) ad network in 2004. The system is still in operation and at one point was handling several thousand searches per second. As the sole technical resource on the system, I had to educate myself very quickly about how to scale up.

As I contemplated how doomed I would be should traffic levels increase much more, I remember wondering to myself, “How does Google deal with all that data?” The answer came to me in the form of the Google File System (GFS) paper and later the MapReduce paper, both from Google. It dawned on me that because Google was forced to solve a much larger problem, they had come up with an elegant solution for a whole range of more modest data problems running on commodity hardware. But it wouldn’t be until 2010 that I would get to work with this technology firsthand.

As I wrote in an earlier article, I started re-architecting USASearch, the U.S. government’s search system, in 2009 based on a solution stack of free, open source software including Ruby on Rails, Solr, and MySQL. A wave of déja vu hit me as I started worrying about what to do with the growing mountain of data piling up in MySQL and our increasing need to analyze it in different ways. I had heard that a new company called Cloudera, founded by some big data people from Yahoo!, Google, and Facebook, was making Hadoop available for the masses in a reliable distribution, much in the same way that RedHat did for Linux. Curiosity got the best of me and I bought the newly minted Hadoop: The Definitive Guide from O’Reilly. The most insightful part of the book to me was the very first sentence. It’s a quote from Grace Hopper: “In pioneer days, they used oxen for heavy pulling, and when one ox couldn’t budge a log, they didn’t try to grow a bigger ox.” I didn’t want to grow a bigger server; I wanted to harness a bunch of small servers together to work in unison. The more I learned the more curious I got, so I started reading more. And that’s when I hit my first roadblock.

I think people who have been working with Hadoop technologies for years and years sometimes forget just how rich and diverse the big data software ecosystem has become, and how daunting it can be to folks approaching it for the first time. When people at the Meetups say they are evaluating solutions to their data scaling problem, the answers they hear sound something like this: “Just use Hadoop Hive Pig Mahout Avro HBase Cassandra Oozie Sqoop Flume ZooKeeper Cascading NoSQL RCFile. Oh, almost forgot…cloud.”

The thought of wading through all of that just to learn about what I needed to learn about was a bit too overwhelming for me, so I put the whole matter aside for a few months. Over time, I started to dive into each of these projects to understand the primary use case, how active the developer community was and which organizations were using it in production. I converged on the idea of using Hive as a warehouse for our data. I opted for Cloudera’s distribution since I wanted to reduce the risk of running into compatibility issues between all the various subsystems. Having tracked down anomalies in a highly multi-threaded and contentious distributed Java system before, I liked the idea of someone else taking on that problem for me.

At some point, I had read everything I could read and grew impatient to get my hands dirty, so I decided to just download CDH3 on my laptop and give it a try. The tutorial instructions for the standalone version worked, and I successfully computed more digits of pi than I ever thought I’d need. After creating some sample data in Hive and running a few queries, I felt pretty confident that Hive would be the right tool for the job. I just needed to find somewhere to install and run HDFS (namenode, secondary namenode, and data nodes), Hadoop (jobtracker and tasktracker nodes), Hive, and Hue for a nice front end to it all.

I knew from my past experience how to stretch the limits of CPU, disk, IO, and memory on commodity servers, and I identified a few potential servers at our primary datacenter with resources I figured I could leverage. Once again I followed the tutorial instructions, this time for the fully distributed version of CDH3, and once again I started to compute pi. And that’s when I hit my second roadblock. It took me a few days to figure out that I had a problem with DNS. Each machine needs to be able to resolve every other machine’s name and IP in the cluster. Whether you do that via /etc/hosts or a local DNS server is up to you, but it needs to happen or the whole thing gets wedged. Once I got that sorted out, everything just started falling into place and I had Hive working in production within a few days. A week later, I started pulling out the MySQL jobs and deleting big tables, and that’s been the trend ever since.

Over time, I’ve gone on to learn about using custom Ruby mappers in Hive, moving data back and forth between MySQL and Hive with Sqoop, and getting the data into HDFS in real-time with Flume. All of these components from the Cloudera distribution are working nicely in our production environment now, and I sleep well at night knowing I have such a solid, deliberate plan for growth. My initial investment in learning about the Hadoop ecosystem is really paying dividends, but when I think about all those people at the Meetups stuck in evaluation mode, I feel their pain. Does it have to be such a struggle?

The big challenge in my opinion is not that any one piece of the puzzle is too difficult. Any reasonably smart (or in my case stubborn) engineer can set themselves on the task of learning about a new technology once they know that it needs to be learned. The challenge with the Hadoop ecosystem is that it presents the newbie with the meta-problem of figuring out which of these tools are appropriate for their use case at all, and whether or not to even consider the problem today versus deferring it until later. In a way Facebook has it easy, because when you are adding 15TB of data per day, that decision is pretty much made for you.

For all the companies sitting in the twilight between the gigabyte and the petabyte who don’t have Hadoop expertise in-house, there is a collection of free information to help guide people to the right solution space (Hadoop Tutorial, White Papers). These days, when I talk to people who are evaluating solutions to their big data problems, my advice to them is to break down their problems into a few discrete use cases and then work on ferreting out the technologies that are designed for that use case. Get a proof of concept to demonstrate that the technology can address your use case and convince yourself and others that you’re on the right track. Work toward putting something simple into production. Lather, rinse, and repeat. I am still in that cycle myself, as these days I’m exploring HBase and OpenTSDB to give me low-latency access to time series data and Mahout to do frequent item set mining, but that’s another article for another day.

The project

My intern project was to build a log searching tool, specialized for Apache Hadoop. My mini-app allows Hadoop cluster admins and operators to search their error logs across many machines, filter by time range, text in the log message, and find the namenode machine, for example. The results are then ordered by time, and shown to the user.

This project was inspired by the extreme wizardry required to search logs with traditional tools, such as grep and ssh (or parallel ssh), especially since these tools do not order the results by time. Ordering by time is very important, as it allows one to triage the sources of failures across your cluster, and figure out where it all started.

How do I feel about my project in retrospect?

I had a ton of independence when it came to building my app. As part of the Enterprise Team, which wrote Cloudera Manager allowing one to easily deploy Hadoop to a whole cluster within a few clicks, I wrote the REST API available on the individual machines of the cluster. I wrote the master server code that makes requests in parallel to each of the cluster machines, asking for their search results. I designed and wrote the UI for the app, in addition to conceiving ways to make life easier for users who interact with it (with some help from user-testing, of course).

A couple of the niceties I added to the search page include:

• Search without page refresh.
• Saves the current search’s options to the URL bar, so that if you send the URL to a fellow admin, they can run the exact same search and see exactly what you’re seeing. Or you can save the URL and re-run the search at a later date.
• A context view that does the same thing as the above, but lets you browse a single log file, with pagination (a single day’s log file can get as big as 1GB, so it wouldn’t be a great idea to send it all to the client at the same time).

In this process I learned more about python, the difficulty that python’s built-in date capabilities can cause, and how it can be quite fun to run code distributed across hundreds or thousands of machines.

I also spent some time profiling the internal Cloudera log searching library (written by Adam Warrington) which is the workhorse of the REST API (the master server communicates with its minions over HTTP). We were able to cut the worst-case run time on sample data by ~88%, which made me happy. During the process, I learned when possible, it’s best to meta-program other people’s code by asking them to make it faster, as the process of learning and reading all the code they’ve written can take some time, especially when you only need to make what you hope is a small change. It’s really great to arrive at work and hear that someone else has just finished coding up the optimizations your app needed.

Technical Portion: how the log search feature works.

Whenever you run a search, the main page of the search UI makes a request to a JSON endpoint, asking for log search results from, say, yesterday on all datanodes in the cluster. This request reaches the master server (SCM), which knows all about the machines in the cluster. The master server has a number of threads which make requests in parallel to each of the applicable cluster machines, each of which exposes a JSON endpoint. Each individual cluster machine then runs some python code that searches the relevant log files and returns the result as JSON. The master server collates the results from each cluster machine, and returns this to the browser. The results are then displayed to the user in what can sometimes be a very long list.

We decided it would be best not to maintain an index on the log files, as there can be many terabytes of data to sift through. For this reason, searches are done on demand by each individual cluster machine. Searches which include a time range are quite fast, as binary search is used to find the relevant time range, and then the first 20 results are returned. We also made an effort to optimize searches similar to grep helloworld that filter out certain words when we scan the particular line for the word, and skip the line without parsing it into an event if that line does not contain helloworld. We made this optimization because parsing each log event into date, message, and source was quite slow when searching large files.

Because I wrote the three components that make the search work (UI, JSON route on master server, JSON route on cluster machines), I got a good overview of many aspects of the code base.

A brief overview of the indirectly-dev related skills I learned a Cloudera

I’ve learned git really well, which I totally love now. I can rebase, cherry-pick, :/search, and reflog like the best of them. My git skills could be considered quite fetching among certain branches of society.

While here I’ve also had the opportunity to really flesh out my dotfiles, especially my .gitconfig and my .profile (aka .bashrc).

I also got a real feel for Dojo while I’ve been here, and I’d say that my next choice of javascript toolkit will be much better informed because of this.

Code reviews kept my code quality up, helping me catch little things like comments that were no longer relevant. I have yet to master the art of spotting and veto-ing changes that will break what I’ve written. Maybe next year!

Dev environment

From the web developer’s point of view, there’s are a couple times when you’d normally need to do a couple extra alt-tabs and/or refreshes. Of course, I’m pampered because I’ve never had to build anything that goes into production in a massive and/or distributable way, as we do here at Cloudera with our management tools. Also, I’ve traditionally done my work with web-servers that don’t require one to compile source code.

I found the following things quite annoying about the dev environment:

1. static files, when changed, would not show their changes upon refresh
2. .less files, an advanced and superior alternative to CSS, would not
   auto-recompile upon change

A solution to the first one was found by a coworker. I fixed the second problem by writing a little node script that watches less files and recompiles when any of them change.

While I was at it, I also made it easy for devs to use vogue, which reloads stylesheets whenever they’ve changed, without requiring a page refresh. This further improves development, as pages can get quite heavy when in development mode, where every javascript file is loaded individually, and it’s nice to have CSS changes automatically reflected in the UI.

Thanks Cloudera!

That’s about all I have on my mind when it comes to my internship. I learned a ton, enjoyed the free lunches a lot, as well as the 30 inch monitors. These things make a big difference, and also make me feel way cooler than some of my friends who don’t get these things.

So long Cloudera and thanks for all the fish! Now I’m off to another planet for a year in the world of academia!

David Trejo
Software Engineer Intern
Cloudera Summer ‘11 Enterprise Team
Brown University Computer Science ‘13

Go to Cloudera Careers >

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Abstract: CBS Interactive successfully adopted Hadoop as the web analytics platform, processing one Billion weblogs daily from hundreds of web site properties that CBS Interactive oversees. After introducing Lumberjack—the Extraction, Transformation and Loading framework we built based on python and streaming, which is under review for Open-Source release—Michael will talk about web metrics processing on Hadoop, focusing on weblog harvesting, parsing, dimension look-up, sessionization, and loading into a database. Since migrating processing from a proprietary platform to Hadoop, CBS Interactive achieved robustness, fault-tolerance and scalability, and significant reduction of processing time to reach SLA (over six hours reduction so far).

Gateway: Cluster Virtualization Framework
Konstantin Shvachko, eBay

Abstract: Access to Hadoop clusters through dedicated portal nodes (typically located behind firewalls and performing user authentication and authorization) can have several drawbacks — as shared multitenant resources they can create contention among users and increase the maintenance overhead for cluster administrators. This session will discuss the Gateway system, a cluster virtualization framework that provides multiple benefits: seamless access from users’ workplace computers through corporate firewalls; the ability to failover to active clusters for scheduled or unscheduled downtime, as well as the ability to redirect traffic to other clusters during upgrades; and user access to clusters running different versions of Hadoop.

SHERPASURFING – Open Source Cyber Security Solution
Wayne Wheeles, Novii Design

Abstract: Every day billions of packets, both benign and some malicious, flow in and out of networks. Every day it is an essential task for the modern Defensive Cyber Security Organization to be able to reliably survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform. SHERPASURFING is an open source platform built on the proven Cloudera’s Distribution including Apache Hadoop that enables organizations to perform the Cyber Security mission and at scale at an affordable price point. This session will include an overview of the solution and components, followed by a demonstration of analytics.

Integrating Hadoop with Enterprise RDBMS Using Apache SQOOP and Other Tools
Arvind Prabhakar, Cloudera
Guy Harrison, Quest Software

Abstract: As Hadoop graduates from pilot project to a mission critical component of the enterprise IT infrastructure, integrating information held in Hadoop and in Enterprise RDBMS becomes imperative. We’ll look at key scenarios driving Hadoop and RDBMS integration and review technical options. In particular, we’ll deep dive into the Apache SQOOP project, which expedites data movement between Hadoop and any JDBC database, as well as providing an framework which allows developers and vendors to create connectors optimized for specific targets such as Oracle, Netezza etc.

Next Generation Apache Hadoop MapReduce
Mahadev Konar, Hortonworks

Abstract: The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale, high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. We will be presenting the architecture and design of the next generation of map reduce and will delve into the details of the architecture that makes it much easier to innovate. We will also be presenting large scale and small scale comparisons on some benchmarks with MRV1.”

There are several training classes and certification sessions provided surrounding the Hadoop World conference. Don’t forget to register and become Cloudera Certified in Apache Hadoop.

When I started my internship at Cloudera, I knew almost nothing about systems programming or Apache Hadoop, so I had no idea what to expect. The most important lesson I learned is that structured data is great as long as it is perfect, with the addendum that it is rarely perfect.

My project was to develop a unified view of our customer data. The requirements were simple: pull in data from a variety of systems, group it by customer, and display it. The goal is that when someone at Cloudera needs to see all of the key information about our customers, it is available in one place. In addition, downloading and grouping data will make performing analysis much easier, allowing us to draw new insights about our business and our customers.

I started by writing a script for each data source to download the necessary data and insert it into an HBase table in raw form. Next I wrote a script for each data source that grouped the data by customer, possibly transformed the data (filtering, sorting, inserting child objects within their parents, etc) and inserted it into a separate HBase table where each row corresponds to a single customer, with a column family for each data source. Finally, I exposed the data on an internal website using Django, integrating the different data sources as much as possible.

One of the big challenges was turning the raw data into meaningful information. There were several discrepancies I needed to address with the data. As an example, companies have different names in different systems. Sometimes the difference is simply a matter of capitalization and/or spacing, but a greater challenge is that sometimes abbreviations were used in one system but not another, or one system ended the name with “, inc.” but another did not. I considered using fuzzy matching to solve all of these problems and realized that the Levenshtein distance between “Cloudera” and “Cloudera, inc.” is quite high, so I started looking at other forms of fuzzy matching and thinking of developing one that in particular favored long identical sub-strings. For example, I wanted my algorithm to see “Cloudera” and “Cloudera, inc.” as being highly similar for sharing the whole “Cloudera” part. As I contemplated embarking on a task to which I could have easily devoted the whole summer, I realized that I was heading down a rabbit hole. I took a step back and determined that trying to solve this problem in a fully automated way was not worth my time. It would have been time consuming, and it would have only made my problems worse; I still would have had to deal with names that should be merged but weren’t (since no scheme could perfectly determine if two names represent the same customer), but I also would have had to worry about names that shouldn’t have been merged but were. Why would I devote time to building a complex matching algorithm that doubled the number of problems I had to deal with?

Instead, I created an alias table in HBase. The key is the customer name, with white-space removed and letters lower-cased to catch the easiest cases. One of the columns contains a UUID that is used as the key for that customer throughout the rest of the system. When my transform scripts move data from the raw table to the table where each row is a customer, they use the alias table to determine into which row to insert the transformed data. When my code merges two customers, it merges the current data and makes all alias entries that were pointing to either row point to the newly merged row, so that when the scripts next load new data into the table, they put it directly into the correct location. This approach does require manual intervention (in practice, all schemes were going to), but at least it was simple. This was an important lesson from my internship; I learned that some problems aren’t worth solving.

Another major issue I had to tackle was cases where data was incorrect or incomplete. Our opportunity data, for example had various fields that were not used when the sytem was first configured. For example, contract terms were always a set period of time. Some fields such as the product quantity were changed so older records had a value but in different units than the units used in new records. For this tricky data rather than simply reading the values directly I wrote helper methods to return the value, sometimes trying 4 or 5 different ways to infer the actual value. For example, the contract end date, I had to base the value on close date about half the time. In this case one available helper method returned a tuple of the value that it was trying to infer (the end date) and a Boolean value representing whether the value returned was explicitly specified or inferred from another field. In the web view I used the explicit or inferred flag to add an annotation in the UI indicating when the value was approximated. Users could then look elsewhere if precision was important. This annotation also exposes where source data is missing fields, which can help us update the source data.

Businesses are always changing, so the data that a business decided to keep track of two years ago may not be the same data that makes sense to keep track of today. Many of the values I was trying to infer from data were from fields that we added to our system. The old data still lacked meaningful values because no one had gone back and fixed all of the past data. Each data source having a “name” field seems great until you realize that none of the names are the same, and having a field representing the exact information you want is great until you realize that it’s null in 40% of cases.

Another challenge was that I needed to interact with many different APIs, each with their own quirks, and similarly, I had to use various libraries to parse the different kinds of data and handle the transformations. I always expected APIs and libraries to be perfectly documented and to be designed with my usecase in mind. This was rarely the reality, so these tasks frequently took much longer than I expected. I knew from past experience that building good software always takes longer than planned, but it seemed even more true this summer. I realized that one factor contributing to this was that my whole project was centered around touching as many different “things” as possible. Each time I integrated with a new library, API, or existing codebase, there was always an additional cost of figuring out how to approach it. Additionally, there was always a chance that some aspect of the new process would not work as advertised or not provide a direct, optimized way for me to do what I needed. Significantly underestimating the difficulty of every single piece of my project helped me improve my ability to make those estimations.

Overall, my internship at Cloudera was amazing. I got to work with very smart people building and shipping high quality software using HBase. I got to sit next to, eat lunch with, and hear internal talks by people working on an array of fascinating things, happy to share knowledge and advice. I saw how a software company operates and caught glimpses into how diverse companies—Cloudera’s customers—operate. At the beginning of my summer I’d thought my project was going to be a series of unexciting tasks, many of which I’ve done before. As it turned out I encountered some very interesting problems and a lot of good lessons to carry forward.

Find available opportunities via theCloudera Careers web page.

One of the cool activities happening at the conference was live streaming video brought to us from the good folks at SiliconAngle. Using a mobile production system called The Cube, Silicon Angle hosts John Furrier (@furrier) and Dave Vellante interviewed industry luminaries and up and comers while bringing their own perspective. After streaming live for nearly two days these hosts are still able to keep the energy high and the tone light.

In the interviews below John and Dave interview Amr Awadallah, CTO and Co-Founder of Cloudera (@awadallah), and John Kreisa, VP Marketing at Cloudera (@marked_man); followed by a John and Dave interview with Sarah Sproehnle director of education at Cloudera. During the interviews they cover many different aspects of Cloudera and Apache Hadoop.

Interview 1

Interview 2

Here are the top 10 blog posts from 2010:

1. How to Get a Job at Cloudera
   Cloudera is hiring around the clock, and this blog highlights the best course of action to increase your chances of becoming a Clouderan.
2. Why Europe’s Largest Ad Targeting Platform Uses Hadoop
   “As data volumes increased and performance suffered, we recognized a new approach was needed (Hadoop).” –Richard Hutton, Nugg.ad CTO
3. What’s New in CDH3b2 Flume
   Flume, our data movement platform, was introduced to the world and into the open source environment.
4. What’s New in CDH3b2 Hue
   Hue, a web UI for Hadoop, is a suite of web applications as well as a platform for building custom applications with a nice UI library.
5. Natural Language Processing with Hadoop and Python
   Data volumes are increasing naturally from text (blogs) and speech (YouTube videos) posing new questions for Natural Language Processing. This involves making sense of lots of data in different forms and extracting useful insights.
6. How Raytheon BBN Technologies Researchers are Using Hadoop to Build a Scalable, Distributed Triple Store
   Raytheon BBN Technologies built a cloud-based triple-store technology, known as SHARD, to address scalability issues in the processing and analysis of Semantic Web data.
7. Cloudera’s Support Team Shares Some Basic Hardware Recommendations
   The Cloudera support team discusses workload evaluation and the critical role it plays in hardware selection.
8. Integrating Hive and HBase
   Facebook explains integrating Hive and HBase to keep their warehouse up to date with the latest information published by users.
9. Pushing the Limits of Distributed Processing
   Google built a 100,000 node Hadoop cluster running on Nexus One mobile phone hardware and powered by Android. The environmental cost of this solution is 1/100^th the equivalent of running it within their data center. (April Fools)
10. Using Flume to Collect Apache 2 Web Server Logs
    This post presents the common use case of using a Flume node to collect Apache 2 web server logs and deliver them to HDFS.

Aside from How to Get a Job at Cloudera, Cloudera blog readers viewed posts related to CDH and its components, posts exemplifying possibilities with Hadoop in production, and posts highlighting integrations with Hadoop.

Looking forward we plan to continue to feature technical and non-technical topics, as well as guest posts from customers and the community, and plan to increase our number of published posts. If there is a topic you would like to learn more about, or you have a Hadoop story you would like to share we would love to hear your ideas. Email suggestions to community@cloudera.com.

Cloudera Corporate & a chicken, aka Cloudera engineers at their finest.

Hadoop came and joined the festivities dressed in a very realistic pumpkin costume. To keep Hadoop’s pumpkin costume up to earthquake code specifications, several paper-clips were deemed necessary in the pumpkin’s structure.

The Cloudera fun continued into the weekend as part of the Cloudera team gathered October 30th in Hollister, CA to take part in the Northern California Warrior Dash.

Cloudera has a very friendly and accepting atmosphere, filled with great people that are passionate about their work. If this sounds like an environment you would like to be a part of check out the careers page on our website. There are numerous job openings here at head quarters (Palo Alto), along with rumors of a San Francisco office to be coming in the very near future. Check out the careers page to learn more.

“I also think we have a better kitchen. We’ve put a lot of effort into building a really powerful infrastructure at Google, the development environment at Google is very good.”

The goal of the Kitchen team at Cloudera is to create a powerful infrastructure for developing, building, testing, shipping, and supporting our software. Kitchen contributes its expertise to every product Cloudera builds, while also building out new infrastructure and tools to facilitate future development. Everyone on the Kitchen team writes software.

While the Kitchen team’s culture was initially inspired by Google’s infrastructure, we agree with Piaw Na who recently provided some words of caution for companies looking to follow this example:

“In short, I think startups have to be very careful about building generic infrastructure just because that’s the way Google did things.”

The Kitchen team builds the infrastructure that is needed to solve our company’s problems. For example, our build system must be capable of coalescing many disparate open source projects into a unified platform. If there is an existing open source tool or framework that meets our needs we use it, improve it, and contribute it back to the project rather then “rolling our own”

We use many of the open source tools you might expect, such as Hudson for continuous integration. Our Hudson instance manages tens of hosts running over seventy projects:

• Unit tests running on every commit, across multiple platforms, and flavors of Java or Python
• Hadoop clusters running on EC2 using Apache Whirr
• Various code improvement tools such as jcarder, Cobertura, Clover, FindBugs, CheckStyle and others

If a tool does not exist the Kitchen team tries to leverage existing frameworks to build what is required. For example, our automated build and release system, which is at the heart of the Cloudera Distribution for Hadoop (CDH) platform, is built on top of boto. From a single git repository, we use crepo (another Kitchen project) to check out the latest source of each project within CDH. Then we build source artifacts for all of the projects, which get uploaded to S3. We then spin up an EC2 cluster to build everything for all the supported CentOS releases, Ubuntu, and Debian releases, including both 32 and 64-bit architectures. The resulting packages are stored back in S3, and then staged to a fresh EC2 instance of archive.cloudera.com for testing. Additional EC2 instances follow and run end-to-end package tests for each package that was built. We turn the crank nightly, not just for each release.

The Kitchen team is in the process of building a status, dashboard, radiator, single-pane-of-glass to prominently display Hudson’s status, nightly builds, JIRA stats, CDH download statistics, and many other metrics we use daily.

No software company is complete without a cluster or two. Kitchen maintains a development cluster, a long-lived CDH cluster, a security-enabled CDH cluster, and a “dog-food” cluster. We’re currently building out a Eucalyptus cluster so we can also run our build and test infrastructure in house. We have a large scale cluster in the works and we are busy building out our infrastructure to accommodate it.  We use Cobbler, run Ganglia (bias alert, we employ one of the original authors), debate Chef and Puppet.

Our Kitchen team is growing. If this sounds like a team you would like to be a part of, get in touch with me on twitter or IRC (#cloudera on freenode.net) or apply directly. Stay tuned for more blog posts about what’s cooking in our Kitchen.

Image courtesy of Chef Olive at Kitchen On Fire

I’ll begin the post, though, by telling you what doesn’t work. Several times a day, we get an unsolicited email or phone message from a contingency recruiter like this one:

I specialize in the industry and wanted to contact you to let you know that I have a strong candidate for your [deleted] position, and wanted to know if you would like to review the resume that I have? My candidate is interested in interviewing as soon as possible.

If you’re that candidate, bad news: Your resume goes straight to the bottom of the pile, and it’s a big pile.

Contingency recruiters get paid by the hiring company when the candidate they introduce gets the job. The amount they get paid varies a little bit, but is generally a quarter to a third of the annual salary of the person they place. That means that a software developer who earns $80K a year shows up with a $20,000 price tag, minimum, due and payable to the recruiter, up front. If you’re that software developer, you need to be not merely as good as the rest of the candidates we’re looking at. It’s not even enough to be a little bit better. You need to be so astonishingly good that you’re worth our writing a fat check to a third party on your very first day in the office. And, honestly, if you’re that good, how come we don’t know about you already?

Now, we do — occasionally, for a few positions that are especially tough to fill — work with contingency search firms. If a recruiter talks to you about Cloudera, you should ask: Do you have a signed engagement letter with Cloudera already? If the answer is yes, then you’re talking to someone we trust (but whom we’ll have to pay if we hire you, so we’d still prefer to hear from you directly). If the answer is no, or if they waffle, then you shouldn’t waste your time with them. They’re not going to help you get a job at Cloudera.

If you want a job with us, we absolutely want to talk to you. Here’s some advice on how to reach us on your own.

First, don’t just email us your resume with a cover letter. It’s not that we don’t read those; it’s that it’s very, very hard to stand out from the crowd that way. Every company that’s hiring and posts jobs on its web site gets a lot of resumes by email. Reading all of them is hard work, and (shame on us) we might be a little bit hurried or distracted when yours comes in.

It’s much better to come in by way of an introduction from someone we already know. We know all the people who work here very well, of course. If one of them sends your name and resume along to a hiring manager, you can bet that it gets special attention. It turns out that Cloudera people are easy to find: We speak at conferences, attend trade shows, hang out in IRC and browse public forums about topics that matter to the company. We blog here, and many of us post on Twitter regularly — see @cloudera/cloudera. Check out our events page. Look for us on-line or in person at shows. Reach out, person to person, in those places.

You should really try to engage with the Cloudera person, of course. Don’t just ask for an intro; talk to us us about the topics that you know matter to us. Show us, first, that you know a little bit about what we’re doing. If you’re a developer especially, making contributions to the open source projects we work on, or building cool applications on Cloudera’s Distribution for Hadoop, is a great way to show your chops.

We’ve got a referral program in place — when an employee brings us a candidate we hire, that person gets some extra Cloudera stock (we would much rather grant equity to our employees than write checks to recruiters!). As a result, if you’re a good fit for a position and can establish a personal rapport with one of our current employees, you’ll have an enthusiastic champion who’ll make sure we take a hard look at you. Everybody wins.

Before you reach out, though, do your homework. Figure out what we do. We put a lot of work into the web site — read the stuff there. Understand our products and customers. Know the role you’re interested in and why you’re a good fit. It’s really surprising to me how many people send us unsolicited resumes asking if we have any jobs open that would match their backgrounds. If you can’t do simple homework, you’re just not the kind of person we’re going to hire.

If you can’t get to us directly, take a look at our customers, partners and investors. Do you have contacts there? Personal introductions really do get noticed, and your effort in making them happen demonstrates, all by itself, that you’re motivated and clever and willing to do a little extra work.