“I also think we have a better kitchen. We’ve put a lot of effort into building a really powerful infrastructure at Google, the development environment at Google is very good.”

The goal of the Kitchen team at Cloudera is to create a powerful infrastructure for developing, building, testing, shipping, and supporting our software. Kitchen contributes its expertise to every product Cloudera builds, while also building out new infrastructure and tools to facilitate future development. Everyone on the Kitchen team writes software.

While the Kitchen team’s culture was initially inspired by Google’s infrastructure, we agree with Piaw Na who recently provided some words of caution for companies looking to follow this example:

“In short, I think startups have to be very careful about building generic infrastructure just because that’s the way Google did things.”

The Kitchen team builds the infrastructure that is needed to solve our company’s problems. For example, our build system must be capable of coalescing many disparate open source projects into a unified platform. If there is an existing open source tool or framework that meets our needs we use it, improve it, and contribute it back to the project rather then “rolling our own”

We use many of the open source tools you might expect, such as Hudson for continuous integration. Our Hudson instance manages tens of hosts running over seventy projects:

• Unit tests running on every commit, across multiple platforms, and flavors of Java or Python
• Hadoop clusters running on EC2 using Apache Whirr
• Various code improvement tools such as jcarder, Cobertura, Clover, FindBugs, CheckStyle and others

If a tool does not exist the Kitchen team tries to leverage existing frameworks to build what is required. For example, our automated build and release system, which is at the heart of the Cloudera Distribution for Hadoop (CDH) platform, is built on top of boto. From a single git repository, we use crepo (another Kitchen project) to check out the latest source of each project within CDH. Then we build source artifacts for all of the projects, which get uploaded to S3. We then spin up an EC2 cluster to build everything for all the supported CentOS releases, Ubuntu, and Debian releases, including both 32 and 64-bit architectures. The resulting packages are stored back in S3, and then staged to a fresh EC2 instance of archive.cloudera.com for testing. Additional EC2 instances follow and run end-to-end package tests for each package that was built. We turn the crank nightly, not just for each release.

The Kitchen team is in the process of building a status, dashboard, radiator, single-pane-of-glass to prominently display Hudson’s status, nightly builds, JIRA stats, CDH download statistics, and many other metrics we use daily.

No software company is complete without a cluster or two. Kitchen maintains a development cluster, a long-lived CDH cluster, a security-enabled CDH cluster, and a “dog-food” cluster. We’re currently building out a Eucalyptus cluster so we can also run our build and test infrastructure in house. We have a large scale cluster in the works and we are busy building out our infrastructure to accommodate it.  We use Cobbler, run Ganglia (bias alert, we employ one of the original authors), debate Chef and Puppet.

Our Kitchen team is growing. If this sounds like a team you would like to be a part of, get in touch with me on twitter or IRC (#cloudera on freenode.net) or apply directly. Stay tuned for more blog posts about what’s cooking in our Kitchen.

Image courtesy of Chef Olive at Kitchen On Fire

We’ve come a long way to get to a production quality release. At the beginning of September we announced the first beta of CDH2. After 6 months of additional testing we announced a release candidate. The release candidate spent over a month hardening in Cloudera’s internal QA process and on a wide variety of customer clusters. CDH2 is now stable and ready for use – we are pleased to recommend it to all our production users.

CDH2 is based on Apache Hadoop 0.20 – a release that has been available for almost a year. During this time, the Apache Hadoop community has produced hundreds of bug fixes, improvements and features. Cloudera is proud to have contributed many of these and incorporated them into CDH2.  For more information, please review the following resources:

• The release notes for CDH2. All bug fixes and improvements are covered in detail.
• For new features you’ll want to checkout CDH3, which is now in beta.
• For how to get started, please have a look at CDH documentation, which includes a helpful bit on determining which version is right for you.

Hadoop is a community effort. We’d like to thank everyone who contributes to Hadoop, especially the substantial contribution made by the big team at Yahoo! and all the other users who have contributed to this release. We appreciate the feedback on Get Satisfaction, twitter and IRC (#cloudera on freenode.net). Keep it coming, and thanks for using Cloudera’s Distribution for Hadoop!

• Hadoop 0.20.1 – 73 more patches of extra Hadoop’y goodness (that is 225 total patches over vanilla 0.20.1)
• Lots of libhdfs and fusefs love resulting in stability and usability improvements currently in use at scale
• HDFS fixes that improve the write pipeline
• Lots of general stability fixes for Hadoop
• Pig 0.5.0 release – Working out of the box with our Hadoop 0.18 and 0.20 builds
• Hive 0.4.1 release – Works with both of our Hadoop 0.18 and 0.20
• HBase 0.20.3 – We worked with the HBase team to bring the latest rpms to a yum repo near you

We are excited about our CDH2 release. Its running at scale at some really great companies. We are looking forward to promoting it to stable shortly and moving on to the next big thing, CDH3. I’ll let you know as soon as this happens. When CDH2 becomes stable, it also means that CDH3 is ready to start its journey through testing. Stay tuned for more details as to what CDH3 will encompass; I’ll just say that I’m pretty excited about it.

You can subscribe to our CDH mailing list (cdh-announce-subscribe@cloudera.com) to get information about new releases as we push them out. Check out the new release, and remember to let us know what you think!

We plan on pushing new packages into the testing repository every 3 to 6 weeks.  And it just so happens it is just about 3 weeks after we announced the first testing release. So it must be time for a new one. Here are some of the highlights:

• Hadoop 0.20.1 – Bumps the hadoop package up to the 0.20.1 release and adds 133 patches worth of extra goodness
• Alternatives for Hadoop – Now you can have both 0.18 and 0.20 installed and use the alternatives system to pick a default
• Pig 0.50 pre-release – We included some magic to get things working out of the box with both 0.18 and 0.20
• Hive 0.40 pre-release - Integrated with the alternatives setup out the box works with 0.18 and 0.20
• HBase 0.20 – We worked with the HBase team to bring rpms to a yum repo near you

A project as large as Hadoop is a communal effort. Cloudera is proud to be part of that community and hope that our products and services make Hadoop even more accessible to a wider audience. We’d like to thank everyone who contributes to Hadoop, especially the Yahoo! team for all of their hard work on getting 0.20.1 released, the developers at Facebook and those working on Pig, Hive and HBase.

We are just getting the ball rolling here. You can subscribe to our CDH mailing list (cdh-announce-subscribe@cloudera.com) to get information about new releases as we push them out. Check out the new release, and remember to let us know what you think!

One of the more common requests we receive from the community is to package HBase with Cloudera’s Distribution for Hadoop. Lately, I’ve been doing a lot of work on making Cloudera’s packages easy to use, and recently, the HBase team has pitched in to help us deliver compatible HBase packages. We’re pretty excited about this, and we’re looking forward to your feedback. A big thanks to Andrew Purtell, a Senior Architect at TrendMicro and HBase Contributor, for leading this packaging project and providing this guest blog post. -Chad Metcalf

What is HBase?
HBase is an open-source, distributed, column-oriented store modeled after Google’s Bigtable large scale structured data storage system. You can read Google’s Bigtable paper here.

“Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from back end bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products.”

HBase extends the publicly shared aspects of the Bigtable architecture and design as described in the Bigtable OSDI’06 paper with community developed improvements and enhancements:

• Convenient base classes for backing Hadoop MapReduce jobs with HBase tables
• Query predicate push down via server side scan and get filters
• Optimizations for real time queries
• A high performance Thrift gateway
• A REST-ful Web service gateway that supports XML, Protobuf, and binary data encoding options
• Cascading source and sink modules
• A JRuby-based shell
• Support for exporting metrics via the Hadoop metrics subsystem to files or Ganglia; or via JMX

This most recent version of HBase, 0.20.0, has greatly improved on its
predecessors:

• No HBase single point of failure
• Rolling restart for configuration changes and minor upgrades
• Generally, one order of magnitude performance improvement for every class of operation
• Random access performance on par with open source relational databases such as MySQL

We use ZooKeeper as a substitute for Google’s “Chubby” to enable hot fail over should a Master node fail. We do other interesting things with ZooKeeper as well and on our roadmap is increasingly distributed function via emergent behaviors with no central point of control.

Unfortunately the HDFS NameNode is still a single point of failure in Hadoop 0.20, and HBase depends on HDFS. For more information on mitigating this risk, see this Cloudera blog post on NameNode High Availability.

For more detail, please visit the HBase wiki. Links and references to additional information appear below.

Why Would You Need HBase?

Use HBase when you need fault-tolerant, random, real time read/write access to data stored in HDFS. Use HBase when you need strong data consistency. HBase provides Bigtable-like capabilities on top of Hadoop. HBase’s goal is the hosting of very large tables — billions of rows times millions of columns — atop clusters of commodity hardware.

HBase is an answer for effectively managing terabytes of mutating structured storage on the Hadoop platform at reasonable cost. HBase manages structured data on top of HDFS for you, efficiently using the underlying replicated storage as backing store to gain the benefits of its fault tolerance and data availability and locality. HBase hides the gory details of how one would provide random real time read/write access on top of a filesystem tuned for MapReduce jobs that process terabytes of data, where file block sizes are huge, and where a file can be open for reading or for writing, but not both.

At large scale traditional relational databases (RDBMSes) fall down. We are considering here big queries, typically range or table scans; and big tables, typically terabytes or petabytes. Such workloads generally exceed the ability of these systems to process them in a timely, cost-effective manner. Managing very large storage with them alone is an expensive proposition. Processing that data incurs other cost — in time, in productivity. Waits and deadlocks rise nonlinearly with transaction size and concurrency, the square of concurrency, the third power of the transaction size. In contrast, HBase table scans run in linear time, and row lookup or update times are logarithmic with respect to the size of the table.

Features of the relational model get in the way as data volumes scale up and analytics get more complex and interesting. Expensive commercial RDBMS systems can deliver large storage capacities and they can execute some queries over all that data in reasonable time, but only at high dollar cost. Using open source RDBMSes at scale simply requires giving up all relational features (e.g. secondary indexes) for performance. Sharding is a brittle and complex non-solution to these scalability problems. It is something done when there are no better alternatives.

What if we trade relational features for performance since using RDBMSes at scale often requires giving them up anyway? The first casualty of sharding is the normalized schema. We can avoid waits and deadlocks by restricting transaction scope to groups of row mutations only. What if we generalize the data model? Then we can provide transparent horizontal scalability without architectural limits — generic “self-sharding”. We can also provide fault tolerance and data availability by way of the same mechanisms which allow this scalability.

Bigtable and HBase are able to avoid the scalability issues that trouble RDBMSes by eschewing the relational data model. HBase provides something else. It is like a large distributed map. It is a row indexed list of tags and data, values of variable length, bounded by a configuration setting.  It is a column based data store. Keys are arbitrary byte data and are multidimensional: row, column, optional column qualifier, and timestamp. Columns in HBase are multiversioned. You can store more than one version of a value in a particular row and column, and the timestamp provides an extra dimension of indexability.  This can be a particularly useful feature: Multiversioning and timestamps avoid edit conflicts caused by concurrent decoupled processes. Rows are stored in byte lexicographic sorted order.

Lexicographically similar values are packed adjacent to one another into blocks in the column stores and are retrieved efficiently together. Column stores may optionally be compressed on disk. Tables are dynamically split into regions. Regions are hosted on a number of region servers. Adding additional capacity to a HBase cluster is a simple and transparent process: Provision another region server, configure it, and start it. Typically, HBase region servers are co-deployed with Hadoop HDFS DataNodes. The underlying storage capacity grows also. As regions grow, they are split and distributed evenly among the storage cluster to level load. Splits are almost instantaneous. A cluster master process manages region assignment for fast recovery and fine grained load balancing. The Master role falls over to spares as necessary for fault tolerance. The Master rapidly redeploys regions from failed nodes to others. Because the stores are in HDFS, all region servers in the cluster have immediate access to the replicated table data.

When Would You Not Want To Use HBase?

When your data access patterns are largely sequential over immutable data. Use plain MapReduce.

When your data is not large.

When the large overheads of the extract-transform-load (ETL) of your data into alternatives such as Hive is not an issue because you are purely operating on the data in a batching manner and can afford to wait, and some feature of the alternative is simply a must-have.

If you need to make a different trade off between consistency and availability. HBase is a strongly consistent system. HBase regions can be temporarily unavailable during fault recovery. The HBase client API will suspend pending reads and writes until the regions come back online. Perhaps for your use case blocking of any kind for any length of time is intolerable.

If you just can’t live without SQL.

When you really do require normalized schemas or a relational query engine.

However, this last point can use some additional detail. HBase supports random, real time read/write access to your data by way of a single index. However, secondary indexes can be emulated by managing additional index tables at the application level. To achieve fast query response times under real world conditions, “Web 2.0″ applications often denormalize and replicate and synchronize values in multiple tables anyway. Bigtable’s simpler data model is sufficient for many such use cases and furthermore does not support constructs that can get you into trouble. What you do get is:

• Fast (logarithmic time) lookup using row key, optional column and column qualifiers for result set filtering and optional timestamp;
• Full table scans;
• Range scans, with optional timestamp;
• Queries for most recent version or N versions;
• Partial key lookups: When combined with compound keys, these have the same properties as leading left edge indexes with the benefit of a distributed index;
• Server side filters, a form of query push down.

And, while HBase does not support joining data from multiple tables, you can implement your data workflows using Cascading or a similar higher level construct on top of HBase to recover some relational algebraic operators; or you can simply do ?insert time joins? — denormalization, view materialization, and so on.

How Do You Try Out HBase?

Installing and configuring HBase on the CDH2 is fast and easy.

Before we begin, note that HBase requires an available Zookeeper ensemble.  The CDH2 packages for HBase includes a Zookeeper package. You can install and configure it and then point HBase to it, or you can let HBase create and manage a private Zookeeper ensemble using the bundled Zookeeper jar. For new users who do not have Zookeeper already set up, it is easiest to just let HBase take care of it. The instructions below assume this is the case.

Also, let’s consider what is a reasonable test deployment.

Google aims for ~100 regions per region server, and each region is kept to around 200 MB. Large RAM per node and reasonable region counts and sizing means many tables can be cached and served entirely out of RAM. Bigtable is big because there are 100s if not 1000s of nodes participating. The performance numbers in the Bigtable paper are impressive because of the above. It is cheap (for Google) because they build their own hardware and buy components in bulk.

HBase operates in a different world. Many evaluators or new users expect a lot more for a lot less. They don’t build their own hardware — but could and maybe should — and don’t make bulk purchases. Rather, test deployments of 3 or 4 standard type servers are common. Often the hardware is underpowered for the attempted load. Sometimes even smaller deployments are considered, or even virtual machines are used, but those do not make any sense except as programmer tools. While a “pseudo-distributed” configuration for HBase is included in the distribution, a single server deployment is suitable only for very limited testing. We recommend that three servers be considered a minimum test deployment. These can be Amazon EC2 instances, but use c1.xlarge instances.

A reasonable physical server configuration could be:

• Dual quad core CPU
• 8 GB RAM or more (4 GB is passable, but constrain MapReduce to only 1 concurrent mapper and reducer per node)
• 4 x 250 GB data disk attached as JBOD (for the DataNode process)

The reason for the resource demand is simple: The typical deployment combines HDFS, MapReduce, and HBase over all servers uniformly. A good rule of thumb here is each Hadoop and HBase daemon requires 1 available CPU core and 1 GB of heap. Each mapper or reducer task requires 1 CPU core and 200 MB of heap by default, more if asked for. For HBase to achieve best performance, the region servers must be given sufficient heap to buffer writes and cache blocks for repeated reads. The higher the write load or the larger the working set, the more larger heap allocations will be useful. Configuring 2GB or 4GB heap for HBase region servers is not uncommon.

On to a quick install:

1) On each server, install the core HBase RPMs: hbase, hbase-native, hbase-master, hbase-regionserver, hbase-zookeeper, hbase-conf-pseudo, hbase-docs.

2) On each server, create the cluster configuration and use ‘alternatives’ to enable it.

Create the configuration:

% mkdir /etc/hbase-0.20/conf.my_cluster
% cp /etc/hbase-0.20/conf.pseudo/* /etc/hbase-0.20/conf.my_cluster
% vi /etc/hbase-0.20/conf.my_cluster/hbase-site.xml

Set up the ZooKeeper quorum:

<property>
<name>hbase.zookeeper.quorum</name>
<value>host1,host2,host3</value>
</property>

Point HBase root to a folder to be created in HDFS:

<property>
<name>hbase.rootdir</name>
<value>hdfs://namenode:nnport/hbase</value>
</property>

Note: Do not create this folder yourself.

Enable distributed operation:

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>

On small clusters reduce DFS replication to speed writes:

<property>
<name>dfs.replication</name>
<value>2</value>
</property>

Use the new configuration:

% alternatives --install /etc/hbase-0.20/conf hbase-0.20-conf \
/etc/hbase-0.20/conf.my_cluster 50

3) Bring Hadoop HDFS up as you would normally.

4) On all cluster nodes, start zookeeper:

% service hbase-zookeeper start

5) On the designated master, start the master process:

% service hbase-master start

6) On the designated backup master, start another master process:

% service hbase-master start

7) On the designated slaves, start the region server processes:

% service hbase-regionserver start

8 ) Anywhere on the cluster, launch the HBase shell and create a table:

% su - hadoop
% hbase shell

HBase Shell; enter ‘help<RETURN>‘ for list of supported commands. Version: 0.20.0~1-1.cloudera

hbase(main):001:0> create 'TestTable', {NAME=>'test'}

hbase(main):002:0>

9) Somewhere else on the cluster, launch the HBase shell and describe your new table:

% su - hadoop
% hbase shell

HBase Shell; enter ‘help<RETURN>‘ for list of supported commands. Version: 0.20.0~1-1.cloudera

hbase(main):001:0> describe 'TestTable'

DESCRIPTION                                                    ENABLED
{NAME => 'TestTable', FAMILIES => [{NAME => 'test',            true
COMPRESSION => 'NONE', VERSIONS => '3', TTL => '2147483647',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}]}
1 row(s) in 1.6820 seconds


You are now ready to try out your new HBase installation!

For More Information

Visit the HBase Website and Wiki.

Users and supporting projects.

HBase Roadmap

HBase Mailing List

IRC Channel: #hbase on Freenode

Committers and core contributors are here on a regular basis. More active than the Hadoop forums!

Follow us on Twitter: @hbase