• ZOOKEEPER-1367 Data inconsistencies and unexpired ephemeral nodes after cluster restart
• ZOOKEEPER-1412 Java client watches inconsistently triggered on reconnect
• ZOOKEEPER-1277 Servers stop serving when lower 32bits of zxid roll over
• ZOOKEEPER-1309 Creating a new ZooKeeper client can leak file handles
• ZOOKEEPER-1389 It would be nice if start-foreground used exec $JAVA in order to get rid of the intermediate shell process
• ZOOKEEPER-1089 zkServer.sh status does not work due to invalid option of nc

Stability, Compatibility and Testing

3.3.5 is a stable release that’s fully backward compatible with 3.3.4. Only bug fixes relative to 3.3.4 have been applied. Version 3.3.5 will be incorporated into the upcoming CDH3U4 release.

Getting Involved

The Apache ZooKeeper project is working on a number of new features. Our How To Contribute page is a great place to start if you’re interested in getting involved as a developer. You can also follow me on twitter.

Acknowledgements

A special thanks to everyone who contributed to the release (reporting issues, fixing bugs, reviewing changes, writing documentation, etc).

ZooKeeper 3.4 is incorporated into CDH4 and now available in beta 1!

ZOOKEEPER-1367 is the most serious of the issues addressed, it could cause data corruption on restart. This version also adds support for compiling the client on ARM architectures.

• ZOOKEEPER-1367  Data inconsistencies and unexpired ephemeral nodes after cluster restart
• ZOOKEEPER-1343  getEpochToPropose should check if lastAcceptedEpoch is greater or equal than epoch
• ZOOKEEPER-1373  Hardcoded SASL login context name clashes with Hadoop security configuration override
• ZOOKEEPER-1089  zkServer.sh status does not work due to invalid option of nc
• ZOOKEEPER-973    bind() could fail on Leader because it does not setReuseAddress on its ServerSocket
• ZOOKEEPER-1374  C client multi-threaded test suite fails to compile on ARM architectures.
• ZOOKEEPER-1348  Zookeeper 3.4.2 C client incorrectly reports string version of 3.4.1

If you are running 3.4.2 or earlier, be sure to upgrade immediately. See my earlier post for details on what’s new in 3.4.

Stability, Compatibility and Testing

The 3.4 series has been through a number of releases, incorporating feedback from users and addressing found issues. The Apache community is now considering the 3.4.3 release to be of beta quality. 

Getting Involved

The Apache ZooKeeper project is working on a number of new features. Our How To Contribute page is a great place to start if you’re interested in getting involved as a developer. You can also follow me on twitter.

Acknowledgements

A special thanks to everyone who contributed to the release (reporting issues, fixing bugs, reviewing changes, writing documentation, etc).

• ZOOKEEPER-1208 Ephemeral nodes may not be removed after the client session is invalidated
• ZOOKEEPER-961 Watch recovery fails after disconnection when using chroot connection option
• ZOOKEEPER-1049 Session expire/close flooding renders heartbeats to delay significantly
• ZOOKEEPER-1156 Log truncation truncating log too much – can cause data loss
• ZOOKEEPER-1046 Creating a new sequential node incorrectly results in a ZNODEEXISTS error
• ZOOKEEPER-1097 Quota is not correctly rehydrated on snapshot reload
• ZOOKEEPER-1117 zookeeper 3.3.3 fails to build with gcc >= 4.6.1 on Debian/Ubuntu

Stability, Compatibility and Testing

3.3.4 is a stable release that’s fully backward compatible with 3.3.3. Only bug fixes relative to 3.3.3 have been applied. Version 3.3.4 will be incorporated into the upcoming CDH3U3 release.

Note that these changes are included in the recent 3.4.0 release which I detailed earlier.

Getting Involved

The Apache ZooKeeper project is working on a number of new features. Our How To Contribute page is a great place to start if you’re interested in getting involved as a developer. You can also follow me on twitter.

Acknowledgements

A special thanks to everyone who contributed to the release (reporting issues, fixing bugs, reviewing changes, writing documentation, etc).

Things of interest to developers implementing ZooKeeper clients:

• Native Windows version of C client
• Support Kerberos authentication of clients
• Multi-update client API
• Improved REST Interface

The native windows support allows visual studio users to now compile a native client. This will increase ZooKeeper use substantially on that platform, in particular enabling client bindings other than just Java (on Unix the c binding is used to provide a number of bindings to various languages such as python, perl, ruby, go, etc…).

Kerberos authentication was initially developed in ZooKeeper to support the efforts going on in HBase to enable security for those users. However Kerberos client based authentication is now available to anyone using ZooKeeper.

Multi-update is one of the only changes in recent history to extend the ZooKeeper client interface. It allows for multiple operations to be “batched” together as a single atomic operation that either succeeds or fails in its entirety. This feature will greatly simplify implementation of certain domain specific (i.e. client side) business logic.

Improved Operational support:

Operations loves ZooKeeper because of its resilience to failure and self-recovery features, here are some of the new features available in 3.4.0:

• Existing monitoring support has been extended through the introduction of a new ‘mntr’ 4 letter word
• Add tools and recipes for monitoring as a contrib
• Web-based Administrative Interface
• Automating log and snapshot cleaning
• Add logging/stats to identify production deployment issues
• Support for building RPM and DEB packages

A number of contributions were accepted around enabling improved monitoring of the system. In particular the new monitoring 4 letter word and the Ganglia/Nagios integration that now ship with the release will go a long way in helping operations support ZooKeeper in production.

Stability, Compatibility and Testing

It is important to note that 3.4.0 is not yet ready for production. It is an early release that users can start testing so that we can stabilize later 3.4.x releases.  We expect a later dot release to be production-ready soon, and will be incorporated into CDH4.

Getting Involved

The Apache ZooKeeper project is working on a number of new features. Our How To Contribute page is a great place to start if you’re interested in getting involved as a developer. You can also follow me on twitter.

Acknowledgements

A special thanks to everyone who contributed to the release (reporting issues, fixing bugs, reviewing changes, writing documentation, etc), and big thank you to our amazing release manager Mahadev Konar.

The Development track at Hadoop World is a technical deep dive dedicated to discussion about Apache Hadoop and application development for Apache Hadoop. You will hear committers, contributors and expert users from various Hadoop projects discuss the finer points of building applications with Hadoop and the related ecosystem. The sessions will touch on foundational topics such as HDFS, HBase, Pig, Hive, Flume and other related technologies. In addition, speakers will address key development areas including tools, performance, bringing the stack together and testing the stack. Sessions in this track are for developers of all levels who want to learn more about upcoming features and enhancements, new tools, advanced techniques and best practices.

Preview of Development Track Sessions

Building Web Analytics Processing on Hadoop at CBS Interactive
Michael Sun, CBS Interactive

Abstract: CBS Interactive successfully adopted Hadoop as the web analytics platform, processing one Billion weblogs daily from hundreds of web site properties that CBS Interactive oversees. After introducing Lumberjack—the Extraction, Transformation and Loading framework we built based on python and streaming, which is under review for Open-Source release—Michael will talk about web metrics processing on Hadoop, focusing on weblog harvesting, parsing, dimension look-up, sessionization, and loading into a database. Since migrating processing from a proprietary platform to Hadoop, CBS Interactive achieved robustness, fault-tolerance and scalability, and significant reduction of processing time to reach SLA (over six hours reduction so far).

Gateway: Cluster Virtualization Framework
Konstantin Shvachko, eBay

Abstract: Access to Hadoop clusters through dedicated portal nodes (typically located behind firewalls and performing user authentication and authorization) can have several drawbacks — as shared multitenant resources they can create contention among users and increase the maintenance overhead for cluster administrators. This session will discuss the Gateway system, a cluster virtualization framework that provides multiple benefits: seamless access from users’ workplace computers through corporate firewalls; the ability to failover to active clusters for scheduled or unscheduled downtime, as well as the ability to redirect traffic to other clusters during upgrades; and user access to clusters running different versions of Hadoop.

SHERPASURFING – Open Source Cyber Security Solution
Wayne Wheeles, Novii Design

Abstract: Every day billions of packets, both benign and some malicious, flow in and out of networks. Every day it is an essential task for the modern Defensive Cyber Security Organization to be able to reliably survive the sheer volume of data, bring the NETFLOW data to rest, enrich it, correlate it and perform. SHERPASURFING is an open source platform built on the proven Cloudera’s Distribution including Apache Hadoop that enables organizations to perform the Cyber Security mission and at scale at an affordable price point. This session will include an overview of the solution and components, followed by a demonstration of analytics.

Integrating Hadoop with Enterprise RDBMS Using Apache SQOOP and Other Tools
Arvind Prabhakar, Cloudera
Guy Harrison, Quest Software

Abstract: As Hadoop graduates from pilot project to a mission critical component of the enterprise IT infrastructure, integrating information held in Hadoop and in Enterprise RDBMS becomes imperative. We’ll look at key scenarios driving Hadoop and RDBMS integration and review technical options. In particular, we’ll deep dive into the Apache SQOOP project, which expedites data movement between Hadoop and any JDBC database, as well as providing an framework which allows developers and vendors to create connectors optimized for specific targets such as Oracle, Netezza etc.

Next Generation Apache Hadoop MapReduce
Mahadev Konar, Hortonworks

Abstract: The Apache Hadoop MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of Apache Hadoop MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale, high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. We will be presenting the architecture and design of the next generation of map reduce and will delve into the details of the architecture that makes it much easier to innovate. We will also be presenting large scale and small scale comparisons on some benchmarks with MRV1.”

There are several training classes and certification sessions provided surrounding the Hadoop World conference. Don’t forget to register and become Cloudera Certified in Apache Hadoop.

Network

Memory

Disk

CPU

Summary

Why Migrate?

If you’re not familiar with CDH3b2, here’s what you need to know.

All versions of CDH provide:

• RPM and Debian packages for simple installation and management.
• Clean integration with the host operating system. Logs are in /var/log, common binaries in /usr/bin, and configuration in /etc.
• A Cloudera support-ready distribution. As Hadoop becomes a mission critical component of your production infrastructure, you’ll want the option of engaging Cloudera for support or consulting services. Running CDH makes this process simple.

CDH3b2 additionally is:

• A complete platform with smooth integration of popular projects such as Hive, HBase, Pig, Zookeeper, Flume, Sqoop, Oozie, and HUE. HDFS and Hadoop Map Reduce are only two parts of a larger system. CDH3b2 brings together tools frameworks to get data in and out of HDFS, coordinate complex processing pipelines, as well as process and analyze your data. Learn more about this.
• Based on Apache Hadoop 0.20.2 with 320 patches worth of feature back ports, stability enhancements, and bug fixes.

Overview

The migration process does require a moderate understanding of Linux system administration. You should make a plan before you start. You will be restarting some critical services such as the name node and job tracker, so some downtime is necessary. Given the value of the data on your cluster, you’ll also want to be careful to take recent back ups of any mission-critical data sets as well as the name node meta-data.

Backing up your data is most important if you’re upgrading from a version of Hadoop based on an Apache Software Foundation release earlier than 0.20. There were changes in the open source HDFS implementation prior to 0.20 that force this upgrade. See the section below on compatibility for more details.

The process I’ll outline here is as follows:

• CDH version selection
• Options for installation
• Installation process
• Migration of configuration data
• Testing your cluster

Selecting a Branch

One of the first questions you should ask yourself is what level of stability versus new features you require from Hadoop. If you’re managing a production Hadoop cluster with jobs with SLAs, you need a rock solid, production-proven Hadoop distribution. This is Cloudera’s stable or production branch. At the time of this writing, this is CDH2 based on Hadoop 0.20.1+169.89. In certain cases, features may be of greater priority, in which case, CDH3 0.20.2+320 is appropriate.

It’s important to note that both CDH2 and CDH3 pass all functional and unit tests at Cloudera. The real difference between them is that CDH2 has been in the field longer. We generally promote a release to stable when we’ve seen it running production workloads for a substantial period of time, and when the rate of issues opened against the distro in our support group tails off. We have customers running in production today on both CDH2 an CDH3.

On Compatibility

Before we dive into the installation process I’ll highlight some points on compatibility. When upgrading to CDH from an older version or another distribution of Hadoop, it’s possible that HDFS data needs to be taken through an upgrade process. This is relatively simple, but as with any upgrade of critical data, it is absolutely necessary to back up your data.

Currently, it is not necessary to perform an HDFS upgrade if you’re upgrading to CDH3 from CDH2 or Apache Hadoop versions 0.20.0 or later. In fact, any distribution of Hadoop based on Apache 0.20.0 is likely to be a clean transition without an update to HDFS required, but you should always check with the distributor.

During RPC operations, all Hadoop daemons will check to ensure they are speaking to the same exact version as themselves. This means that you cannot, at present, perform a rolling upgrade of CDH. There has been some discussion about relaxing this requirement so compatible versions of Hadoop can communicate, but this has not yet been implemented.

Installation Options

CDH is available in three forms: RPMs, debs, and tarball distributions. The preferred method of installation is usually the RPM or deb packages as they automate a lot of the work required to get CDH up and running quickly. Tarballs of CDH are useful for users on systems that do not use yum/rpm or apt/dpkg, or where you do not have root access to the host operating system.

Installing CDH

When installing CDH from from RPMs or Debian packages you will definitely want to take advantage of Cloudera’s yum or apt repository support. If you’re on a system that is not rpm or deb format packages, you can still use Cloudera’s binary tarball packages.

You should follow the normal process for installing CDH on your systems. The CDH packages should be installed on all nodes in the cluster. The rpm and deb packages of CDH will automatically create a hadoop user and group as well as SYSV init scripts as part of the install process. The CDH tarballs do not contain the init scripts and obviously do not create the hadoop user and group.

Detailed installation instructions for all formats of CDH are available.

After the packages are installed, you’ll want to make sure you set the proper daemons to start on the proper machines upon boot. There is a separate init script for each Hadoop daemon so only what is necessary is started.

Redhat example:
% chkconfig --level 3 hadoop-0.20-namenode on

Debian example:
% update-rc.d hadoop-0.20-namenode start 80 3 .

Make sure you specify the correct run level. While run level 3 is common for multiuser Linux servers, this may not be the case in your installation. You can use the runlevel command to find the currently active run level.

For now, do not start any of the Hadoop daemons.

Migrating Your Configuration

If you’re coming from older version of CDH, your configuration should already be setup with alternatives. If not, now is a good time to bring your configuration layout in line with CDH by moving your conf directory to /etc/hadoop-0.20/conf.mycluster. You should also configure alternatives to know about your new configuration. The CDH documentation covers this in detail. For now, register your new configuration with alternatives and set it to be the preferred configuration.


% alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.mycluster 100
% alternatives --set hadoop-0.20-conf /etc/hadoop-0.20/conf.mycluster


Users who are on systems that don’t have alternatives or who are installing CDH from tarballs should simply update the configuration files in $HADOOP_HOME/conf. Normally, $HADOOP_HOME is /usr/local/hadoop-$VERSION or /opt/hadoop-$VERSION but you can put it wherever it makes sense. This includes running CDH from your home directory if you don’t have root access.

Testing CDH

Now that CDH is installed and you’ve migrated your cluster configuration it’s time to fire up a few nodes and make sure everything is working as expected. Rather than bring up all the daemons at once, let’s focus on the name node first.

Start by logging on to the name node machine. You may want to manually rotate the log file just to minimize the noise during testing. You can do this by simply moving today’s log file to a different name.

% mv /var/log/hadoop/hadoop-hadoop-namenode-nn.mycompany.com.log \
/var/log/hadoop/hadoop-hadoop-namenode-nn.mycompany.com.log.old

Next, start the CDH name node daemon using the provided init script. If an HDFS upgrade is required, you can use the upgrade argument in place of start below. This will be your last chance to grab a backup of the name node’s metadata prior to starting the daemon.

% /etc/init.d/hadoop-0.20-namenode start

Note that the CDH init scripts require you to be root whereas the Apache Hadoop start-all.sh / stop-all.sh scripts should not be run as root.

It’s a good idea to check the contents of the name node log file now to ensure it has come up cleanly. You should see a warning about the name node being in safe mode due to missing blocks. This is OK because we haven’t brought up any data nodes yet. If something doesn’t look right, jump ahead to the getting help section before proceeding.

Before you start any of your data nodes, you’ll want to place the name node in safe mode manually. This will prevent the name node from “panicking” and trying to repair missing block replicas as data nodes begin to register themselves. You’ll need to run this command as the hadoop user.

% hadoop dfsadmin -safemode enter

Next start one of the data nodes and watch its logs as you did for the name node.

% /etc/init.d/hadoop-0.20-datanode start

If everything is setup correctly, you should see the data node start up, register with the name node, and start its periodic block scanner thread. You should also check the name node logs to confirm you see the data node registration message there as well. Once you’ve confirmed that things look good, you should move on to starting additional data nodes checking them in batches as you go.

After all data nodes are up and running, you can use the Hadoop fsck tool to confirm that the file system is healthy.

% hadoop fsck /

Your cluster should still be in safe mode. If the file system is healthy, you can go ahead and take it out of safe mode.

% hadoop dfsadmin -safemode leave

Follow this with a quick test of HDFS by copying a file into the file system.

% date > now.txt
% hadoop fs -put now.txt /now.txt
% hadoop fs -cat /now.txt
% hadoop fs -rm /now.txt
% rm now.txt

Congratulations! You now have HDFS running on CDH.

If you had to upgrade the HDFS data – that is, you started the init script with the upgrade option – you should do some more extensive testing of your data. Once you’ve confirmed everything is working as expected, finalize the HDFS upgrade.

% hadoop namenode -finalize

Starting and testing the map reduce daemons follows a similar procedure but is a bit simpler. Start the job tracker daemon on the proper machine and monitor the logs as you did with the name node. Once you’ve confirmed the job tracker is running, proceed with starting the task tracker daemons in groups checking the job tracker UI as you go. You should see the map and reduce task capacity increasing with each node you start. Don’t panic if the job tracker doesn’t see the nodes immediately; it can take a few seconds.

Don’t forget to start the secondary name node daemon as well. It’s usually a good idea to wait an hour or so and check the modification time on the files in the configured fs.checkpoint.dir. You should see that the files have been updated within the last hour. You can also check the secondary name node logs; you’ll see an indication things are working there as well in the form of some log messages about performing the checkpoint.

Documentation and References

In addition to the community articles and blog posts on Hadoop, Cloudera provides CDH-specific documentation at docs.cloudera.com. Here you can find information on CDH including all of its components like Hadoop, Hive, Flume, Sqoop, HUE, and others.

How to Get Help

There are a number of ways to get help if you run into trouble during your migration or if you just have questions.

• Cloudera Documentation
• Cloudera mailing lists
• Cloudera videos
• IRC users can join #cloudera on freenode

ZooKeeper is distributed coordination

Often distributed applications need some way to coordinate across processes; locking resources, managing queues of events, electing a “leader” process, configuration, etc… Coordination operations such as these are notoriously hard to get right. ZooKeeper provides a relatively simple API which allows clients to correctly implement these and many other coordination mechanisms.

?ZooKeeper is itself a replicated service based on a quorum algorithm. One or more ZooKeeper servers form what’s called an “ensemble”, which are in constant communication. As the size of the ensemble increases the reliability of the service itself increases – as long as a majority of the configured ensemble servers are available the service is available. As an example, say you have an ensemble of size three (three ZooKeeper servers), if one of the three fail the service is still “up”. If two of the three fail the service is down. One could run with five servers, in which case if two servers fail the service as a whole would still be available. Seven server ensembles can survive three failures, and so on.

Who should be interested in the ZooKeeper project? (I say “project/us” here because the team & community are just as important as the software, if not more so) Well, developers appreciate us because we make it simple to implement some very difficult distributed communication problems. Operations teams like us because we ensure that they only need to learn, operate and maintain a single, sane, coordination mechanism that’s easy to manage. Business folks like the fact that we are a proven technology helping to ensure high availability, allowing the development/ops teams to focus on domain specific problems.

For more detail on ZooKeeper see the overview page. Additional documentation relative to CDH releases is available here.

Powered By ZooKeeper

Yahoo!, Facebook, Twitter, Digg, Rackspace and a number of other companies are making use of ZooKeeper in their production environments today. Additionally technologies such as HBase, Solr, Katta, and Neo4j have all increased reliability/availability and extended their capabilities by adopting ZooKeeper.

Here at Cloudera we recently open sourced Flume, a distributed, real-time event collection service which is also part of CDH3 beta 2. Flume makes use of ZooKeeper to coordinate its various distributed components – for example to store and manage dynamically updated configuration information. You can find out more about Flume here.

ZooKeeper and CDH

A significant amount of work has gone into ZooKeeper integration with CDH3. In particular the Cloudera team ensures that the CDH3 components relying on ZooKeeper (HBase and Flume) are fully compatible and will work together. CDH based ZooKeeper packages (tar, RPM, and DEB files) containing libraries as well as startup scripts for running as a service are available on the Cloudera website.

Cloudera is an active member of the Hadoop and ZooKeeper communities – we have two active commiters working on ZooKeeper, myself and Henry Robinson. Henry recently contributed a major new feature called “observers” which greatly extends ZooKeeper’s read scalability, Henry also created and maintains the popular ZooKeeper python client binding.

In addition to leading the ZooKeeper project at Apache I’ve personally been working on a number of projects for upcoming releases; I’m currently adding transport level security (encryption and authentication of communications) to the service. The team is constantly on the lookout for other areas where the technology may be applied. As I mentioned HBase and Flume are currently using ZooKeeper and we hope to extend this further, in particular around the idea of centralized configuration and monitoring for Hadoop. There are just too many configuration files floating around when setting up a Hadoop based service, it seems like ZooKeeper would be a perfect fit for this. A great example of this in use today is LinkedIn’s Norbert project, where ZooKeeper is used to maintain and manage cluster metadata.

Join the ZooKeeper community

Find out more about ZooKeeper on the official Apache project pages. On that page you’ll also find links for documentation, user and developer mailing lists, issue tracking, etc… we welcome new users and contributors. You might also follow me on twitter, where I frequently post on community related issues.

In order to be useful, ZooKeeper must be both highly reliable and available as systems will rely upon it as a critical component. For example, if locks cannot be taken, processes cannot make progress and the whole system will grind to a halt. ZooKeeper is built on a suite of reliable distributed systems techniques and protocols, and is typically run on a cluster of machines so that if some should fail, the remaining ones can continue to provide service. Under the hood, ZooKeeper is responsible for ordering calls made by clients so that each request is processed atomically and in a fixed and firm order.

One of my first contributions to the project was a set of bindings to allow programs written in the Python language to act as clients to a ZooKeeper cluster. ZooKeeper was natively written in Java, and there are already C and Perl bindings. Adding Python bindings increases the number of people that can use the system, and brings the strengths of Python, such as rapid prototyping, to bear when designing distributed systems.

The Python ZooKeeper bindings are available from the ZooKeeper SVN repository and should be part of the 3.2 release, planned for the next couple of weeks. To use the bindings now, you can either check out the latest version of the code from the SVN repository, or download a tarball containing a recent snapshot here. The zookeeper module exposes the ZooKeeper API to Python, so to get started all you need do is add import zookeeper to your Python script once the module is installed. Instructions on getting up and running are at the end of this post.

To illustrate some of the ZooKeeper API, I’ve written a distributed FIFO queue in Python – the source code is here – which I wanted to share. The combination of Python and Zookeeper meant that I was able to write the queue in just over 60 lines of code, and most of that deals with local coordination issues between two threads rather than any tricky issues trying to make remote processes behave correctly. I can only give a taste here of how programming with Python and ZooKeeper works. I hope there’s enough here to convince you that ZooKeeper might make a useful component for distributed systems that need a little herding.

ZooKeeper

ZooKeeper provides a tree abstraction where every node in that tree (or znode, in ZooKeeper parlance) is a file on which a variety of simple operations can be performed. ZooKeeper orders operations on znodes so that they occur atomically. Therefore there is no need to use complex locking protocols to ensure that only one process can access a znode at a time. The tree represents a hierarchical namespace, so that many distinct distributed systems can use a single ZooKeeper instance without worrying about their files having the same name.

Each znode has some associated data – up to a megabyte in current builds – that can be updated atomically. Every update to a znode increases its version number, which allows clients to perform compare-and-swap operations by reading the version and then updating a znode only if the version is still the one that was read.

As a notification mechanism, ZooKeeper provides watches, which are callback methods that are called asynchronously when an event of interest occurs. Watches are attached, typically, to an individual znode. When that znode changes any watcher on the znode will be fired asynchronously on the client. Many methods of the ZooKeeper API have an optional watch argument. Some languages have to work hard to provide callable objects as parameters, but Python makes this easy as callables are first class language constructs. Simply pass any callable, like a method or a lambda expression, to the zookeeper module and when an event of interest occurs, the callable will be executed.

This call comes from a separate thread of execution, so great care must be taken to ensure that unexpected things do not happen due to your watcher being fired at an arbitrary point in the execution of your script. Normally you will use watchers to notify another thread of a state change. It will often be the case that the main thread will be waiting for the watcher to fire before it can continue. An example of this is in the __init__ method of our ZooKeeperQueue when we try to connect to the server. Compared to the time a script takes to execute, connections can take a long time to run. So it’s useful that the ZooKeeper API allows us to connect asynchronously, in case there were any work that we wanted to get done while we were waiting for the connection to be established. However, in our case, we just want to wait until the connection is successful, and so we need a mechanism to wait for the watcher to notify us.

A useful tool for this inter-thread communication is the Condition object in Python, which represents a condition variable, a well-known concurrent programming abstraction. Condition objects may be acquired and released just like locks, but they also expose an API to wait for a notification from another thread and to fire that notification. While a thread is waiting on a Condition it goes to sleep, leaving the operating system with some free CPU to dedicate to other processes. Once a Condition is notified, a thread that is waiting on it is woken up and allowed to continue execution once the notifying thread has released the Condition.

This leads to a simple pattern for communicating between watchers and the main thread. Here’s an excerpt from the connection code:

def watcher(handle,type,state,path):
    print "Connected"
    self.cv.acquire()
    self.connected = True
    self.cv.notify()
    self.cv.release()

self.cv.acquire()
self.handle = zookeeper.init("localhost:2181", watcher, 10000, 0)
self.cv.wait(10.0)

First we define our watcher which takes four parameters (if you want to provide more parameters or local state to a watcher, one way to do it is to wrap a function call in a local lambda which captures the state). The next line acquires an exclusive lock on a condition variable cv. Why do this now? Once we set our watcher in place, it could be fired at any time – even before the main thread makes progress to the next line of code. If we don’t prevent it from sending a notification on the condition variable before we’re ready to look for it, the notification could get lost and we could wait forever. Notifications aren’t buffered – if no one is waiting on a condition variable, no one gets woken up.

Then the code initialises ZooKeeper. The zookeeper module gives us an integer handle which we can use to refer to our connection in the future (we can open many connections per client). The next line tells us to wait until we receive a notification on the condition variable that the connection has succeeded. The parameter is a timeout in seconds, after which if we are still not connected we presume that something is wrong and abort.

The ZooKeeper queue

A FIFO queue is a simple data structure where producers put items in, and consumers retrieve them in the order they were put in. There are only two operations on a basic queue: enqueue adds an item and dequeue removes it. Despite their simplicity, queues crop up very often in distributed systems – for example, in job submission systems where clients submit requests to a set of workers which serve the requests on a first-come, first-served basis.

The ZooKeeper queue is structured very simply. All items are stored as znodes under a single top-level znode which represents a queue instance. Consumers can retrieve items by getting and then deleting a child of the top-level znode. The code creates a queue by calling a single create command. If the queue already exists, the Python module will throw an exception which we catch. This is a design decision that is still in review – future versions of the bindings might return integer error codes, and rely on the user to throw an exception if required.

zookeeper.create(self.handle,self.queuename,"queue top level", [ZOO_OPEN_ACL_UNSAFE],0)

The first two arguments to this call identify the connection to the ZooKeeper service and the name of the znode. The third is the data the znode contains. We won’t be accessing the data so we write some placeholder text.

The fourth argument is an access control list of permissions that controls who can access the znode in the future. ZooKeeper provides fairly fine-grained control over access, but the subject is beyond the scope of this post. What we have done here is to create the queue znode so that any client can read or write to it.

Adding and deleting items from the queue

Although I explained how consumers retrieve items from the queue, I said nothing about how they make sure they are retrieving items in FIFO order. What we would like is a way of naming each item such that later items are ordered lexicographically after earlier ones. If we can retrieve items in the same order, we’ll have our queue. Thankfully, ZooKeeper provides a very handy flag for the create call that helps us out. Specifying the zookeeper.CREATE_SEQUENCE flag appends each znode name with an sequence number suffix that increases monotonically with each new znode that is created. ZooKeeper ensures that the sequence numbers are applied in order and are not reused.

Enqueuing an item is therefore a simple one liner. We don’t have to take out any locks to ensure that access to the queue znode is serialised. Items may be queued concurrently, and ZooKeeper takes care of assigning sequence numbers to them in the order they were received.

Dequeuing an item is also straightforward, but a bit more involved. First we retrieve a list of all the items waiting to be queued from ZooKeeper with the get_children procedure call. Then, after sorting the list of items on the client, we get the contents of the znode (i.e. the item’s data) and then try to delete it.

It is possible that this deletion will fail because some other consumer has managed to successfully retrieve the item beforehand. We could ensure that this would never happen by organising for a queue-wide lock – this is easily implemented in ZooKeeper (although left as an exercise for the reader). However, this would severely impact performance by only allowing a single consumer to access the queue at one time. Instead, the client simply deals with the failed delete – again, indicated via an exception – and moves on to the next child znode in the list. If the client reaches the end of the list without successfully deleting an item, it should issue another get_children call to make sure that no items were added while the original list was being scanned. Once the get_children call returns an empty list, the dequeue procedure gives up and returns None.

Blocking reads

Sometimes we might want to block until an item is available to retrieve. It would be inefficient to copy exactly the non-blocking approach and simply loop, issuing get_children requests until an item was found. Instead, we can leverage ZooKeeper’s watcher mechanism to provide an asynchronous notification when a new znode is created as a child of the queue znode. The code to accomplish this is a combination of the patterns we’ve seen already in the dequeue and connection code.

def block_dequeue(self):
    def queue_watcher(handle,event,state,path):
        self.cv.acquire()
        self.cv.notify()
        self.cv.release()
    while True:
        self.cv.acquire()
        children = sorted(zookeeper.get_children(self.handle, self.queuename, queue_watcher))
        for child in children:
            data = self.get_and_delete(self.queuename+"/"+children[0])
            if data != None:
                self.cv.release()
                return data
        self.cv.wait()
        self.cv.release()

First the client acquires a lock to prevent the watcher sending a notification when the client is unready. Then, as in the dequeue method, the client retrieves a list of items, but here a watcher parameter is specified. The watcher will fire whenever any event is seen that is relevant to the queue znode. The watcher acquires the lock – blocking until the client has given it up – and then notifies the client that there may be more items available.

The client only waits for this notification if all the children returned from get_children have already been consumed by others – otherwise it will successfully retrieve an item and return it. Once all possible items have been exhausted, the client waits on the condition variable. After being woken up, it repeats the same list-read-delete-wait loop.

Failure modes

ZooKeeper operations can fail in a number of ways. In order to keep this example simple, most errors are raised as exceptions and the queue aborts. A more robust implementation should catch errors at every ZooKeeper invocation, as many can be recovered from with a little effort.

The zookeeper.CONNECTIONLOSS error condition is particularly worth noting. ZooKeeper may drop a client connection at any time, due to physical link loss, network congestion or other connection problem. This can cause ZooKeeper API invocations to abort before the ZooKeeper cluster is able to inform the client of the operation’s success. This is problematic for our queue, as enqueue operations may or may not have succeeded when we receive a CONNECTIONLOSS error.

There are several approaches we can take to this problem. The first is to blindly retry enqueue when a connection is lost. This could result in an item being queued several times, but for some systems this is not a significant problem. For example, if a web page is crawled twice, apart from the time cost there will be no hardship caused to a indexing engine.

For some applications, duplication of enqueue operations is problematic. The obvious ‘solution’ is to check whether an item is in the queue after it has been queued. However, it is possible that a consumer will have retrieved and deleted the item between the connection loss event and the subsequent reconnection and existence check. Instead, a two-phase protocol is necessary where a producer marks an item as ‘consumable’ only when it is sure it is in the queue, by atomically updating its associated data with a flag. Consumers may only retrieve items for which the flag is set. If a connection loss occurs during the setting of this flag, recovery is easier as the set call may be reissued – if the item is no longer present in the queue, the only possible explanation is that the original flag update succeeded and the item has been consumed. This is not built into the example code, but a production system should implement a similar form of connection loss recovery.

Taking care of failure modes like this one often comprises most of the work of building a distributed system. The key is to understand every exception that API calls can throw, and to know what your code does in every circumstance.

Using the queue

To use the queue, you must first make sure you have built and installed both the C client libraries and the zookeeper Python module. There are two prerequisite packages: the cppunit development package and the Python development package. On yum-based systems, these are named cppunit-devel and python-devel. Both packages are available through standard platform package managers like yum, apt and Darwin ports.

As a prerequisite to building the C client libraries, the Java based-server must be built. This auto-generates some header files that the C libraries rely on. From the root directory of the downloaded distribution:

ant


The C client libraries for ZooKeeper must be installed as the Python module makes use of them to actually communicate with a ZooKeeper cluster. It’s easiest to build these from source. From the src/c directory, type the following:

autoreconf -if
./configure
make && sudo make install

The downloadable package contains the source code for the Python module. To build and install, one command should do the trick from the src/contrib/zkpython directory:

ant install

To test the installation, start a Python shell and type import zookeeper. If you don’t see any errors or warnings, the module has been built and installed successfully. The bindings have been tested with Python 2.3, 2.4, 2.5 and 2.6, and are known not to work with 2.2 and earlier. We haven’t yet tested them against Python 3.x – we’d love to hear your feedback about your experiences with the latest versions of Python.

To run the queue example, you must have a ZooKeeper server running on the local machine at port 2181 (to change the location of the server, edit the string passed to zookeeper.__init__). The Java-based server will have been built when you ran ant from the root directory of the distribution earlier. Before the server can run, it needs a configuration file to read:

cat >> conf/zoo.cfg
tickTime=2000
dataDir=/tmp/zookeeper
clientPort=2181


Now you can run bin/zkServer.sh start to start a standalone server on the local machine. To stop the server in the future, run bin/zkServer.sh stop.

You’re finally ready to run the queue example:

python queue.py


The example is very simple. It queues three items, and then dequeues them.

Wrapping up

I hope that I’ve shown you that ZooKeeper is a very useful system, with powerful primitives that makes writing tricky distributed concurrent programs easier. There are many applications that ZooKeeper could help you build – lock servers, name services, metadata stores and even a unique kind of filesystem can be built in a straightforward way using the ZooKeeper API. The project is active and always looking for volunteers. ZooKeeper integration is already being built into HBase, and there are moves to bring greater reliability to Hadoop and HDFS by delegating some server functionality to ZooKeeper. As far as the Python bindings go, the next version will include better documentation, some more Python niceties such as default parameters and docstrings, and a more Pythonic wrapper object to wrap up some of the bookkeeping that ZooKeeper requires.