Some of the configuration properties found in Hadoop have a direct effect on clients, such as HBase. One of those properties is called “dfs.datanode.max.xcievers”, and belongs to the HDFS subproject. It defines the number of server side threads and – to some extent – sockets used for data connections. Setting this number too low can cause problems as you grow or increase utilization of your cluster. This post will help you to understand what happens between the client and server, and how to determine a reasonable number for this property.

The Problem

Since HBase is storing everything it needs inside HDFS, the hard upper boundary imposed by the ”dfs.datanode.max.xcievers” configuration property can result in too few resources being available to HBase, manifesting itself as IOExceptions on either side of the connection. Here is an example from the HBase mailing list [1], where the following messages were initially logged on the RegionServer side: 

2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream
2008-11-11 19:55:52,451 INFO org.apache.hadoop.dfs.DFSClient: Abandoning block blk_-5467014108758633036_595771
2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block.
2008-11-11 19:55:58,455 WARN org.apache.hadoop.dfs.DFSClient: Error Recovery for block blk_-5467014108758633036_595771 bad datanode[0]
2008-11-11 19:55:58,482 FATAL org.apache.hadoop.hbase.regionserver.Flusher: Replay of hlog required. Forcing server shutdown

Correlating this with the Hadoop DataNode logs revealed the following entry:

ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(10.10.10.53:50010,storageID=DS-1570581820-10.10.10.53-50010-1224117842339,infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException: xceiverCount 258 exceeds the limit of concurrent xcievers 256  

In this example, the low value of “dfs.datanode.max.xcievers” for the DataNodes caused the entire RegionServer to shut down. This is a really bad situation. Unfortunately, there is no hard-and-fast rule that explains how to compute the required limit. It is commonly advised to raise the number from the default of 256 to something like 4096 (see [1], [2], [3], [4], and [5] for reference). This is done by adding this property to the hdfs-site.xml file of all DataNodes (note that it is misspelled): 

<property>    <name>dfs.datanode.max.xcievers</name>
    <value>4096</value>
  </property>

Note: You will need to restart your DataNodes after making this change to the configuration file.

This should help with the above problem, but you still might want to know more about how this all plays together, and what HBase is doing with these resources. We will discuss this in the remainder of this post. But before we do, we need to be clear about why you cannot simply set this number very high, say 64K and be done with it.

There is a reason for an upper boundary, and it is twofold: first, threads need their own stack, which means they occupy memory. For current servers this means 1MB per thread[6] by default. In other words, if you use up all the 4096 DataXceiver threads, you need around 4GB of heap to accommodate them. This cuts into the space you have assigned for memstores and block caches, as well as all the other moving parts of the JVM. In a worst case scenario, you might run into an OutOfMemoryException, and the RegionServer process is toast. You want to set this property to a reasonably high number, but not too high either.

Second, having these many threads active you will also see your CPU becoming increasingly loaded. There will be many context switches happening to handle all the concurrent work, which takes away resources for the real work. As with the concerns about memory, you want the number of threads not grow boundlessly, but provide a reasonable upper boundary – and that is what “dfs.datanode.max.xcievers” is for.

Hadoop File System Details

From the client side, the HDFS library is providing the abstraction called Path. This class represents a file in a file system supported by Hadoop, represented by the FileSystem class. There are a few concrete implementation of the abstract FileSystem class, one of which is the DistributedFileSytem, representing HDFS. This class in turn wraps the actual DFSClient class that handles all interactions with the remote servers, i.e. the NameNode and the many DataNodes.

When a client, such as HBase, opens a file, it does so by, for example, calling the open() or create() methods of the FileSystem class, here the most simplistic incarnations

  public DFSInputStream open(String src) throws IOException
  public FSDataOutputStream create(Path f) throws IOException

The returned stream instance is what needs a server-side socket and thread, which are used to read and write blocks of data. They form part of the contract to exchange data between the client and server. Note that there are other, RPC-based protocols in use between the various machines, but for the purpose of this discussion they can be ignored.

The stream instance returned is a specialized DFSOutputStream or DFSInputStream class, which handle all of the interaction with the NameNode to figure out where the copies of the blocks reside, and the data communication per block per DataNode.

On the server side, the DataNode wraps an instance of DataXceiverServer, which is the actual class that reads the above configuration key and also throws the above exception when the limit is exceeded.

When the DataNode starts, it creates a thread group and starts the mentioned DataXceiverServer instance like so:

  this.threadGroup = new ThreadGroup(“dataXceiverServer”);
  this.dataXceiverServer = new Daemon(threadGroup,
      new DataXceiverServer(ss, conf, this));
  this.threadGroup.setDaemon(true); // auto destroy when empty 

Note that the DataXceiverServer thread is already taking up one spot of the thread group. The DataNode also has this internal class to retrieve the number of currently active threads in this group:

  /** Number of concurrent xceivers per node. */
  int getXceiverCount() {
    return threadGroup == null ? 0 : threadGroup.activeCount();
  }

Reading and writing blocks, as initiated by the client, causes for a connection to be made, which is wrapped by the DataXceiverServer thread into a DataXceiver instance. During this hand off, a thread is created and registered in the above thread group. So for every active read and write operation a new thread is tracked on the server side. If the count of threads in the group exceeds the configured maximum then the said exception is thrown and recorded in the DataNode’s logs:

  if (curXceiverCount > dataXceiverServer.maxXceiverCount) {
    throw new IOException(“xceiverCount ” + curXceiverCount
                          + ” exceeds the limit of concurrent xcievers “
                          + dataXceiverServer.maxXceiverCount);
  }

Implications for Clients

Now, the question is, how does the client reading and writing relate to the server side threads. Before we go into the details though, let’s use the debug information that the DataXceiver class logs when it is created and closed

  LOG.debug(“Number of active connections is: ” + datanode.getXceiverCount());
  …
  LOG.debug(datanode.dnRegistration + “:Number of active connections is: “     + datanode.getXceiverCount());

and monitor during a start of HBase what is logged on the DataNode. For simplicity’s sake this is done on a pseudo distributed setup with a single DataNode and RegionServer instance. The following shows the top of the RegionServer’s status page.

The important part is in the “Metrics” section, where it says “storefiles=22″. So, assuming that HBase has at least that many files to handle, plus some extra files for the write-ahead log, we should see the above logs message state that we have at least 22 “active connections”. Let’s start HBase and check the DataNode and RegionServer log files:

Command Line:

$ bin/start-hbase.sh
…

DataNode Log:

2012-03-05 13:01:35,309 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 1
2012-03-05 13:01:35,315 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 2
12/03/05 13:01:35 INFO regionserver.MemStoreFlusher: globalMemStoreLimit=396.7m, globalMemStoreLimitLowMark=347.1m, maxHeap=991.7m
12/03/05 13:01:39 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 60030
2012-03-05 13:01:40,003 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 1
12/03/05 13:01:40 INFO regionserver.HRegionServer: Received request to open region: -ROOT-,,0.70236052
2012-03-05 13:01:40,882 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 3
2012-03-05 13:01:40,884 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 4
2012-03-05 13:01:40,888 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 3
…
12/03/05 13:01:40 INFO regionserver.HRegion: Onlined -ROOT-,,0.70236052; next sequenceid=63083
2012-03-05 13:01:40,982 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 3
2012-03-05 13:01:40,983 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 4
…
12/03/05 13:01:41 INFO regionserver.HRegionServer: Received request to open region: .META.,,1.1028785192
2012-03-05 13:01:41,026 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 3
2012-03-05 13:01:41,027 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 4
…
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined .META.,,1.1028785192; next sequenceid=63082
2012-03-05 13:01:41,109 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 3
2012-03-05 13:01:41,114 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 4
2012-03-05 13:01:41,117 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 5
12/03/05 13:01:41 INFO regionserver.HRegionServer: Received request to open 16 region(s)
12/03/05 13:01:41 INFO regionserver.HRegionServer: Received request to open region: usertable,,1330944810191.62a312d67981c86c42b6bc02e6ec7e3f.
12/03/05 13:01:41 INFO regionserver.HRegionServer: Received request to open region: usertable,user1120311784,1330944810191.90d287473fe223f0ddc137020efda25d.
…

2012-03-05 13:01:41,246 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 6
2012-03-05 13:01:41,248 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 7
…
2012-03-05 13:01:41,257 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 10
2012-03-05 13:01:41,257 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 9
…
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined usertable,user1120311784,1330944810191.90d287473fe223f0ddc137020efda25d.; next sequenceid=62917
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined usertable,,1330944810191.62a312d67981c86c42b6bc02e6ec7e3f.; next sequenceid=62916
…
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined usertable,user1361265841,1330944811370.80663fcf291e3ce00080599964f406ba.; next sequenceid=62919
2012-03-05 13:01:41,474 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 6
2012-03-05 13:01:41,491 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 7
2012-03-05 13:01:41,495 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 8
2012-03-05 13:01:41,508 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 7
…
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined usertable,user1964968041,1330944848231.dd89596e9129e1caa7e07f8a491c9734.; next sequenceid=62920
2012-03-05 13:01:41,618 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 6
2012-03-05 13:01:41,621 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 7
…
2012-03-05 13:01:41,829 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 7
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined usertable,user515290649,1330944849739.d23924dc9e9d5891f332c337977af83d.; next sequenceid=62926
2012-03-05 13:01:41,832 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 6
2012-03-05 13:01:41,838 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 7
12/03/05 13:01:41 INFO regionserver.HRegion: Onlined usertable,user757669512,1330944850808.cd0d6f16d8ae9cf0c9277f5d6c6c6b9f.; next sequenceid=62929
…
2012-03-05 14:01:39,711 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 4
2012-03-05 22:48:41,945 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 4
12/03/05 22:48:41 INFO regionserver.HRegion: Onlined usertable,user757669512,1330944850808.cd0d6f16d8ae9cf0c9277f5d6c6c6b9f.; next sequenceid=62929
2012-03-05 22:48:41,963 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 4

You can see how the regions are opened one after the other, but what you also might notice is that the number of active connections never climbs to 22 – it barely even reaches 10. Why is that? To understand this better, we have to see how files in HDFS map to the server-side DataXceiver’s instance – and the actual threads they represent. 

Hadoop Deep Dive

The aforementioned DFSInputStream and DFSOutputStream are really facades around the usual stream concepts. They wrap the client-server communication into these standard Java interfaces, while internally routing the traffic to a selected DataNode – which is the one that holds a copy of the current block. It has the liberty to open and close these connection as needed. As a client reads a file in HDFS, the client library classes switch transparently from block to block, and therefore from DataNode to DataNode, so it has to open and close connections as needed. 

The DFSInputStream has an instance of a DFSClient.BlockReader class, that opens the connection to the DataNode. The stream instance calls blockSeekTo() for every call to read() which takes care of opening the connection, if there is none already. Once a block is completely read the connection is closed. Closing the stream has the same effect of course. 

The DFSOutputStream has a similar helper class, the DataStreamer. It tracks the connection to the server, which is initiated by the nextBlockOutputStream() method. It has further internal classes that help with writing the block data out, which we omit here for the sake of brevity.

Both writing and reading blocks requires a thread to hold the socket and intermediate data on the server-side, wrapped in the DataXceiver instance. Depending what your client is doing, you will see the number of connections fluctuate around the number of currently accessed files in HDFS.

Back to the HBase riddle above: the reason you do not see up to 22 (and more) connections during the start is that while the regions open, the only required data is the HFile’s info block. This block is read to gain vital details about each file, but then closed again. This means that the server-side resource is released in quick succession. The remaining four connections are harder to determine. You can use JStack to dump all threads on the DataNode, which in this example shows this entry:

“DataXceiver for client /127.0.0.1:64281 [sending block blk_5532741233443227208_4201]” daemon prio=5 tid=7fb96481d000 nid=0x1178b4000 runnable [1178b3000]
   java.lang.Thread.State: RUNNABLE
   …

“DataXceiver for client /127.0.0.1:64172 [receiving block blk_-2005512129579433420_4199 client=DFSClient_hb_rs_10.0.0.29,60020,1330984111693_1330984118810]” daemon prio=5 tid=7fb966109000 nid=0x1169cb000 runnable [1169ca000]
   java.lang.Thread.State: RUNNABLE
   …

These are the only DataXceiver entries (in this example), so the count in the thread group is a bit misleading. Recall that the DataXceiverServer daemon thread already accounts for one extra entry, which combined with the two above accounts for the three active connections – which in fact means three active threads. The reason the log states four instead, is that it logs the count from an active thread that is about to finish. So, shortly after the count of four is logged, it is actually one less, i.e. three and hence matching our head count of active threads.

Also note that the internal helper classes, such as the PacketResponder occupy another thread in the group while being active. The JStack output does indicate that fact, listing the thread as such:

 ”PacketResponder 0 for Block blk_-2005512129579433420_4199″ daemon prio=5 tid=7fb96384d000 nid=0x116ace000 in Object.wait() [116acd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder \
       .lastDataNodeRun(BlockReceiver.java:779)
     - locked <7bc79c030> (a org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder)
     at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870)
     at java.lang.Thread.run(Thread.java:680)

This thread is currently in TIMED_WAITING state and is not considered active. That is why the count emitted by the DataXceiver log statements is not including these kind of threads. If they become active due to the client sending sending data, the active thread count will go up again. Another thing to note its that this thread does not need a separate connection, or socket, between the client and the server. The PacketResponder is just a thread on the server side to receive block data and stream it to the next DataNode in the write pipeline.

The Hadoop fsck command also has an option to report what files are currently open for writing:

$ hadoop fsck /hbase -openforwrite
FSCK started by larsgeorge from /10.0.0.29 for path /hbase at Mon Mar 05 22:59:47 CET 2012
……/hbase/.logs/10.0.0.29,60020,1330984111693/10.0.0.29%3A60020.1330984118842 0 bytes, 1 block(s), OPENFORWRITE: ………………………………..Status: HEALTHY
 Total size:     2088783626 B
 Total dirs:     54
 Total files:    45
 …

This does not immediately relate to an occupied server-side thread, as these are allocated by block ID. But you can glean from it, that there is one open block for writing. The Hadoop command has additional options to print out the actual files and block ID they are comprised of:

$ hadoop fsck /hbase -files -blocks
FSCK started by larsgeorge from /10.0.0.29 for path /hbase at Tue Mar 06 10:39:50 CET 2012

…
/hbase/.META./1028785192/.tmp <dir>
/hbase/.META./1028785192/info <dir>
/hbase/.META./1028785192/info/4027596949915293355 36517 bytes, 1 block(s):  OK
0. blk_5532741233443227208_4201 len=36517 repl=1

…
Status: HEALTHY
 Total size:     2088788703 B
 Total dirs:     54
 Total files:     45 (Files currently being written: 1)
 Total blocks (validated):     64 (avg. block size 32637323 B) (Total open file blocks (not validated): 1)
 Minimally replicated blocks:     64 (100.0 %)
 …

This gives you two things. First, the summary states that there is one open file block at the time the command ran – matching the count reported by the “-openforwrite” option above. Secondly, the list of blocks next to each file lets you match the thread name to the file that contains the block being accessed. In this example the block with the ID “blk_5532741233443227208_4201″ is sent from the server to the client, here a RegionServer. This block belongs to the HBase .META. table, as shown by the output of the Hadoop fsck command. The combination of JStack and fsck can serve as a poor mans replacement for lsof (a tool on the Linux command line to “list open files”).

The JStack also reports that there is a DataXceiver thread, with an accompanying PacketResponder, for block ID “blk_-2005512129579433420_4199″, but this ID is missing from the list of blocks reported by fsck. This is because the block is not yet finished and therefore not available to readers. In other words, Hadoop fsck only reports on complete (or synced[7][8], for Hadoop version that support this feature) blocks. 

Back to HBase

Opening all the regions does not need as many resources on the server as you would have expected. If you scan the entire HBase table though, you force HBase to read all of the blocks in all HFiles: 

HBase Shell:

hbase(main):003:0> scan ‘usertable’
…
1000000 row(s) in 1460.3120 seconds

DataNode Log:

2012-03-05 14:42:20,580 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 6
2012-03-05 14:43:23,293 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 7
2012-03-05 14:43:23,299 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 8
…
2012-03-05 14:49:24,332 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 11
2012-03-05 14:49:24,332 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 10
2012-03-05 14:49:59,987 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 11
2012-03-05 14:51:12,603 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 12
2012-03-05 14:51:12,605 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 11
2012-03-05 14:51:46,473 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 12
…
2012-03-05 14:56:59,420 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 15
2012-03-05 14:57:31,722 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 16
2012-03-05 14:58:24,909 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 17
2012-03-05 14:58:24,910 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 16
…
2012-03-05 15:04:17,688 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 21
2012-03-05 15:04:17,689 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 22
2012-03-05 15:04:54,545 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 21
2012-03-05 15:05:55,901 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1423642448-10.0.0.64-50010-1321352233772, infoPort=50075, ipcPort=50020):Number of active connections is: 22
2012-03-05 15:05:55,901 DEBUG org.apache.hadoop.hdfs.server.datanode.DataNode: Number of active connections is: 21

The number of active connections reaches the elusive 22 now. Note that this count already includes the server thread, so we are still a little short of what we could consider the theoretical maximum – based on the number of files HBase has to handle.

What does that all mean?

So, how many “xcievers (sic)” do you need? Given you only use HBase, you could simply monitor the above “storefiles” metric (which you get also through Ganglia or JMX) and add a few percent for intermediate and write-ahead log files. This should work for systems in motion. However, if you were to determine that number on an idle, fully compacted system and assume it is the maximum, you might find this number being too low once you start adding more store files during regular memstore flushes, i.e. as soon as you start to add data to the HBase tables. Or if you also use MapReduce on that same cluster, Flume log aggregation, and so on. You will need to account for those extra files, and, more importantly, open blocks for reading and writing. 

Note again that the examples in this post are using a single DataNode, something you will not have on a real cluster. To that end, you will have to divide the total number of store files (as per the HBase metric) by the number of DataNodes you have. If you have, for example, a store file count of 1000, and your cluster has 10 DataNodes, then you should be OK with the default of 256 xceiver threads per DataNode.

The worst case would be the number of all active readers and writers, i.e. those that are currently sending or receiving data. But since this is hard to determine ahead of time, you might want to consider building in a decent reserve. Also, since the writing process needs an extra – although shorter lived – thread (for the PacketResponder) you have to account for that as well. So a reasonable, but rather simplistic formula could be:

This formula takes into account that you need about two threads for an active writer and another for an active reader. This is then summed up and divided by the number of DataNodes, since you have to specify the “dfs.datanode.max.xcievers” per DataNode.

If you loop back to the HBase RegionServer screenshot above, you saw that there were 22 store files. These are immutable and will only be read, or in other words occupy one thread only. For all memstores that are flushed to disk you need two threads – but only until they are fully written. The files are finalized and closed for good, cleaning up any thread in the process. So these come and go based on your flush frequency. Same goes for compactions, they will read N files and write them into a single new one, then finalize the new file. As for the write-ahead logs, these will occupy a thread once you have started to add data to any table. There is a log file per server, meaning that you can only have twice as many active threads for these files as you have RegionServers.

For a pure HBase setup (HBase plus its own HDFS, with no other user), we can estimate the number of needed DataXceiver’s with the following formula:

Since you will be hard pressed to determine the active number of store files, flushes, and so on, it might be better to estimate the theoretical maximum instead. This maximum value takes into account that you can only have a single flush and compaction active per region at any time. The maximum number of logs you can have active matches the number of RegionServers, leading us to this formula:

Obviously, the number of store files will increase over time, and the number of regions typically as well. Same for the numbers of servers, so keep in mind to adjust this number over time. In practice, you can add a buffer of, for example, 20%, as shown in the formula below – in an attempt to not force you to change the value too often. 

On the other hand, if you keep the number of regions fixed per server[9], and rather split them manually, while adding new servers as you grow, you should be able to keep this configuration property stable for each server.

Final Advice & TL;DR

Here is the final formula you want to use:

It computes the maximum number of threads needed, based on your current HBase vitals (no. of store files, regions, and region servers). It also adds a fudge factor of 20% to give you room for growth. Keep an eye on the numbers on a regular basis and adjust the value as needed. You might want to use Nagios with appropriate checks to warn you when any of the vitals goes over a certain percentage of change.

Note: Please make sure you also adjust the number of file handles your process is allowed to use accordingly[10]. This affects the number of sockets you can use, and if that number is too low (default is often 1024), you will get connection issues first. 

Finally, the engineering devil on one of your shoulders should already have started to snicker about how horribly non-Erlang-y this is, and how you should use an event driven approach, possibly using Akka with Scala[11] – if you want to stay within the JVM world. Bear in mind though that the clever developers in the community share the same thoughts and have already started to discuss various approaches[12][13]. 

Links:

• [1] http://old.nabble.com/Re%3A-xceiverCount-257-exceeds-the-limit-of-concurrent-xcievers-256-p20469958.html
• [2] http://ccgtech.blogspot.com/2010/02/hadoop-hdfs-deceived-by-xciever.html
• [3] https://issues.apache.org/jira/browse/HDFS-1861 ”Rename dfs.datanode.max.xcievers and bump its default value”
• [4] https://issues.apache.org/jira/browse/HDFS-1866 ”Document dfs.datanode.max.transfer.threads in hdfs-default.xml”
• [5] http://hbase.apache.org/book.html#dfs.datanode.max.xcievers
• [6] http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_oom
• [7] https://issues.apache.org/jira/browse/HDFS-200 ”In HDFS, sync() not yet guarantees data available to the new readers”
• [8] https://issues.apache.org/jira/browse/HDFS-265 ”Revisit append”
• [9] http://search-hadoop.com/m/CBBoV3z24H1 ”HBase, mail # user – region size/count per regionserver”
• [10] http://hbase.apache.org/book.html#ulimit ”ulimit and nproc”
• [11] http://akka.io/ ”Akka”
• [12] https://issues.apache.org/jira/browse/HDFS-223 ”Asynchronous IO Handling in Hadoop and HDFS”
• [13] https://issues.apache.org/jira/browse/HDFS-918 ”Use single Selector and small thread pool to replace many instances of BlockSender for reads”

What follows is my bare-bones tutorial on getting Solr up and running to index each word of the complete works of Shakespeare. Note: Special thanks to Sematext for looking over the Solr bits and making sure they are sane. Check them out if you’re going to be doing a lot of work with Solr, ElasticSearch, or search in general and want to bring in the experts.

First things first

The way that I got started was by instantiating a new CentOS 6 Virtual Machine. You can pick a different flavor of Linux if that suits you; Hadoop should work fine on any (though advocated distros are SuSE, Ubuntu/Debian, RedHat/CentOS).

If you are fine with CentOS and want to skip some of the manual labor here, you can download a pre-loaded Virtual Machine from the Cloudera Downloads section, that already includes an installation of Sun Java 6u21 and CDH3u3. You can then skip ahead to installing Solr and downloading sample data as outlined below.

If you are proceeding with a new (virtual) machine, then follow along as follows: Make sure to disable SELinux, if applicable, and enable sshd. For CentOS6, that was done with the following commands:

[user@localhost ~]$ sudo chkconfig --levels 2345 sshd on
[user@localhost ~]$ /etc/init.d/sshd start
[user@localhost ~]$ vim /etc/selinux/config [set to disabled]

On that machine, download and install:

• Java – I’d recommend Java 6u26 as it has been tested with CDH
  http://www.oracle.com/technetwork/java/javase/downloads/index-jsp-138363.html
  Oracle Java never seems to play nicely with the /etc/alternative system (in my experience), so I force it to be the preferred JRE the old fashioned way:

[user@localhost ~]$ sudo rm /usr/bin/java
[user@localhost ~]$ sudo ln -s /usr/java/jdk1.6.0_26/jre/bin/java \
 /usr/bin/java

• Solr – download and unzip/untar in whatever directory that you like. For the purpose of this article, I’ll refer to it as
  http://lucene.apache.org/solr/downloads.html
• Hadoop – I am, obviously, biased. But my recommendation would be to use Cloudera Manager (free up to 50 nodes) to set up your VM as a pseudo-distributed cluster
  http://www.cloudera.com/products-services/tools/
• Sample data – Complete works of William Shakespeare. I’d recommend unzipping into a single directory. I’ll refer to it as
  http://www.ipl.org/div/shakespeare/

You can validate that all of the pieces are installed and running correctly by doing the following:

• Java

[user@localhost ~]$ java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

• Solr

[user@localhost ~]$ cd /example
[user@localhost ~]$ java -jar start.jar

The goal is to get up and running quickly here, so I am opting to use the Solr example configuration. Worth noting also that when run in this manner, the Solr server will be started with the default JVM heap size – which I believe to be the smaller of {1/4 system memory or 1GB}.

Now, you should be able to access the Solr administration GUI (one of the niceties of Solr!) via a web browser inside your VM with the address: http://localhost:8983/solr/admin

• Hadoop

You can validate that Hadoop is installed and running successfully by navigating in your VM’s browser to: http://localhost:7180, logging in as admin/admin, and seeing the following:

You probably want to export the client config XML files (can be done with a single click via Cloudera Manager – see the Generate Client Configuration buttong), copy them to /usr/lib/hadoop/conf, and then copy the sample text into hdfs:

[user@localhost ~]$ hadoop fs -put <shakespeare> shakespeare

Creating the indexing code

I have some history with Lucene from a past life, so the high level functionality of Solr was familiar to me. In a nutshell, you index files within Java code by creating a SolrInputDocument, which represents a single entity to index – a file or document generally – and using the .addField() to attach fields to this document that you’d later like to search.

The driver code for the indexer is very simple, in that it takes input file path(s) off the command line, and runs the mapper on the files that it finds. Note that it will accept a directory, and parse all of the files that it finds within.

public class IndexDriver extends Configured implements Tool {     

  public static void main(String[] args) throws Exception {
    //TODO: Add some checks here to validate the input path
    int exitCode = ToolRunner.run(new Configuration(),
     new IndexDriver(), args);
    System.exit(exitCode);
  }

  @Override
  public int run(String[] args) throws Exception {
    JobConf conf = new JobConf(getConf(), IndexDriver.class);
    conf.setJobName("Index Builder - Adam S @ Cloudera");
    conf.setSpeculativeExecution(false);

    // Set Input and Output paths
    FileInputFormat.setInputPaths(conf, new Path(args[0].toString()));
    FileOutputFormat.setOutputPath(conf, new Path(args[1].toString()));
    // Use TextInputFormat
    conf.setInputFormat(TextInputFormat.class);

    // Mapper has no output
    conf.setMapperClass(IndexMapper.class);
    conf.setMapOutputKeyClass(NullWritable.class);
    conf.setMapOutputValueClass(NullWritable.class);
    conf.setNumReduceTasks(0);
    JobClient.runJob(conf);
    return 0;
  }
}

The Map code is where things get more interesting. A couple notes before we proceed:

Solr servers may be used in 2 ways:

1. Via embedding a Solr server object within your Java code using EmbeddedSolrServer
2. Via HTTP requests, using the class CommonsHttpSolrServer with a URL (in our case, http://localhost:8983/solr)

In what follows, I elected to go with the StreamingUpdateSolrServer – which is a subclass of CommonsHttpSolrServer. More comments on that towards the end.

I will assume now that the reader has some familiarity with the Map/Reduce programming paradigm. The salient points for us here are that we will use a Map-only job to read through each file in the input that we provide, and index our chosen fields. Taking the path of least resistance, I used the fact that each line of text is it’s own Key/Value pair if we read the input as TextInputFormat, and I chose to index the following fields:

1. As a unique identifier for each word, I concatenated the filename, line offset (conveniently provided to the Map code as the “Key” because we are using TextInputFormat), and the position on that line of the word
2. The word itself

The Solr server obeys field definitions (specifying field names, data types, uniqueness, etc.) as dictated by a schema file. For this example, running Solr as indicated above, the schema is defined by/example/solr/conf/schema.xml

Per my choice to index 2 distinct fields, the relevant fields in the schema are:

<field name="id" type="string" indexed="true" \
 stored="true" required="true" />
<field name="text" type="text_general" indexed="true" \
 stored="true" multiValued="true"/>

Without further ado, then, the code looks like the following:

public class IndexMapper extends MapReduceBase implements
 Mapper <LongWritable, Text, NullWritable, NullWritable> {
  private StreamingUpdateSolrServer server = null;
  private SolrInputDocument thisDoc = new SolrInputDocument();
  private String fileName;
  private StringTokenizer st = null;
  private int lineCounter = 0;

  @Override
  public void configure(JobConf job) {
    String url = "http://localhost:8983/solr";
    fileName = job.get("map.input.file").substring(
      (job.get("map.input.file")).lastIndexOf(
      System.getProperty("file.separator")) +1);
      try {
        server = new StreamingUpdateSolrServer(url, 100, 5);
      } catch (MalformedURLException e) {
        e.printStackTrace();
      }
  }

  @Override
  public void map(LongWritable key, Text val,
   OutputCollector <NullWritable, NullWritable> output,
   Reporter reporter) throws IOException {

    st = new StringTokenizer(val.toString());
    lineCounter = 0;
    while (st.hasMoreTokens()) {
      thisDoc = new SolrInputDocument();
      thisDoc.addField("id", fileName + " "
       + key.toString() + " " + lineCounter++);
      thisDoc.addField("text", st.nextToken());
      try {
        server.add(thisDoc);
      } catch (SolrServerException e) {
        e.printStackTrace();
      } catch (IOException e) {
        e.printStackTrace();
      }
    }
  }

  @Override
  public void close() throws IOException {
  try {
      server.commit();
    } catch (SolrServerException e) {
      e.printStackTrace();
    }
  }
}

Compile the code how you see fit (I am old school and still use ant), and the job is ready to run!

To index all of the comedies, you can run the job with the compiled jar file as follows. Note that you must tell hadoop to include an additional Solr jar at runtime:

[user@localhost SolrTest]$ hadoop jar solrtest.jar \
 -libjars <solr_install_dir>/dist/apache-solr-solrj-3.5.0.jar \
 shakespeare/comedies shakespeare_output

If you then query the Solr server (via the web GUI at http://localhost:8983/solr/admin, the default search is *:* which works well for a quick test) you should see something like the following:

<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">35</int>
  <lst name="params">
    <str name="indent">on</str>
    <str name="start">0</str>
    <str name="q">*:*</str>
    <str name="version">2.2</str>
    <str name="rows">10</str>
  </lst>
 </lst>
<result name="response" numFound="377452" start="0">
<doc>
 <str name="id">troilusandcressida 0 0</str>
 <arr name="text">
  <str>TROILUS</str>
 </arr>
</doc>
<doc>
 <str name="id">troilusandcressida 0 1</str>
 <arr name="text">
  <str>AND</str>
 </arr>
</doc>
<doc>
 <str name="id">troilusandcressida 0 2</str>
 <arr name="text">
  <str>CRESSIDA</str>
 </arr>
</doc>

…

Further Tuning/Investigation Opportunities

Performance Implications of StreamingUpdateSolrServer – possibility of using EmbeddedSolrServer: What are the optimal tuning parameters for number of threads and batch size when using StreamingUpdateSolrServer? More investigation could be done here. It is also possible to use an EmbeddedSolrServer (per the Rackspace case study in Hadoop: The Definitive Guide), though it does add some maintenance overhead to create the indexes in a distributed fashion and then later re-combine. I opted to use the StreamingUpdateSolrServer because I believe that it is simpler to get up and running in a small test environment.

How to minimize the memory requirements in the Map code: I haven’t been a full time Java developer in many years, so there are almost certainly things that I’m missing on how to minimize the memory overhead of the objects used in the Map code. Since this is called for each line in the input, it is critical to make this code as lean as possible. One tip that I came across on this topic is to use (mutable) org.apache.hadoop.io.Text objects rather than (immutable) Strings. I avoided creating any new String objects in this example Map code, but the point is worth noting for other exercises.

Resources that I found useful

• A great primer that accomplished the indexing via Cascading:
  http://architects.dzone.com/articles/solr-hadoop-big-data-love
• Solr Tutorial:
  http://lucene.apache.org/solr/tutorial.html
• Some sample code for adding, updating, deleting documents on this wiki:
  http://wiki.apache.org/solr/Solrj
• My outstanding coworkers at Cloudera!

Cloudera Manager 3.7, a major new version of Cloudera’s Management applications for Apache Hadoop, is now available. Cloudera Manager Free Edition is a free download, and the Enterprise edition of Cloudera Manager is available as part of the Cloudera Enterprise subscription.

Cloudera Manager 3.7 includes several new features and enhancements:

• Automated Hadoop Deployment – Cloudera Manager 3.7 allows you to install the complete Hadoop stack in minutes.  We’ve now upgraded Cloudera Manager with the easy installation we first introduced in version 3.6 of SCM Express. (SCM Express is now replaced by Cloudera Manager Free Edition.).
• Centralized Management UI - Version 3.5 of the Cloudera Management Suite included distinct modules for Resource Management, Activity Monitoring and Service and Configuration Management. In Cloudera Manager 3.7, all of these feature sets are now integrated into one centralized browser-based administration console.
• Service & Configuration Management - We added several new configuration wizards to guide you in properly configuring HDFS and HBase host deployments, adding new hosts on demand, and adding/restarting services as needed. Cloudera Manager 3.7 now also manages Oozie and Hue.
• Service Monitoring – Cloudera Manager monitors the health of your key Hadoop services—HDFS, HBase, MapReduce—and displays alerts on suspicious or bad health. For example, to determine the health of HDFS, Cloudera Manager measures the percentage of corrupt, missing, or under-replicated blocks. Cloudera Manager also checks if the NameNode is swapping memory or spending too much time in Garbage Collection, and whether HDFS has enough free space. Trends in relevant metrics can be visualized through time-series charts.
• Log Search – You can search through all logs for Hadoop services across the whole cluster. You can also view results filtered by service, role, host, search phrase and log event severity.
• Events and Alerts – Cloudera Manager proactively reports on important events such as the change in a service’s health, detection of a log message of appropriate severity, or the slowness (or failure) of a job. Cloudera Manager aggregates the events for easy filtering and viewing, and you can configure Cloudera Manager to send email alerts.
• Global Time Control – You can view the state of your system for any time period in the past. Combined with health state, events and log information, this feature serves as a powerful diagnostic tool.
• Role-based Administration - Cloudera Manager 3.7 supports two types of users: admin users, who can change configs and execute commands and workflows; and read-only users, who can only monitor the system.
• Configuration versioning and Audit trails – You can view a complete history of configuration changes with user annotations. You can roll-back to previous configuration states.
• Activity Monitoring – The Activity Monitoring feature includes several performance and scale improvements.
• Operational Reports – The ‘Resource Manager’ feature in the Cloudera Management Suite 3.5 is now in Cloudera Manager’s ‘Reports’ feature. You can visualize disk usage by user, group, and directory; you can track MapReduce activity on the cluster by job, or by user.
• Support Integration – We’ve improved the Cloudera support experience by adding a feature that lets you send a snapshot of your cluster state to our support team for expedited resolution.
• Cloudera Manager Free Edition and 1-click Upgrade – The Free Edition of Cloudera Manager includes a subset of the features described above. After you install Cloudera Manager Free Edition, you can easily upgrade to the Enterprise edition by entering a license key. Your data will be preserved as the Cloudera Manager wizard guides you through the upgrade.

You can download the new Cloudera Manager 3.7 at: https://ccp.cloudera.com/display/SUPPORT/Downloads . Check it out. We look forward to your feedback.

P.S. : We’re hiring! Visit: http://www.cloudera.com/company/careers

Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. Flume is currently undergoing incubation at The Apache Software Foundation. More information on this project can be found at http://incubator.apache.org/flume. Flume NG is work related to new major revision of Flume and is the subject of this post.

Prior to entering the incubator, Flume saw incremental releases leading up to version 0.9.4. As Flume became adopted it became clear that certain design choices would need to be reworked in order to address problems reported in the field. The work necessary to make this change began a few months ago under the JIRA issue FLUME-728. This work currently resides on a separate branch by the name flume-728, and is informally referred to as Flume NG. At the time of writing this post Flume NG had gone through two internal milestones – NG Alpha 1, and NG Alpha 2 and a formal incubator release of Flume NG is in the works.

At a high-level, Flume NG uses a single-hop message delivery guarantee semantics to provide end-to-end reliability for the system. To accomplish this, certain new concepts have been incorporated into its design, while certain other existing concepts have been either redefined, reused or dropped completely.

In this blog post, I will describe the fundamental concepts incorporated in Flume NG and talk about it’s high-level architecture. This is a first in a series of blog posts by Flume team that will go into further details of it’s design and implementation.

Core Concepts

The purpose of Flume is to provide a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store. The architecture of Flume NG is based on a few concepts that together help achieve this objective. Some of these concepts have existed in the past implementation, but have changed drastically. Here is a summary of concepts that Flume NG introduces, redefines, or reuses from earlier implementation:

• Event: A byte payload with optional string headers that represent the unit of data that Flume can transport from it’s point of origination to it’s final destination.
• Flow: Movement of events from the point of origin to their final destination is considered a data flow, or simply flow. This is not a rigorous definition and is used only at a high level for description purposes.
• Client: An interface implementation that operates at the point of origin of events and delivers them to a Flume agent. Clients typically operate in the process space of the application they are consuming data from. For example, Flume Log4j Appender is a client.
• Agent: An independent process that hosts flume components such as sources, channels and sinks, and thus has the ability to receive, store and forward events to their next-hop destination.
• Source: An interface implementation that can consume events delivered to it via a specific mechanism. For example, an Avro source is a source implementation that can be used to receive Avro events from clients or other agents in the flow. When a source receives an event, it hands it over to one or more channels.
• Channel: A transient store for events, where events are delivered to the channel via sources operating within the agent. An event put in a channel stays in that channel until a sink removes it for further transport. An example of channel is the JDBC channel that uses a file-system backed embedded database to persist the events until they are removed by a sink. Channels play an important role in ensuring durability of the flows.
• Sink: An interface implementation that can remove events from a channel and transmit them to the next agent in the flow, or to the event’s final destination. Sinks that transmit the event to it’s final destination are also known as terminal sinks. The Flume HDFS sink is an example of a terminal sink. Whereas the Flume Avro sink is an example of a regular sink that can transmit messages to other agents that are running an Avro source.

These concepts help in simplifying the architecture, implementation, configuration and deployment of Flume.

Flow Pipeline

A flow in Flume NG starts from the client. The client transmits the event to it’s next hop destination. This destination is an agent. More precisely, the destination is a source operating within the agent. The source receiving this event will then deliver it to one or more channels. The channels that receive the event are drained by one or more sinks operating within the same agent. If the sink is a regular sink, it will forward the event to it’s next-hop destination which will be another agent. If instead it is a terminal sink, it will forward the event to it’s final destination. Channels allow the decoupling of sources from sinks using the familiar producer-consumer model of data exchange. This allows sources and sinks to have different performance and runtime characteristics and yet be able to effectively use the physical resources available to the system.

Figure 1 below shows how the various components interact with each other within a flow pipeline.

Figure 1: Schematic showing logical components in a flow. The arrows represent the direction in which events travel across the system. This also illustrates how flows can fan-out by having one source write the event out to multiple channels.

By configuring a source to deliver the event to more than one channel, flows can fan-out to more than one destination. This is illustrated in Figure 1 where the source within the operating Agent writes the event out to two channels – Channel 1 and Channel 2.

Conversely, flows can be converged by having multiple sources operating within the same agent write to the same channel. A example of the physical layout of a converging flow is show in Figure 2 below.

Figure 2: A simple converging flow on Flume NG.

Reliability and Failure Handling

Flume NG uses channel-based transactions to guarantee reliable message delivery. When a message moves from one agent to another, two transactions are started, one on the agent that delivers the event and the other on the agent that receives the event. In order for the sending agent to commit it’s transaction, it must receive success indication from the receiving agent. The receiving agent only returns a success indication if it’s own transaction commits properly first. This ensures guaranteed delivery semantics between the hops that the flow makes. Figure 3 below shows a sequence diagram that illustrates the relative scope and duration of the transactions operating within the two interacting agents.

Figure 3: Transactional exchange of events between agents.

This mechanism also forms the basis for failure handling in Flume NG. When a flow that passes through many different agents encounters a communication failure on any leg of the flow, the affected events start getting buffered at the last unaffected agent in the flow. If the failure is not resolved on time, this may lead to the failure of the last unaffected agent, which then would force the agent before it to start buffering the events. Eventually if the failure occurs when the client transmits the event to its first-hop destination, the failure will be reported back to the client which can then allow the application generating the events to take appropriate action.

On the other hand, if the failure is resolved before the first-hop agent fails, the buffered events in various agents downstream will start draining towards their destination. Eventually the flow will be restored to its original characteristic throughput levels. Figure 4 below illustrates a scenario where a flow comprising of two intermediary agents between the client and the central store go through a transient failure. The failure occurs between agent 2 and the central store, resulting in the events getting buffered at the agent 2 itself. Once the failing link has been restored to normal, the buffered events drain out to the central store and the flow is restored to its original throughput characteristics.

Figure 4: Failure handling in flows. In (a) the flow is normal and events can travel from the client to the central store. In (b) a communication failure occurs between Agent 2 and the event store resulting in events being buffered on Agent 2. In (c) the cause of failure was addressed and the flow was restored and any events buffered in Agent 2 were drained to the store.

Wrapping up

In this post I described the various concepts that are a part of Flume NG and its high-level architecture. This is first of a series of posts from the Flume team that will highlight the design and implementation of this system. In the meantime, if you need anymore information, please feel free to drop an email on the project’s user or developer lists, or alternatively file the appropriate JIRA issues. Your contribution in any form is welcome on the project.

Links:

Project Website: http://incubator.apache.org/flume/

Flume NG Getting Started Guide: https://cwiki.apache.org/confluence/display/FLUME/Getting+Started

Mailing Lists: http://incubator.apache.org/flume/mail-lists.html

Issue Tracking: https://issues.apache.org/jira/browse/FLUME

IRC Channel: #flume on irc.freenode.net

Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and write) over HTTP/S.

Hoop can be used to:

Hoop has a Hoop client and a Hoop server component:

Hoop and Hadoop HDFS Proxy

Hoop server is a full rewrite of Hadoop HDFS Proxy. Although it is similar to Hadoop HDFS Proxy (runs in a servlet-container, provides a REST API, pluggable authentication and authorization), Hoop server improves many of Hadoop HDFS Proxy shortcomings by providing:

Accessing HDFS files -via Hoop- using Unix ‘curl’ command

Assuming Hoop is running on http://hoopbar:14000, the following examples show how the Unix ‘curl’ command can be used to access data in HDFS via Hoop using pseudo authentication.

Getting the home directory:

$ curl -i "http://hoopbar:14000?op=homedir&user.name=babu"
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"homeDir":"http:\/\/hoopbar:14000\/user\/babu"}
$

Reading a file:

$ curl -i "http://hoopbar:14000?/user/babu/hello.txt&user.name=babu"
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Hello World!
$

Writing a file:

$ curl -i -X POST "http://hoopbar:14000/user/babu/data.txt?op=create" --data-binary @mydata.txt --header "content-type: application/octet-stream"
HTTP/1.1 200 OK
Location: http://hoopbar:14000/user/babu/data.txt
Content-Type: application/json
Content-Length: 0
$

Listing the contents of a directory:

$ curl -i "http://hoopbar:14000?/user/babu?op=list&user.name=babu"
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

[
  {
    "path" : "http:\/\/hoopbar:14000\/user\/babu\/data.txt"
    "isDir" : false,
    "len" : 966,
    "owner" : "babu",
    "group" : "supergroup",
    "permission" : "-rw-r--r--",
    "accessTime" : 1310671662423,
    "modificationTime" : 1310671662423,
    "blockSize" : 67108864,
    "replication" : 3
  }
]
$

Click this link for more details about the Hoop HTTP REST API.

Getting Hoop

Hoop is distributed with an Apache License 2.0.

The source code is available at http://github.com/cloudera/hoop.

Instructions on how to build, install and configure Hoop server and the rest of documentation is available at http://cloudera.github.com/hoop.

Contributing Hoop to Apache Hadoop

The goal is to contribute Hoop to Apache Hadoop as the next generation of Hadoop HDFS proxy. We are just waiting on the Mavenization of Hadoop Common and Hadoop HDFS which will make integration easier.

Data Interoperability

One might address this data interoperability in a variety of manners, including the following:

• Each system might be extended to read all the formats generated by the other systems. In the limit, this approach is not practical, since one cannot easily anticipate all of the formats new systems might generate.
• A library of data conversion programs could be assembled. This would unfortunately add a processing step, to convert the data between formats, slowing processing pipelines. Note however that many data conversion libraries operate by converting data into and out of a lingua franca format, using a single format as a pivot point.  This hints at a third possibility.
• Enable each system to read and write a common format.  Some systems might use other formats internally for performance, but whenever data is meant to be accessible to other systems a common format is used.

In practice all of these strategies will used to some extent. However the last strategy, a common format, seems to offer the most efficient path both in terms of engineering effort and processing time. This article will focus on the use of Avro’s data file format as such a common format.

Avro

Apache Avro is a data serialization format. Avro shares many features with Google’s Protocol Buffers and Apache Thrift, including:

• Rich data types.
• Fast, compact serialization.
• Support for many programming languages.
• Datatype evolution, also known as versioning.

Avro additionally provides some other features that are especially useful when storing data, namely:

• Avro defines a standard file format. Avro data files are self-describing, containing the full schema for the data in the file. Thus users can exchange Avro data files without also having to separately communicate metadata.  Once an Avro data file is written, one will always be able to read it, with full datatype information, without relying on any external software or metadata repository.  Avro data files also support compression, using Gzip or Snappy codecs.
• Avro’s serialization is more compact. Avro avoids storing a field identifier with each field value. For some datasets this savings can be significant.
• Avro implementations permit one to dynamically define new datatypes and to easily process previously unseen datatypes, without generation and loading of code. This provides natural support for script and query languages.
• Avro datatypes can define their sort-order, facillitating use of Avro data in MapReduce or ordered key/value stores.

Avro as a Common Format

Most of the major ecosystem components already or will soon support reading and writing Avro data files:

• MapReduce: I added support for Java MapReduce programs, included in Avro 1.4 and greater.
• Streaming: Tom White from Cloudera has added support for Hadoop Streaming programs to Avro (AVRO-808 & AVRO-830).
• Flume 0.9.2 and above support collecting data in Avro’s format (FLUME-133), contributed by Jon Hsieh of Cloudera.  Note also that Flume has recently been accepted into the Apache Incubator and will soon be known as Apache Flume.
• Sqoop 1.3 can import data as Avro data files in HDFS from a relational database (SQOOP-207), contributed by Tom White of Cloudera.  Sqoop has also recently been accepted into the Apache Incubator.
• Pig release 0.9 will be able read and write Avro data files (PIG-1748), thanks to Lin Guo and Jakob Homan at LinkedIn.
• Hive support for reading and writing Avro data files has been posted by Jakob Homan of LinkedIn, and should hopefully be included in Hive 0.9 (HIVE-895).
• HCatalog input and output drivers have been contributed by Tom White of Cloudera (HCATALOG-49).
• Thiruvalluvan M. G. from Yahoo! is working on a column-major format for Avro, which would accelerate Hive and Pig queries (AVRO-806).

For folks who are currently using Protocol Buffers or Thrift to store data, some tools for conversion are planned:

• Raghu Angadi from Twitter is working on tools that will let folks read and write their Thrift-defined data structures as Avro format data (AVRO-804).
• We also hope to soon add tools to convert between Protocol Buffers and Avro (AVRO-805).

At Cloudera we’re committed to helping Avro become a common format for the Hadoop ecosystem.  It’s great to see so many other companies and individuals also investing in Avro.

New users of Hadoop have generally found this assembly work to be too much trouble. To solve that problem, Cloudera currently distributes a patched version of Apache Hadoop, assembling work from Yahoo!, Cloudera, Facebook and others that has been committed to the Apache project, but not necessarily collectively available in one Apache release.

The Apache Hadoop project contains MapReduce, HDFS and Common. Cloudera packages these along with a number of complementary open-source projects — Apache HBase, Apache Pig, Apache Hive, Apache Zookeeper, Oozie, Flume, Hue, and others — that provide useful services for data management, access and use. Right now, HDFS, MapReduce and Common — the Apache Hadoop packages — are the only packages that we have to ship with a large collection of patches.

You can think of Apache Hadoop as similar to the Linux kernel: the heart of a larger system. In that case, Cloudera acts like Red Hat or Canonical, providing a complete platform that includes both the kernel and the most popular higher-level packages. We assemble & test the combined components, package them for easy installation, certify the integration of complementary systems and provide a predictable release schedule so users can plan upgrades and updates. Cloudera’s Distribution for Apache Hadoop is this larger package. It exists to make the power of Hadoop easily available to a larger audience of users.

We thank Yahoo! for its renewed efforts to make Apache Hadoop releases the very best versions of Hadoop. A more robust and powerful kernel makes the entire ecosystem stronger. One of the strengths of Apache Hadoop ecosystem has been the collective contributions of many organizations and individuals that has added up to hundreds of person-years of engineering investment. That investment dwarfs what any single organization or proprietary vendor could muster and this explains the strength and sophistication of the overall system.

Yahoo!’s commitment to open source development of Hadoop dates to the creation of the project. By concentrating its efforts on the Apache repository, Yahoo! makes a meaningful contribution to everyone in the Apache Hadoop community.  We very much hope that the larger Hadoop community will continue to work in the same way, working together to create excellent Apache releases that everyone can use. Certainly, Cloudera and our customers will benefit from high-quality releases from Apache that require minimal patching for production deployment. We believe that everyone else will, too.

A New Machine

Baseline

This section details the pre-hadoop installs I did.

Java

Previously I was running on Leopard, i.e. 10.4, and had to install soylatte to get the latest version of Java. In Snow Leopard, java jdk 1.6.0_22 is installed by default. That’s good enough for me, for now.

Gcc, etc.

In order to get these on the box, I had to install XCode, making sure to check the ‘linux dev tools’ option.

MacPorts

I installed MacPorts in case I needed to upgrade any native libs or tools.

Eclipse

I downloaded the 64 bit Java EE version of Helios.

Tomcat

Tomcat is part of my daily fun, and these instructions to install tomcat6 where helpful. One thing to note is that in order to access the tomcat manager panel, you also need to specify

<role rolename="manager"/>

prior to defining

<user username="admin" password="password" roles="standard,manager,admin"/>

Also, I run tomcat standalone (no httpd), so the mod_jk install part didnt apply. Finally, I chose not to daemonize tomcat because this is a dev box, not a server, and the instructions for compiling and using jsvc for 64 bit sounded iffy at best.

Hadoop

I use the CDH distro. The install was amazingly easy, and their support rocks. Unfortunately, they don’t have a dmg that drops Hadoop on the box configured and ready to run, so I need to build up my own psuedo mac node. This is what I want my mac to have (for starters):

1. 1. distinct processes for namenode, job tracker node, and datanode/task tracker nodes.
2. 2. formatted HDFS
3. 3. Pig 0.8.0

I’m not going to try to auto start hadoop because (again) this is a dev box, and start-all.sh should handle bringing up the JVMs around namenode, job tracker, datanode/tasktracker.

I am installing CDH3, because I’ve been running it in psuedo-mode on my Ubuntu dev box for the last month and have had no issues with it. Also, I want to run Pig 0.8.0, and that version may have some assumptions about the version of Hadoop that it needs.

All of the CDH3 Tarballs can be found at http://archive.cloudera.com/cdh/3/, and damn, that’s a lot of tarballs.

I downloaded hadoop 0.20.2+737, it’s (currently) the latest version out there. Because this is my new dev box, I decided to forego the usual security motivated setup of the hadoop user. When this decision comes back to bite me, I’ll be sure to update this post. In fact, for ease of permissions/etc, I decided to install under my home dir, under  a CDH3 dir, so I could group all CDH3 related installs together. I symlinked the hadoop-0.20+737 dir to hadoop, and I’ll update it if CDH3 updates their version of hadoop.

After untarring to the directory, all that was left was to make sure the ~/CDH3/hadoop/bin directory was in my .profile PATH settings.

Psuedo Mode Config

I’m going to set up Hadoop in psuedo distributed mode, just like I have on my Ubuntu box. Unlike Debian/Red Hat CDH distros, where this is an apt-get or yum command, I need to set up conf files on my own.

Fortunately the example-confs subdir of the Hadoop install has a conf.psuedo subdir. I needed to modify the following in core-site.xml:

 <property> 
     <name>hadoop.tmp.dir</name> 
     <value>changed_to_a_valid_dir_I_own</value> 
 </property>

and the following in hdfs-site.xml:

 <property> 
     <!-- specify this so that running 'hadoop namenode -format' formats the right dir --> 
     <name>dfs.name.dir</name> 
     <value>changed_to_a_different_dir_I_own</value> 
  </property> 

finally, I symlinked the conf dir at the top level of the Hadoop install to example-configs/conf.pseudo after saving off the original conf:

mv ./conf install-conf
ln sf ./example-confs/conf.pseudo conf

Pig

Installing Pig is as simple as downloading the tar, setting the path up, and going, sort of. The first time I ran pig, it tried to connect to the default install location of hadoop, /usr/lib/hadoop-0.20/. I made sure to set HADOOP_HOME to point to my install, and verified that the grunt shell connected to my configured HDFS (on port 8020).

More To Come

This psuedo node install was relatively painless. I’m going to continue to install Hadoop/HDFS based tools that may need more (HBase) or less (Hive) configuration, and update in successive posts.

Apache Hadoop has had methods of doing user authorization for some time. The Hadoop Distributed File System (HDFS) has a permissions model similar to Unix to control file and directory access, and MapReduce has access control lists (ACLs) per job queue to control which users may submit jobs. These authorization schemes allow Hadoop users and administrators to specify exactly who may access Hadoop’s resources. However, until recently, these mechanisms relied on a fundamentally insecure method of identifying the user who is interacting with Hadoop. That is, Hadoop had no way of performing reliable authentication. This limitation meant that any authorization system built on top of Hadoop, while helpful to prevent accidental unwanted access, could do nothing to prevent malicious users from accessing other users’ data.

Prior to the availability of Hadoop’s security features, the only way an organization could meet the requirement for data access protection was to run multiple distinct Hadoop clusters, and to segregate the groups who have network access to these clusters. This has obvious cost effectiveness implications, but, more importantly, limits the flexibility an organization has with respect to data storage options. One of the inherent powers of Hadoop is the ability to store and correlate all of an organization’s data. This is impossible if one must a priori relegate data to multiple distinct clusters based on security requirements. Furthermore, because of some organizations’ internal security policies, certain types of data could not be stored in Hadoop at all.

While this was acceptable for many of the first organizations to leverage Hadoop, the increase in Hadoop’s popularity and penetration into traditional enterprises necessitated the addition of better authentication mechanisms.

Among many of the new features introduced as part of CDH3 Beta 3, Hadoop now has the ability to provide strong authentication guarantees. The core Hadoop security work was done almost completely by Yahoo! and subsequently contributed to Apache Hadoop. Rather than create an ad hoc Hadoop-specific authentication scheme, Hadoop’s authentication system leverages Kerberos. Kerberos is an industry-standard authentication system developed by MIT which has been in existence since 1989. There are multiple open source implementations of Kerberos, including one produced and maintained by MIT itself. Kerberos is also the authentication system underpinning many proprietary identity management systems commonly found in enterprise environments, including Microsoft’s Active Directory. Hadoop’s support of Kerberos enables organizations to seamlessly integrate the new authentication features of Hadoop with their existing authentication and single sign-on systems.

All of the components of CDH3 Beta 3 now have support for interacting with secure Hadoop clusters, and many have incorporated additional security features which were previously impossible or impractical to implement with the security limitations inherent in Hadoop itself. Because of the complexity of integrating with multiple third-party authentication systems, configuring Hadoop and its associated components to use these systems is non-trivial.

Cloudera is pleased to announce the general availability of Cloudera’s “CDH3 Security Guide”. In this comprehensive guide, you’ll find instructions for enabling the security features of Hadoop itself, as well as for configuring all of the other components of CDH to be able to interact with a Hadoop cluster with security enabled. You’ll also find a troubleshooting guide for debugging common errors encountered when configuring a secure Hadoop environment, as well as details for configuring Hadoop’s authentication mechanism to use Active Directory. Please email cdh-user@cloudera.org if you have any questions or encounter any issues.