Comments on: Hadoop HA Configuration

By: masterpapers

masterpapers — Wed, 29 Jun 2011 17:09:36 +0000

Great job, you’ve helped me so much and I am so glad I have chosen your service.

By: ???????HBase (1) | Brizzz.com – ???

???????HBase (1) | Brizzz.com – ??? — Wed, 20 Apr 2011 08:03:30 +0000

[...] used DRBD and Heartbeat to have HA for the Hadoop NameNode, because this was the last Single Point of Failure (SPOF) in our [...]

By: nishat akhtar

nishat akhtar — Sun, 05 Sep 2010 19:13:08 +0000

I am trying to set up hive on my hadoop. I am also able to start hive after mentioning the hadoop path but i am unable to create any tables as every time i am getting the metadata error. Can you please help me out to find the solution.

By: PATRICK

PATRICK — Sat, 21 Aug 2010 17:11:05 +0000

Charlie, et.al.,

Charlie – you are right, that appears to be an error in the blog. You do not want to set dfs.data.dir to a drbd partition. I’m also glad that you resolved the split-brain issue.

That said, we want to clarify our position that we do not officially support nor fully endorse this method of doing NameNode HA. DRBD is hard to get right, and you can easily corrupt your fsimage file with a subtle mistake. Furthermore, the backup NameNode will start up and enter safe-mode (read only) until it has gone through and processed the edit log to reconstitute the up-to-date fsimage in-memory. This process can take several minutes, sometimes over an hour if you let the edit log grow unchecked. You can alleviate this by having the SecondaryNameNode checkpoint more often, and backing up the checkpoints so you have a fairly recent image as a fallback in case of catastrophic failure.

The community is hard at work at building high-availability natively into HDFS. Until then, our recommendation is to design your application around HDFS’s availability semantics. This might include using Flume for reliable delivery, or having mirrored HDFS systems.

By: Charlie

Charlie — Fri, 20 Aug 2010 14:46:34 +0000

Scrap question number 2, it was a split-brain with drbd which stopped the sync’ing, :-/

By: Charlie

Charlie — Fri, 20 Aug 2010 13:29:34 +0000

Firstly thanks for the post, its very helpful but I have a couple of questions:

1. You say in the few configuration options that you have specified that dfs.data.dir should be set to a directory on the drbd partition, but from what I understand about this option all it does is tells the data nodes where to store its blocks, is this right? If so, the masters/namenodes do not worry about this option and it should therefore not be set to a directory on the drbd partition.

2. I have set this up on 4 machines for testing, 2 namenodes with drbd/heartbeat and 2 nodes, of which 1 of them is also acting as the secondary namenode. All is working fine apart from the very big problem of when the namenode is switched over from 1 master to the other, the data stored in hdfs is no longer there, I therefore seem to have 2 different hdfs’s, I can create directories in both and they only appear while that particular master is running. Any’s idea’s why this would happen?

By: Hadoop/HBase automated deployment using Puppet at hstack

Hadoop/HBase automated deployment using Puppet at hstack — Thu, 01 Jul 2010 09:34:04 +0000

[...] standalone puppet module to configure the Hadoop NameNode in High-Availability mode via DRBD, heartbeat and mon. For more details on this recipe check out the cloudera blog post on this topic. [...]

By: Carlos

Carlos — Wed, 09 Jun 2010 14:09:38 +0000

I am confuse about the picture on this post.
We are trying to construct out namenode with HA ,
so we wanna using DRBD + HeartBeat to provide namenode a failover system.
But in the picture , it looks like the backup namemode have it’s own datanode cluster , why ?
Do we need two cluster for the HA configuration ??

By: Sean

Sean — Tue, 18 May 2010 14:14:40 +0000

Thx for posting! Very helpful. I was able to get most everything working. Just a somewhat minor issue when failing over: DRBD can’t release its resource resulting in an what Heartbeat calls an ungraceful shutdown forcing a reboot. Master2 comes up fine though.

By: Jeff

Jeff — Tue, 30 Mar 2010 17:46:20 +0000

The one thing missing from this post is the actual hadoop config info for the namenode and backup namenod. The post simply says you used the cloudera configuration tool to generate a custom config….As we all know this tool has been unavailable for quite some time. So a little more insight as to the actual hdfs/mapred configs would be like the icing on the cake that this post already is.