Configuring CDH Components for Hue

To enable communication between the Hue Server and CDH components, you must make minor changes to your CDH installation by adding the properties described in this section to your CDH configuration files in /etc/hadoop-0.20/conf/ or /etc/hadoop/conf/. If you are installing on a cluster, make the following configuration changes to your existing CDH installation on each node in your cluster.

WebHDFS or HttpFS Configuration

Hue can use either of the following to access HDFS data:

  • WebHDFS provides high-speed data transfer with good locality because clients talk directly to the DataNodes inside the Hadoop cluster.
  • HttpFS is a proxy service appropriate for integration with external systems that are not behind the cluster's firewall.

Both WebHDFS and HttpFS use the HTTP REST API so they are fully interoperable, but Hue must be configured to use one or the other. For HDFS HA deployments, you must use HttpFS.

To configure Hue to use either WebHDFS or HttpFS, do the following steps:

  1. For WebHDFS only:
    1. Add the following property in hdfs-site.xml to enable WebHDFS in the NameNode and DataNodes:
      <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
      </property>
    2. Restart your HDFS cluster.
  2. Configure Hue as a proxy user for all other users and groups, meaning it may submit a request on behalf of any other user:

    WebHDFS: Add to core-site.xml:

    <!-- Hue WebHDFS proxy user setting -->
    <property>
      <name>hadoop.proxyuser.hue.hosts</name>
      <value>*</value>
    </property>
    <property>
      <name>hadoop.proxyuser.hue.groups</name>
      <value>*</value>
    </property>

    HttpFS: Verify that /etc/hadoop-httpfs/conf/httpfs-site.xml has the following configuration:

    <!-- Hue HttpFS proxy user setting -->
    <property>
      <name>httpfs.proxyuser.hue.hosts</name>
      <value>*</value>
    </property>
    <property>
      <name>httpfs.proxyuser.hue.groups</name>
      <value>*</value>
    </property>
    If the configuration is not present, add it to /etc/hadoop-httpfs/conf/httpfs-site.xml and restart the HttpFS daemon.
  3. Verify that core-site.xml has the following configuration:
    <property>  
    <name>hadoop.proxyuser.httpfs.hosts</name>  
    <value>*</value>  
    </property>  
    <property>  
    <name>hadoop.proxyuser.httpfs.groups</name>  
    <value>*</value>  
    </property>  
    If the configuration is not present, add it to /etc/hadoop/conf/core-site.xml and restart Hadoop.
  4. With root privileges, update hadoop.hdfs_clusters.default.webhdfs_url in hue.ini to point to the address of either WebHDFS or HttpFS.
    [hadoop]
    [[hdfs_clusters]]
    [[[default]]]
    # Use WebHdfs/HttpFs as the communication mechanism.
    WebHDFS:
    ...
    webhdfs_url=http://FQDN:50070/webhdfs/v1/

    HttpFS:

    ...
    webhdfs_url=http://FQDN:14000/webhdfs/v1/

MRv1 Configuration

Hue communicates with the JobTracker using the Hue plugin, which is a .jar file that should be placed in your MapReduce lib directory.

If your JobTracker and Hue Server are located on the same host, copy the file over. If you are currently using CDH 4, your MapReduce library directory might be in /usr/lib/hadoop/lib.

$ cd /usr/lib/hue
$ cp desktop/libs/hadoop/java-lib/hue-plugins-*.jar /usr/lib/hadoop-0.20-mapreduce/lib

If your JobTracker runs on a different host, scp the Hue plugins .jar file to the JobTracker host.

Add the following properties to mapred-site.xml:

<property>
  <name>jobtracker.thrift.address</name>
  <value>0.0.0.0:9290</value>
</property>
<property>
  <name>mapred.jobtracker.plugins</name>
  <value>org.apache.hadoop.thriftfs.ThriftJobTrackerPlugin</value>
  <description>Comma-separated list of jobtracker plug-ins to be activated.</description>
</property>

You can confirm that the plugins are running correctly by tailing the daemon logs:

$ tail --lines=500 /var/log/hadoop-0.20-mapreduce/hadoop*jobtracker*.log | grep ThriftPlugin
2009-09-28 16:30:44,337 INFO org.apache.hadoop.thriftfs.ThriftPluginServer: Starting Thrift server
2009-09-28 16:30:44,419 INFO org.apache.hadoop.thriftfs.ThriftPluginServer:
Thrift server listening on 0.0.0.0:9290

Oozie Configuration

In order to run DistCp, Streaming, Pig, Sqoop, and Hive jobs in Job Designer or the Oozie Editor/Dashboard application, you must make sure the Oozie shared libraries are installed for the correct version of MapReduce (MRv1 or YARN). See Installing the Oozie ShareLib in Hadoop HDFS for instructions.

To configure Hue as a default proxy user, add the following properties to /etc/oozie/conf/oozie-site.xml:
<!-- Default proxyuser configuration for Hue -->
<property>
    <name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>
    <value>*</value>
</property>
<property>
    <name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>
    <value>*</value>
</property>

Search Configuration

See Search Configuration for details on how to configure the Search application for Hue.

HBase Configuration

See HBase Configuration for details on how to configure the HBase Browser application.

Hive Configuration

The Beeswax daemon has been replaced by HiveServer2. Hue should therefore point to a running HiveServer2. This change involved the following major updates to the [beeswax] section of the Hue configuration file, hue.ini.

[beeswax]
  # Host where Hive server Thrift daemon is running.
  # If Kerberos security is enabled, use fully-qualified domain name (FQDN).
  ## hive_server_host=<FQDN of HiveServer2>

  # Port where HiveServer2 Thrift server runs on.
  ## hive_server_port=10000

Existing Hive Installation

In the Hue configuration file hue.ini, modify hive_conf_dir to point to the directory containing hive-site.xml.

No Existing Hive Installation

Familiarize yourself with the configuration options in hive-site.xml. See Hive Installation. Having a hive-site.xml is optional but often useful, particularly on setting up a metastore. You can locate it using the hive_conf_dir configuration variable.

Permissions

See File System Permissions in the Hive Installation section.