CDH4 includes a FUSE (Filesystem in Userspace) interface into HDFS. FUSE enables you to write a normal userland application as a bridge for a traditional filesystem interface. The hadoop-hdfs-fuse package enables you to use your HDFS cluster as if it were a traditional filesystem on Linux. It is assumed that you have a working HDFS cluster and know the hostname and port that your NameNode exposes.
To install fuse-dfs On Red Hat-compatible systems:
$ sudo yum install hadoop-hdfs-fuse
To install fuse-dfs on Ubuntu systems:
$ sudo apt-get install hadoop-hdfs-fuse
To install fuse-dfs on SLES systems:
$ sudo zypper install hadoop-hdfs-fuse
You now have everything you need to begin mounting HDFS on Linux. To set up and test your mount point:
$ mkdir -p <mount_point> $ hadoop-fuse-dfs dfs://<name_node_hostname>:<namenode_port> <mount_point>
You can now run operations as if they are on your mount point. Press Ctrl+C to end the fuse-dfs program, and umount the partition if it is still mounted.
If you are using SLES 11 with the Oracle JDK 6u26 package, hadoop-fuse-dfs may exit immediately because ld.so can't find libjvm.so. To work around this issue, add /usr/java/latest/jre/lib/amd64/server to the LD_LIBRARY_PATH.
To clean up your test:
$ umount <mount_point>
You can now add a permanent HDFS mount which persists through reboots. To add a system mount:
- Open /etc/fstab and add lines to the bottom similar to these:
hadoop-fuse-dfs#dfs://<name_node_hostname>:<namenode_port> <mount_point> fuse allow_other,usetrash,rw 2 0
hadoop-fuse-dfs#dfs://localhost:8020 /mnt/hdfs fuse allow_other,usetrash,rw 2 0
- Test to make sure everything is working properly:
$ mount <mount_point>
Your system is now configured to allow you to use the ls command and use that mount point as if it were a normal system disk.
You can change the JVM maximum heap size. By default, the CDH4 package installation creates the /etc/default/hadoop-fuse file with the following default JVM maximum heap size. To change it:
For more information, see the help for hadoop-fuse-dfs:
$ hadoop-fuse-dfs --help