Before You Begin Upgrading to CDH 5 Using the Command Line
Before upgrading, be sure to read about the latest Incompatible Changes and Known Issues in CDH 5 in the CDH 5 Release Notes. If you are currently running MRv1, you should read CDH 5 and MapReduce before proceeding.
If you are upgrading a cluster that is part of a production system, be sure to plan ahead. As with any operational work, be sure to reserve a maintenance window with enough extra time allotted in case of complications. The Hadoop upgrade process is well understood, but it is best to be cautious. For production clusters, Cloudera recommends allocating up to a full day maintenance window to perform the upgrade, depending on the number of hosts, the amount of experience you have with Hadoop and Linux, and the particular hardware you are using.
Install Java 1.7
Delete Symbolic Links in HDFS
If there are symbolic links in HDFS when you upgrade from CDH 4 to CDH 5, the upgrade will fail and you will have to downgrade to CDH 4, delete the symbolic links, and start over. To prevent this, proceed as follows.
- cd to the directory on the NameNode that contains the latest fsimage The location of this directory is specified as the value of dfs.namenode.name.dir (or dfs.name.dir) in hdfs-site.xml.
- Use a command such as the following to write out the path names in the fsimage:
$ hdfs oiv -i FSIMAGE -o /tmp/YYYY-MM-DD_FSIMAGE.txt
- Use a command such as the following to find the path names of any symbolic links listed in /tmp/YYYY-MM-DD_FSIMAGE.txt and write them out to the file /tmp/symlinks.txt:
$ grep -- "->" /tmp/YYYY-MM-DD_FSIMAGE.txt > /tmp/symlinks.txt
- Delete any symbolic links listed in /tmp/symlinks.txt.
Check Hue Table Sizes and Cleanup if Necessary
- Stop the Hue service.
- Back up the Hue database.
- Download the history cleanup script to the host running the Hue Server.
- Run the following as root:
- parcel installation
export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -1 /var/run/cloudera-scm-agent/process | grep HUE| sort -n | tail -1 `" /opt/cloudera/parcels/CDH/share/hue/build/env/bin/hue shell
- package installation
export HUE_CONF_DIR="/var/run/cloudera-scm-agent/process/`ls -1 /var/run/cloudera-scm-agent/process | grep HUE| sort -n | tail -1 `" /usr/share/hue/build/env/bin/hue shell
- parcel installation
- Run the downloaded script in the Hue shell.
Considerations for Secure Clusters
If you are upgrading a cluster that has Kerberos security enabled, you must do the following:
- Before starting the upgrade, make sure your installation is properly configured according to the instructions in the installation and configuration sections of the Cloudera Security guide.
- Before shutting down Hadoop services, put the NameNode into safe mode and perform a saveNamespace operation; see the instructions on backing up the metadata.