Performing a Rolling Upgrade to CDH 5.3

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

The rolling upgrade feature takes advantage of parcels and the HDFS high availability to enable you to upgrade your cluster software and restart the upgraded services without taking the entire cluster down. You must have HDFS high availability enabled to perform a rolling upgrade.

This page described how to perform a rolling upgrade between maintenance and minor versions of CDH 5, except Beta versions. For rolling upgrade between CDH 4 versions, see Performing a Rolling Upgrade on a CDH 4 Cluster.

It is not possible to perform a rolling upgrade from CDH 4 to CDH 5 because of incompatibilities between the two major versions. Instead, follow the instructions for a full upgrade at Upgrading from CDH 4 to CDH 5 Parcels.

Before You Begin

  • Ensure Java 1.7 is installed across the cluster. For installation instructions and recommendations, see Upgrading to Oracle JDK 1.7 in a Cloudera Manager Deployment, and make sure you have read Known Issues and Workarounds in Cloudera Manager 5 before you proceed with the upgrade.
  • To avoid a high number of alerts during the upgrade process, you can enable maintenance mode on your cluster before you start the upgrade. This stops email alerts and SNMP traps from being sent, but does not stop checks and configuration validations. Exit maintenance mode when you have finished the upgrade to reenable Cloudera Manager alerts.
  • When upgrading Impala, whether in CDH or a standalone parcel or package, check your SQL against the newest reserved words listed in incompatible changes. If upgrading across multiple versions or in case of any problems, check against the full list of Impala keywords.

Ensure High Availability is Enabled

To enable high availability, see HDFS High Availability for instructions. You do not need to enable automatic failover for rolling restart to work, though you can enable it if you want. Automatic failover does not affect the rolling restart operation. If you have JobTracker high availability configured, Cloudera Manager will fail over the JobTracker during the rolling restart, but this is not a requirement for performing a rolling upgrade.

Back up HDFS Metadata

If upgrading from:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher

Back up HDFS metadata using the following command:

hdfs dfsadmin -fetchImage local directory

Run the Upgrade Wizard

  1. Log into the Cloudera Manager Admin console.
  2. From the Home > Status tab, click next to the cluster name and select Upgrade Cluster. The Upgrade Wizard starts.
  3. If the option to choose between packages and parcels displays, select the Use Parcels option.
  4. In the Choose CDH Version (Parcels) field, select the CDH version. If there are no qualifying parcels, click the Modify the Remote Parcel Repository URLs link to go to the Parcel Configuration Settings page where you can add the locations of parcel repositories. Click Continue.
  5. Read the notices for steps you must complete before upgrading, click the Yes, I ... checkboxes after completing the steps, and click Continue.
  6. Cloudera Manager checks that hosts have the correct software installed. Click Continue.
  7. The selected parcels are downloaded and distributed. Click Continue.
  8. The Host Inspector runs and displays the CDH version on the hosts. Click Continue. The Choose Upgrade Procedure displays.
  9. Choose the type of upgrade and restart:
    • Rolling Restart (Default) - Cloudera Manager upgrades services and performs a rolling restart. This option is available only if you have enabled high availability for HDFS.
      1. Choose the batch size (default 1) described in Performing a Cluster-Level Rolling Restart and click Advanced Options to specify additional rolling restart options. Services that do not support rolling restart undergo a normal restart and are not available during the restart process.
      2. Click Continue. The Upgrade Cluster Command screen displays the result of the commands run by the wizard as it shuts down services, activates the new parcel, upgrades services, deploys client configuration files, restarts services, and performs a rolling restart of the services that support it.
    • Full Cluster Restart - Cloudera Manager performs all service upgrades and restarts the cluster, during which services are not available.
      1. Click Continue. The Upgrade Cluster Command screen displays the result of the commands run by the wizard as it shuts down all services, activates the new parcel, upgrades services, deploys client configuration files, and restarts services.
      2. Click Continue. The wizard reports the result of the upgrade.
    • Manual Upgrade - Cloudera Manager configures the cluster to the specified CDH version but performs no upgrades or service restarts. Upgrading manually is complex and for advanced users only.
      1. Click Continue. Cloudera Manager displays links to documentation describing the required upgrade steps.
  10. Click Continue.
  11. Click Finish to return to the Home page.

Recover from Failed Steps

The actions performed by the upgrade wizard are listed in Upgrade Wizard Actions. If any of the steps in the Command Progress screen fails, complete the steps as described in that section before proceeding.

Remove CDH 5 Packages

If your previous installation of CDH was done using packages, remove those packages on all hosts on which you installed the parcels and refresh the symlinks so that clients will run the new software versions. Skip this step if your previous installation was using parcels.

  1. If your Hue service uses the embedded SQLite database, back up /var/lib/hue/desktop.db to a location that is not /var/lib/hue because this directory is removed when the packages are removed.
  2. Uninstall the CDH packages on each host:
    • Not including Impala and Search
      Operating System Command
      RHEL
      $ sudo yum remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
      SLES
      $ sudo zypper remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
      Ubuntu or Debian
      $ sudo apt-get purge bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
    • Including Impala and Search
      Operating System Command
      RHEL
      $ sudo yum remove 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
      SLES
      $ sudo zypper remove 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
      Ubuntu or Debian
      $ sudo apt-get purge 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
  3. Restart all the Cloudera Manager Agents to force an update of the symlinks to point to the newly installed components on each host:
    sudo service cloudera-scm-agent restart
  4. If your Hue service uses the embedded SQLite DB, restore the DB you backed up:
    1. Stop the Hue service.
    2. Copy the backup from the temporary location to the newly created Hue database directory, /var/lib/hue.
    3. Start the Hue service.

Finalize HDFS Rolling Upgrade

If upgrading from:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher
Finalize the HDFS metadata upgrade. To determine when to finalize the upgrade, run important workloads and ensure that they are successful. After you have finalized the upgrade, you cannot roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time. Make sure you have enough free disk space, keeping in mind that the following behavior continues until the upgrade is finalized:
  • Deleting files does not free up disk space.
  • Using the balancer causes all moved replicas to be duplicated.
  • All on-disk data representing the NameNodes metadata is retained, which could more than double the amount of space required on the NameNode and JournalNode disks.
To finalize the metadata upgrade:
  1. Go to the HDFS service.
  2. Select Actions > Finalize Rolling Upgrade and click Finalize Rolling Upgrade to confirm.