Upgrading to CDH 5.x Using a Rolling Upgrade

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

This topic describes how to perform a rolling upgrade from any version of CDH 5.x to a higher version of CDH 5.x, using Cloudera Manager and parcels.The minor version of Cloudera Manager you use to perform the upgrade must be equal to or greater than the CDH minor version. To upgrade Cloudera Manager, see Overview of Upgrading Cloudera Manager.

A rolling upgrade allows you to upgrade your cluster software and restart the upgraded services without taking the cluster offline. Performing a rolling upgrade requires the following:

You cannot perform a rolling upgrade from CDH 4 to CDH 5 because of incompatibilities between the two major versions. Instead, follow the instructions for a full upgrade at Upgrading from CDH 4 to CDH 5 Parcels.

Step 1: Collect Upgrade Information

Before starting an upgrade, collect the following information:
  1. Host credentials. You must have SSH access and be able to log in using a root account or an account that has password-less sudo permission.
  2. The version of Cloudera Manager used in your cluster. Go to Support > About.
  3. The version of the JDK deployed in the cluster. Go to Support > About.
  4. The version of CDH. The CDH version number displays next to the cluster name on the Home page.
  5. Whether the cluster was installed using parcels or packages. This information displays next to the CDH version on the Home page of Cloudera Manager.
  6. The services enabled in your cluster. Go to Clusters > Cluster name.
  7. Operating system type and version. Go to Hosts and click on a hostname in the list. The operating system type and version displays in the Distribution row in the Details section.
  8. Database information for the databases used by Sqoop, Oozie, Hue, Hive Metastore, and Sentry Server (information is only required if theses services are enabled in the cluster).
    Gather the following information:
    • Type of database (PostgreSQL, Embedded PostgreSQL, MySQL, MariaDB, or Oracle)
    • Hostnames of the databases
    • Credentials for the databases
    To locate database information:
    • Sqoop, Oozie, and Hue – Go to Cluster Name > Configuration > Database Settings.
    • Hive Metastore – Go to the Hive service, select Configuration, and select the Hive Metastore Database category.
    • Sentry – Go to the Sentry service, select Configuration, and select the Sentry Server Database category.

Step 2: Complete Pre-Upgrade Steps

  1. Review the CDH 5 and Cloudera Manager 5 Requirements and Supported Versions for the new versions you are upgrading to.
  2. Read the CDH 5 Release Notes.
  3. Read the Cloudera Security Bulletins.
  4. Ensure that Java 1.7 or 1.8 is installed across the cluster. For installation instructions and recommendations, see Upgrading to Oracle JDK 1.7 in a Cloudera Manager Deployment or Upgrading to Oracle JDK 1.8, and make sure you have read Known Issues and Workarounds in Cloudera Manager 5 before you proceed with the upgrade.
  5. Ensure that the Cloudera Manager minor version is equal to or greater than the CDH minor version. For example:
    • Targeted CDH Version: 5.0.5

      Minimum Cloudera Manager Version: 5.0.x

    • Targeted CDH Version: 5.8.2

      Minimum Cloudera Manager Version: 5.8.x

  6. If you are upgrading from CDH 5.1 or lower, and use Hive Date partition columns, you may need to update the date format. See Date partition columns.
  7. If the cluster uses Impala, check your SQL against the newest reserved words listed in incompatible changes. If upgrading across multiple versions, or in case of any problems, check against the full list of Impala keywords.
  8. Run the Host Inspector and fix every issue. Go to Cluster > Inspect Hosts.
  9. Run the Security Inspector and fix and reported errors. (Go to Administration > Security and click Security Inspector.)
  10. Log in to any cluster node as the hdfs user, run the following commands, and correct any reported errors:
    hdfs fsck /
    hdfs dfsadmin -report
    See HDFS Commands Guide in the Apache Hadoop documentation.
  11. Log in to any DataNode as the hbase user, run the following command, and correct any reported errors:
    hbase hbck 
    See Checking and Repairing HBase Tables.
  12. Review the upgrade procedure and reserve a maintenance window with enough time allotted to perform all steps. For production clusters, Cloudera recommends allocating up to a full day maintenance window to perform the upgrade, depending on the number of hosts, the amount of experience you have with Hadoop and Linux, and the particular hardware you are using.
  13. To avoid unnecessary alerts during the upgrade process, enable maintenance mode on your cluster before you start the upgrade. This stops email alerts and SNMP traps from being sent, but does not stop checks and configuration validations. Be sure to exit maintenance mode when you have finished the upgrade to reenable Cloudera Manager alerts.
  14. If you have configured Hue to use TLS/SSL and you are upgrading from CDH 5.2 or lower to CDH 5.3 or higher, Hue validates CA certificates and requires a truststore. To create a truststore, follow the instructions in Hue as a TLS/SSL Client.
  15. If your cluster uses the Flume Kafka client, and you are upgrading to CDH 5.8.0 or CDH 5.8.1, perform the extra steps described in Upgrading to CDH 5.8.0 or CDH 5.8.1 When Using the Flume Kafka Client and then continue with the procedures in this topic.
  16. If your cluster uses Impala and Llama, this role has been deprecated as of CDH 5.9 and you must remove the role from the Impala service before starting the upgrade. If you do not remove this role, the upgrade wizard will halt the upgrade.
    To determine if Impala uses Llama:
    1. Go to the Impala service.
    2. Select the Instances tab.
    3. Examine the list of roles in the Role Type column. If Llama appears, the Impala service is using Llama.
    To remove the Llama role:
    1. Go to the Impala service and select Actions > Disable YARN and Impala Integrated Resource Management.

      The Disable YARN and Impala Integrated Resource Management wizard displays.

    2. Click Continue.

      The Disable YARN and Impala Integrated Resource Management Command page displays the progress of the commands to disable the role.

    3. When the commands have completed, click Finish.

Step 3: Ensure High Availability Is Enabled

See HDFS High Availability for instructions. Enabling automatic failover is optional. Automatic failover does not affect the rolling restart operation. If you have JobTracker high availability configured, Cloudera Manager will fail over the JobTracker during the rolling restart, but configuring JobTrackrer high availability is not a requirement for performing a rolling upgrade.

Step 4: Back Up HDFS Metadata

[Not required for CDH maintenance release upgrades.]

The steps in this section are only required for the following upgrades:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher

Back up HDFS metadata using the following command:

hdfs dfsadmin -fetchImage local directory

Step 5: Back Up Databases

Back up the databases for any of the following services that are deployed in your cluster:
Service Databases to Back Up
Service Where to find database information
Sqoop Go to Clusters > Cluster Name > Sqoop service > Configuration and select the Database category.
Hue Go to Clusters > Cluster Name > Hue service > Configuration and select the Database category.
Oozie Go to Clusters > Cluster Name > Oozie service > Configuration and select the Database category.
Cloudera Navigator Audit Server Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Cloudera Navigator Metadata Server Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Activity Monitor Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Reports Manager Go to Clusters > Cloudera Management Service > Configuration and select the Database category.
Sentry Server Go to Clusters > Cluster Name > Sentry service > Configuration and select the Sentry Server Database category.
Hive Metastore Go to Clusters > Cluster Name > Hive service > Configuration and select the Hive Metastore Database category.
To back up the databases:
  1. If not already stopped, stop the service:
    1. On the Home > Status tab, click to the right of the service name and select Stop.
    2. Click Stop in the next screen to confirm. When you see a Finished status, the service has stopped.
  2. Back up the database. See Backing Up Databases for detailed instructions for each supported type of database.
  3. Restart the service:
    1. On the Home > Status tab, click to the right of the service name and select Start.
    2. Click Start that appears in the next screen to confirm. When you see a Finished status, the service has started.

Step 6: Run the Upgrade Wizard

  1. From the Home > Status tab, click next to the cluster name and select Upgrade Cluster.

    The Getting Started page of the upgrade wizard displays.

  2. If the option to pick between packages and parcels displays, select Use Parcels.
  3. In the Choose CDH Version (Parcels) field, select the CDH version. If no qualifying parcels are listed, or you want to upgrade to a different version, click the Modify the Remote Parcel Repository URLs link to go to the configuration page for Remote Parcel Repository URLs and add the appropriate URL to the configuration. See Parcel Configuration Settings for information about entering the correct URL for parcel repositories. Click Continue.
  4. Read the notices for steps you must complete before upgrading, click the Yes, I ... checkboxes after completing the steps, and click Continue.

    Cloudera Manager verifies that hosts have the correct software installed.

  5. Click Continue.

    The selected parcels are downloaded and distributed.

  6. Click Continue.

    The Host Inspector runs and displays the CDH version on the hosts.

  7. Click Continue.

    The Choose Upgrade Procedure screen displays.

  8. Select Rolling Restart . Cloudera Manager upgrades services and performs a rolling restart. This option is only available if you have enabled high availability for HDFS. Services that do not support rolling restart undergo a normal restart, and are not available during the restart process.
  9. (Optional) Configure the following parameters for the rolling restart:
    Batch Size
    Number of roles to include in a batch. Cloudera Manager restarts the worker roles rack-by-rack, in alphabetical order, and within each rack, hosts are restarted in alphabetical order. If you use the default replication factor of 3, Hadoop tries to keep the replicas on at least 2 different racks. So if you have multiple racks, you can use a higher batch size than the default 1. However, using a batch size that is too high means that fewer worker roles are active at any time during the upgrade, which can cause temporary performance degradation. If you are using a single rack, restart one worker node at a time to ensure data availability during upgrade.
    Advanced Options > Sleep between batches
    Amount of time Cloudera Manager waits before starting the next batch.
    Advanced Options > Failed threshold
    The number of batch failures that cause the entire rolling restart to fail. For example if you have a very large cluster, you can use this option to allow some failures when you know that the cluster is functional when some worker roles are down.
  10. Click Continue.

    The Upgrade Cluster Command screen displays the result of the commands run by the wizard as it shuts down services, activates the new parcel, upgrades services, deploys client configuration files, restarts services, and performs a rolling restart of the services that support it.

    If your cluster was previously installed or upgraded using packages, the wizard may indicate that some services cannot start because their parcels are not available. To download the required parcels:
    1. In another browser tab, open the Cloudera Manager Admin Console.
    2. Select Hosts > Parcels.
    3. Locate the row containing the missing parcel and click the button to Download, Distribute, and then Activate the parcel.
    4. Return to the upgrade wizard and click the Retry button.

      The Upgrade Wizard continues upgrading the cluster.

  11. Click Finish to return to the Home page.

Step 7: Recover from Failed Steps or Perform a Manual Upgrade

The actions performed by the upgrade wizard are listed in Performing Upgrade Wizard Actions Manually. If any of the steps in the Upgrade Cluster Command screen fail, complete the steps as described in that section before proceeding.

Step 8: Remove the Previous CDH Version Packages and Refresh Symlinks

[Not required for CDH maintenance release upgrades.]

Skip this step if your previous installation or upgrade used parcels.

If your previous installation of CDH was done using packages, remove those packages on all hosts where you installed the parcels and refresh the symlinks so that clients will run the new software versions.

  1. If your Hue service uses the embedded SQLite database, back up /var/lib/hue/desktop.db to a location that is not /var/lib/hue because this directory is removed when the packages are removed.
  2. Uninstall the CDH packages on each host:
    • Not including Impala and Search
      RHEL
      sudo yum remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
      SLES
      sudo zypper remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
      Ubuntu or Debian
      sudo apt-get purge bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
    • Including Impala and Search
      RHEL
      sudo yum remove 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
      SLES
      sudo zypper remove 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
      Ubuntu or Debian
      sudo apt-get purge 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
  3. Restart all the Cloudera Manager Agents to force an update of the symlinks to point to the newly installed components on each host:
    sudo service cloudera-scm-agent restart
  4. If your Hue service uses the embedded SQLite database, restore the database you backed up:
    1. Stop the Hue service.
    2. Copy the backup from the temporary location to the newly created Hue database directory, /var/lib/hue.
    3. Start the Hue service.

Step 9: Finalize HDFS Rolling Upgrade

[Not required for CDH maintenance release upgrades.]

The steps in this section are only required for the following upgrades:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher

To determine if you can finalize, run important workloads and ensure that they are successful. Once you have finalized the upgrade, you cannot roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time.

  1. Go to the HDFS service.
  2. Select Actions > Finalize Rolling Upgrade and click Finalize Rolling Upgrade to confirm.

Step 10: Exit Maintenance Mode

If you entered maintenance mode during this upgrade, exit maintenance mode.

Step 11: Clear Browser Cache (Hue only)

If you have enabled the Hue service in your upgraded cluster, users may need to clear the cache in their Web browsers before accessing Hue.