Upgrading to CDH 5.5 Using Parcels

Minimum Required Role: Cluster Administrator (also provided by Full Administrator)

You can upgrade your CDH 5 cluster to CDH 5.5 using parcels from within the Cloudera Manager Admin Console. Your current CDH 5 cluster can have been installed with either parcels or packages. The new version will use parcels.

The following procedure requires cluster downtime. If you use parcels, have a Cloudera Enterprise license, and have enabled HDFS high availability, you can perform a rolling upgrade that lets you avoid cluster downtime.

Before You Begin

  • Read the CDH 5 Release Notes.
  • Read the Cloudera Manager 5 Release Notes.
  • Ensure Java 1.7 or 1.8 is installed across the cluster. For installation instructions and recommendations, see Upgrading to Oracle JDK 1.7 in a Cloudera Manager Deployment or Upgrading to Oracle JDK 1.8, and make sure you have read Known Issues and Workarounds in Cloudera Manager 5 before you proceed with the upgrade.
  • Ensure that the Cloudera Manager minor version is equal to or greater than the CDH minor version. For example:
    Target CDH Version Minimum Cloudera Manager Version
    5.0.5 5.0.x
    5.1.4 5.1.x
    5.4.1 5.4.x
  • Date partition columns: as of Hive version 13, implemented in CDH 5.2, Hive validates the format of dates in partition columns, if they are stored as dates. A partition column with a date in invalid form can neither be used nor dropped once you upgrade to CDH 5.2 or higher. To avoid this problem, do one of the following:
    • Fix any invalid dates before you upgrade. Hive expects dates in partition columns to be in the form YYYY-MM-DD.
    • Store dates in partition columns as strings or integers.
    You can use the following SQL query to find any partition-column values stored as dates:
    SELECT "DBS"."NAME", "TBLS"."TBL_NAME", "PARTITION_KEY_VALS"."PART_KEY_VAL"
    FROM "PARTITION_KEY_VALS"
      INNER JOIN "PARTITIONS" ON "PARTITION_KEY_VALS"."PART_ID" = "PARTITIONS"."PART_ID"
      INNER JOIN "PARTITION_KEYS" ON "PARTITION_KEYS"."TBL_ID" = "PARTITIONS"."TBL_ID"
      INNER JOIN "TBLS" ON "TBLS"."TBL_ID" = "PARTITIONS"."TBL_ID"
      INNER JOIN "DBS" ON "DBS"."DB_ID" = "TBLS"."DB_ID"
        AND "PARTITION_KEYS"."INTEGER_IDX" ="PARTITION_KEY_VALS"."INTEGER_IDX"
        AND "PARTITION_KEYS"."PKEY_TYPE" = 'date';
  • Whenever upgrading Impala, whether in CDH or a standalone parcel or package, check your SQL against the newest reserved words listed in incompatible changes. If upgrading across multiple versions or in case of any problems, check against the full list of Impala keywords.
  • Run the Host Inspector and fix every issue.
  • If using security, run the Security Inspector.
  • Run hdfs fsck / and hdfs dfsadmin -report and fix every issue.
  • Run hbase hbck.
  • Review the upgrade procedure and reserve a maintenance window with enough time allotted to perform all steps. For production clusters, Cloudera recommends allocating up to a full day maintenance window to perform the upgrade, depending on the number of hosts, the amount of experience you have with Hadoop and Linux, and the particular hardware you are using.
  • To avoid lots of alerts during the upgrade process, you can enable maintenance mode on your cluster before you start the upgrade. This will stop email alerts and SNMP traps from being sent, but will not stop checks and configuration validations from being made. Be sure to exit maintenance mode when you have finished the upgrade in order to re-enable Cloudera Manager alerts.
  • Hue validates CA certificates and needs a truststore. To create one, follow the instructions in Hue as a TLS/SSL Client.

Stop Cluster Services

If upgrading from:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher
  1. On the Home > Status tab, click to the right of the cluster name and select Stop.
  2. Click Stop in the confirmation screen. The Command Details window shows the progress of stopping services.

    When All services successfully stopped appears, the task is complete and you can close the Command Details window.

Back up the HDFS Metadata on the NameNode

If upgrading from:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher
  1. Go to the HDFS service.
  2. Click the Configuration tab.
  3. In the Search field, search for "NameNode Data Directories" and note the value.
  4. On the active NameNode host, back up the directory listed in the NameNode Data Directories property. If more than one is listed, make a backup of one directory, since each directory is a complete copy. For example, if the NameNode data directory is /data/dfs/nn, do the following as root:
    # cd /data/dfs/nn
    # tar -cvf /root/nn_backup_data.tar .

    You should see output like this:

    ./
    ./current/
    ./current/fsimage
    ./current/fstime
    ./current/VERSION
    ./current/edits
    ./image/
    ./image/fsimage
    If there is a file with the extension lock in the NameNode data directory, the NameNode most likely is still running. Repeat the steps, starting by shutting down the NameNode role.

Back up Metastore Databases

Back up the Hive, Sentry, and Sqoop metastore databases.
  1. For each affected service:
    1. If not already stopped, stop the service.
    2. Back up the database. See Backing Up Databases.

Run the Upgrade Wizard

  1. Log into the Cloudera Manager Admin console.
  2. From the Home > Status tab, click next to the cluster name and select Upgrade Cluster. The Upgrade Wizard starts.
  3. If the option to pick between packages and parcels displays, click the Use Parcels radio button.
  4. In the Choose CDH Version (Parcels) field, select the CDH version. If there are no qualifying parcels, click the Modify the Remote Parcel Repository URLs link to go to the Parcel Configuration Settings page where you can add the locations of parcel repositories. Click Continue.
  5. Read the notices for steps you must complete before upgrading, click the Yes, I ... checkboxes after completing the steps, and click Continue.
  6. Cloudera Manager checks that hosts have the correct software installed. Click Continue.
  7. The selected parcels are downloaded and distributed. Click Continue.
  8. The Host Inspector runs and displays the CDH version on the hosts. Click Continue.
  9. Choose the type of upgrade and restart:
    • Cloudera Manager upgrade - Cloudera Manager performs all service upgrades and restarts the cluster.
      1. Click Continue. The Command Progress screen displays the result of the commands run by the wizard as it shuts down all services, activates the new parcel, upgrades services as necessary, deploys client configuration files, and restarts services. If any of the steps fails or you click the Abort button the Retry button at the top right is enabled.

        You can click Retry to retry the step and continue the wizard or click the Cloudera Manager logo to return to the Home > Status tab and manually perform the failed step and all following steps.
      2. Click Continue. The wizard reports the result of the upgrade.
    • Manual upgrade - Select the Let me upgrade the cluster checkbox. Cloudera Manager configures the cluster to the specified CDH version but performs no upgrades or service restarts. Manually doing the upgrade is difficult and is for advanced users only.
      1. Click Continue. Cloudera Manager displays links to documentation describing the required upgrade steps.
  10. Click Finish to return to the Home page.

Perform Manual Upgrade or Recover from Failed Steps

The actions performed by the upgrade wizard are listed in Upgrade Wizard Actions. If you chose manual upgrade or any of the steps in the Command Progress screen fails, complete the steps as described in that section before proceeding.

Remove the Previous CDH Version Packages and Refresh Symlinks

If your previous installation of CDH was done using packages, remove those packages on all hosts on which you installed the parcels and refresh the symlinks so that clients will run the new software versions. Skip this step if your previous installation was using parcels.

  1. If your Hue service uses the embedded SQLite DB, back up /var/lib/hue/desktop.db to a location that is not /var/lib/hue as this directory is removed when the packages are removed.
  2. Uninstall the CDH packages on each host:
    • Not including Impala and Search
      Operating System Command
      RHEL
      $ sudo yum remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
      SLES
      $ sudo zypper remove bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
      Ubuntu or Debian
      $ sudo apt-get purge bigtop-utils bigtop-jsvc bigtop-tomcat hue-common sqoop2-client
    • Including Impala and Search
      Operating System Command
      RHEL
      $ sudo yum remove 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
      SLES
      $ sudo zypper remove 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
      Ubuntu or Debian
      $ sudo apt-get purge 'bigtop-*' hue-common impala-shell solr-server sqoop2-client hbase-solr-doc avro-libs crunch-doc avro-doc solr-doc
  3. Restart all the Cloudera Manager Agents to force an update of the symlinks to point to the newly installed components on each host:
    $ sudo service cloudera-scm-agent restart
  4. If your Hue service uses the embedded SQLite DB, restore the DB you backed up:
    1. Stop the Hue service.
    2. Copy the backup from the temporary location to the newly created Hue database directory, /var/lib/hue.
    3. Start the Hue service.

Finalize the HDFS Metadata Upgrade

If upgrading from:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher
Finalize the HDFS metadata upgrade. To determine when finalization is warranted, run important workloads and ensure they are successful. Once you have finalized the upgrade, it is not possible to roll back to a previous version of HDFS without using backups. Verifying that you are ready to finalize the upgrade can take a long time. Make sure you have enough free disk space, keeping in mind the following:
  • Deleting files does not free up disk space.
  • Using the balancer causes all moved replicas to be duplicated.
  • All on-disk data representing the NameNodes metadata is retained, which could more than double the amount of space required on the NameNode and JournalNode disks.
To finalize the metadata upgrade:
  1. Go to the HDFS service.
  2. Click the Instances tab.
  3. Click the NameNode instance.
  4. Select Actions > Finalize Metadata Upgrade and click Finalize Metadata Upgrade to confirm.

Upgrade Wizard Actions

Do the steps in this section only if you chose a manual upgrade or the upgrade wizard reports a failure and you choose not to retry.

Upgrade HDFS Metadata

If upgrading from:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.2 or 5.3 to 5.4 or higher
  1. Start the ZooKeeper service.
  2. Go to the HDFS service.
  3. Select Actions > Upgrade HDFS Metadata and click Upgrade HDFS Metadata to confirm.

Upgrade the Hive Metastore Database

Required for the following upgrades:
  • CDH 5.0 or 5.1 to 5.2 or higher
  • CDH 5.3 to 5.4 or higher
  1. Go to the Hive service.
  2. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Upgrade Hive Metastore Database Schema and click Upgrade Hive Metastore Database Schema to confirm.
  4. If you have multiple instances of Hive, perform the upgrade on each metastore database.

Upgrade the Oozie ShareLib

  1. Go to the Oozie service.
  2. Select Actions > Start and click Start to confirm.
  3. Select Actions > Install Oozie ShareLib and click Install Oozie ShareLib to confirm.

Upgrade Sqoop

  1. Go to the Sqoop service.
  2. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Upgrade Sqoop and click Upgrade Sqoop to confirm.

Upgrade the Sentry Database

Required for the following upgrades:
  • CDH 5.1 to 5.2 or higher
  • CDH 5.2 to 5.3 or higher
  • CDH 5.4 to 5.5 or higher
  1. Go to the Sentry service.
  2. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Upgrade Sentry Database Tables and click Upgrade Sentry Database Tables to confirm.

Upgrade Spark

  1. Go to the Spark service.
  2. Select Actions > Stop and click Stop to confirm.
  3. Select Actions > Install Spark JAR and click Install Spark JAR to confirm.
  4. Select Actions > Create Spark History Log Dir and click Create Spark History Log Dir to confirm.

Start Cluster Services

  1. On the Home > Status tab, click to the right of the cluster name and select Start.
  2. Click Start that appears in the next screen to confirm. The Command Details window shows the progress of starting services.

    When All services successfully started appears, the task is complete and you can close the Command Details window.

Deploy Client Configuration Files

  1. On the Home page, click to the right of the cluster name and select Deploy Client Configuration.
  2. Click the Deploy Client Configuration button in the confirmation pop-up that appears.