Performing a Rolling Upgrade on a Cluster
- Cloudera Express - the feature is not available.
- Cloudera Enterprise Data Hub Edition Trial - the feature will not be available after you end the trial or the trial license expires.
Cloudera Manager's rolling upgrade feature takes advantage of parcels and the HDFS High Availability configuration to enable you to upgrade your cluster software and restart the upgraded services, without taking the entire cluster down. You must have HDFS High Availability enabled to perform a rolling upgrade.
You can perform a rolling upgrade between minor versions of CDH 4, or between minor versions of CDH 5, except Beta versions, as they become available. It is not possible to perform a rolling upgrade from CDH 4 to CDH 5 because of incompatibilities between the two major versions. Instead, follow the instructions for a full upgrade at Upgrading from CDH 4 to CDH 5 Parcels.
- Download, distribute, and activate the parcel for the new software you want to install.
- Perform a rolling restart to restart the services in your cluster. You can do a rolling restart of individual services, or if you have High Availability enabled, you can perform a restart of the entire cluster. Cloudera Manager will manually fail over your NameNode at the appropriate point in the process so that your cluster will not be without a functional NameNode.
The steps to perform a rolling upgrade of a cluster are as follows.
Step 1. Ensure High Availability is Enabled
To enable High Availability see HDFS High Availability for instructions. You do not need to enable Automatic Failover for rolling restart to work, though you can enable it if you wish. Automatic Failover does not affect the rolling restart operation. If you have JobTracker High Availability configured, Cloudera Manager will fail over the JobTracker during the rolling restart, but this is not a requirement for performing a rolling upgrade.
Step 2. Download, Distribute, and Activate the Parcels with the New Software
- From the Hosts page, click the Parcels
tab, and download, distribute and activate the parcel(s) you want to upgrade to.
For detailed instructions see Parcels. When the parcel has been activated,
it is not yet running, but is staged to become the running version the next time
the service is restarted. Note
:If you are doing a major version upgrade (that is, from CDH 4 to CDH 5 using parcels), after the distribution phase the button will be labeled Upgrade rather than Activate. In this case, follow the instructions at Upgrading from CDH 4 to CDH 5 Parcels.
- You are asked if you want to restart the cluster. Do not restart the cluster at this time.
- Click Close. There are several additional steps you must perform prior to restarting your services.
Step 3. Upgrade the Hive Metastore
- (Strongly recommended) Make a backup copy of your Hive metastore database.
- Follow the instructions at Step 4. "Upgrading Your Metastore" in the Hive Installation section in CDH 4 Installation to run the metastore upgrade script.
- The upgrade script is located at /opt/cloudera/parcels/<parcel_name>/lib/hive/scripts/metastore/upgrade/<database>. <parcel_name> should be the name of the parcel to which you have upgraded. <database> is the type of database you are running (that is, MySQL, PostgreSQL, and so on) For example, if you are installing a CDH 4.2.0 parcel using the default location for the local repository, and using the default database (PostgreSQL) the script will be at: /opt/cloudera/parcels/CDH-4.2.0-1.cdh4.2.0.p0.10-e16.parcel/lib/hive/scripts/metastore/upgrade/postgres
- You must cd to the directory the scripts are in.
- Execute the script in the appropriate DB command shell.
:You must know the password for the database; if you installed Cloudera Manager using the default (embedded PostgreSQL) database, the password was displayed on the Database Setup page during the Cloudera Manager installation wizard. If you do not know the password for your Hive metastore database, you can find it as follows:
- cat /etc/cloudera-scm-server/db.properties This shows you Cloudera Manager's internal database credentials.
- Run the following command:
psql -p 7432 -U scm scm -c "select s.display_name as hive_service_name, s.name as hive_internal_name, c.value as metastore_password from configs c, services s where attr='hive_metastore_database_password' and c.service_id = s.service_id"
- Use the password shown in the com.cloudera.cmf.db.password property.
- This script will output the passwords for the hive service metastore as follows:
hive_service_name | hive_internal_name | metastore_password -------------------+--------------------+-------------------- hive1 | hive1 | lF3Cv2zsvI (1 row)
- For the embedded PostgreSQL database, the database name and user name are both hive.
- If you have multiple instances of Hive, run the upgrade script on each metastore database.
Step 4. (If Upgrading to CDH 4.2) Upgrade the Oozie Sharelib
- In the Cloudera Manager Admin Console, go to the Oozie service. The service should already be stopped.
- From the Actions button choose Install Oozie Sharelib. The commands to perform this function are run.
Step 5. Restart the Cluster
- On the Home page, click to the right of the cluster name and select Rolling Restart to proceed with a Rolling Restart. Rolling restart is available only if High Availability is enabled. Click Restart to perform a normal restart. Services that do not support Rolling Restart will undergo a normal restart, and will not be available during the restart process.
- For a rolling restart, a pop-up allows you to chose which services you
want to restart, and presents caveats to be aware of for those services that can
undergo a rolling restart.Note
:If you have just upgraded your Cloudera Manager deployment to 4.6, and are now doing a rolling upgrade of your cluster, you need to ensure that MapReduce is restarted before the rest of your services, or the restart may fail. This is necessary to ensure the MapReduce configuration changes are propagated.
Further, if you are upgrading from CDH 4.1 with Impala to CDH 4.2 or 4.3, you must restart MapReduce before Impala restarts (by default Impala is restarted before MapReduce).
The workaround is to perform a restart of MapReduce alone as the first step, then perform a cluster restart of the remaining services.
- Click Confirm to start the rolling restart.
Step 6. Remove the Previous CDH Version Packages
- Uninstall the CDH packages. On each
Operating System Command RHEL $ sudo yum remove bigtop-jsvc bigtop-utils bigtop-tomcat hue-common sqoop2-client hbase-solr-doc solr-doc SLES $ sudo zypper remove bigtop-jsvc bigtop-utils bigtop-tomcat hue-common sqoop2-client hbase-solr-doc solr-doc Ubuntu or Debian $ sudo apt-get purge bigtop-jsvc bigtop-utils bigtop-tomcat hue-common sqoop2-client hbase-solr-doc solr-doc
- Restart all the Cloudera Manager Agents to force an
update of the installed binaries reported by the Agent. On each host:
$ sudo service cloudera-scm-agent restart
- Run the Host Inspector to verify that the packages have
- Click Hosts tab and then click the Host Inspector button.
- When the command completes, click Show Inspector Results.