Upgrading Cloudera Manager 5 Using Tarballs

Before beginning a Cloudera Manager upgrade, do the following:

1. Review the system requirements for the new versions you are upgrading to.

   See CDH 5 and Cloudera Manager 5 Requirements and Supported Versions.

2. If you are upgrading from Cloudera Manager 5.4.x to Cloudera Manager 5.5 or higher, perform the following steps required for the associated Cloudera Navigator upgrade:
   1. Stop the Navigator Metadata Server role.
   2. Back up the Navigator Metadata Server storage directory.
   3. Make sure that the Navigator Metadata Server has sufficient memory to complete the upgrade.
   4. If you are using an Oracle database, in SQL*Plus, ensure that the following additional privileges are set:

      GRANT EXECUTE ON sys.dbms_crypto TO nav;
      GRANT CREATE VIEW TO nav;

      where nav is the user of the Navigator Audit Server database.
   For further information, see Upgrading the Cloudera Navigator Data Management Component.
3. Note the following:
   • Cloudera Management Service TLS/SSL configuration

     If you have enabled TLS security for the Cloudera Manager Admin Console, as of Cloudera Manager 5.1, Cloudera Management Service roles try to communicate with Cloudera Manager using TLS, and fail to start until TLS/SSL properties have been configured.

   • Navigator

     If you have enabled auditing with Cloudera Navigator, during the upgrade to Cloudera Manager 5, auditing is suspended and is only restarted when you restart the roles of audited services. You will be instructed to stop some services in a later step.

4. If you have previously installed Kafka 1.2, and are upgrading from Cloudera Manager 5.4 or lower, remove the Kafka CSD:
   1. Determine the location of the CSD directory:
      1. Select Administration > Settings.
      2. Click the Custom Service Descriptors category.
      3. Retrieve the directory from the Local Descriptor Repository Path property.
   2. Delete the Kafka CSD from the directory.
5. Review package (RPM) dependencies. A Cloudera Manager upgrade may introduce new package dependencies. If your organization has restrictions or requires prior approval for installation of packages, see the list of Package Dependencies before upgrading Cloudera Manager.

1. If your cluster is running the embedded PostgreSQL database, stop all services that are using the embedded database. These may include:
   • Hive service and all services such as Impala and Hue that use the Hive metastore
   • Oozie
   • Sentry
2. Stop Cloudera Manager Server, Database, and Agent:
   1. Use the Cloudera Manager Admin Console to stop any running commands. These include commands a user runs and commands Cloudera Manager automatically triggers in response to a state change or a schedule. You can either wait for commands to complete or abort any running commands. For more information on viewing and aborting running commands, see Viewing Running and Recent Commands.
   2. On the host running the Cloudera Manager Server, stop the Cloudera Manager Server:

      sudo service cloudera-scm-server stop

   3. On the host where the database runs, (usually the Cloudera Manager Server host) stop the database:

      sudo service cloudera-scm-server-db stop

   4. If the Cloudera Manager host is also running the Cloudera Manager Agent, stop the Cloudera Manager Agent:

      sudo service cloudera-scm-agent stop

3. If you are using JDK 1.6, upgrade the JDK on Cloudera Manager Server JDK 1.7 or 1.8. After you install JDK 7, Cloudera Manager can install JDK 7 on all of the cluster hosts. To upgrade to JDK 1.8, you must manually install the JDK on all cluster hosts. Follow the steps in Java Development Kit Installation, and then return here to continue the upgrade.
4. Download tarballs from the locations listed in Cloudera Manager Version and Download Information.
5. Copy the tarballs and unpack them on all hosts on which you intend to install Cloudera Manager Server and Cloudera Manager Agents, in a directory of your choosing. If necessary, create a new directory to accommodate the files you extract from the tarball. For instance, if /opt/cloudera-manager does not exist, create it using a command similar to:

   $ sudo mkdir /opt/cloudera-manager

6. Extract the contents of the tarball to this directory. For example, to copy a tar file to your home directory and extract the contents of all tar files to the /opt/ directory, use a command similar to the following:

   $ sudo tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager

   The files are extracted to a subdirectory named according to the Cloudera Manager version being extracted. For example, files could be extracted to /opt/cloudera-manager/cm-5.0/. This full path is needed later and is referred to as tarball_root directory.

7. On every Cloudera Manager Agent host, configure the Cloudera Manager Agent to point to the Cloudera Manager Server by setting the following properties in the tarball_root/etc/cloudera-scm-agent/config.ini configuration file:

   Property	Description
   server_host	Name of the host where Cloudera Manager Server is running.
   server_port	Port on the host where Cloudera Manager Server is running.

8. By default, a tarball installation has a var subdirectory where state is stored. In a non-tarball installation, state is stored in /var. Cloudera recommends that you configure the tarball installation to use an external directory as the /var equivalent (/var or any other directory outside the tarball) so that when you upgrade Cloudera Manager, the new tarball installation can access this state. Configure the installation to use an external directory for storing state by editing tarball_root/etc/default/cloudera-scm-agent and setting the CMF_VAR variable to the location of the /var equivalent. If you do not reuse the state directory between different tarball installations, duplicate Cloudera Manager Agent entries can occur in the Cloudera Manager database.
9. Start Cloudera Manager Server.
   The way in which you start the Cloudera Manager Server varies depending on which account you want the Server to run under:

   • As root:

     sudo tarball_root/etc/init.d/cloudera-scm-server start 

   • As another user. If you run as another user, ensure the user you created for Cloudera Manager owns the location to which you extracted the tarball including the newly created database files. If you followed the earlier examples and created the directory /opt/cloudera-manager and the user cloudera-scm, you could use the following command to change ownership of the directory:

     sudo chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

     Once you have established ownership of directory locations, you can start Cloudera Manager Server using the user account you chose. For example, you might run the Cloudera Manager Server as cloudera-service. In this case, you have the following options:

     • Run the following command:

       $ sudo -u cloudera-service tarball_root/etc/init.d/cloudera-scm-server start 

     • Edit the configuration files so the script internally changes the user. Then run the script as root:
       1. Remove the following line from tarball_root/etc/default/cloudera-scm-server:

          export CMF_SUDO_CMD=" "

       2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-server to the user you want the server to run as. For example, to run as cloudera-service, change the user and group as follows:

          USER=cloudera-service
          GROUP=cloudera-service

       3. Run the server script as root:

          $ sudo tarball_root/etc/init.d/cloudera-scm-server start 

   • To start the Cloudera Manager Server automatically after a reboot:
     1. Run the following commands on the Cloudera Manager Server host:
        • RHEL-compatible and SLES

          $ cp tarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
          $ chkconfig cloudera-scm-server on

        • Debian/Ubuntu

          $ cp tarball_root/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server
          $ update-rc.d cloudera-scm-server defaults

     2. On the Cloudera Manager Server host, open the /etc/init.d/cloudera-scm-server file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} to tarball_root/etc/default.
10. To stop the Cloudera Manager Agent, run this command on each Agent host:

    $ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed

    If you are running single user mode, stop Cloudera Manager Agent using the user account you chose. For example, if you are running the Cloudera Manager Agent as cloudera-scm, you have the following options:
    • Run the following command:

      $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed 

    • Edit the configuration files so the script internally changes the user, and then run the script as root:
      1. Remove the following line from tarball_root/etc/default/cloudera-scm-agent:

         export CMF_SUDO_CMD=" "

      2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:

         USER=cloudera-scm
         GROUP=cloudera-scm

      3. Run the Agent script as root:

         $ sudo tarball_root/etc/init.d/cloudera-scm-agent hard_stop_confirmed 

11. Start the Cloudera Manager Agent according to the account you want the Agent to run under:
    • To start the Cloudera Manager Agent, run this command on each Agent host:

      $ sudo tarball_root/etc/init.d/cloudera-scm-agent start

      When the Agent starts, it contacts the Cloudera Manager Server.
    • If you are running single user mode, start Cloudera Manager Agent using the user account you chose. For example, to run the Cloudera Manager Agent as cloudera-scm, you have the following options:
      • Run the following command:

        $ sudo -u cloudera-scm tarball_root/etc/init.d/cloudera-scm-agent start 

      • Edit the configuration files so the script internally changes the user, and then run the script as root:
        1. Remove the following line from tarball_root/etc/default/cloudera-scm-agent:

           export CMF_SUDO_CMD=" "

        2. Change the user and group in tarball_root/etc/init.d/cloudera-scm-agent to the user you want the Agent to run as. For example, to run as cloudera-scm, change the user and group as follows:

           USER=cloudera-scm
           GROUP=cloudera-scm

        3. Run the Agent script as root:

           $ sudo tarball_root/etc/init.d/cloudera-scm-agent start 

    • To start the Cloudera Manager Agents automatically after a reboot:
      1. Run the following commands on each Agent host:
         • RHEL-compatible and SLES

           $ cp tarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
           $ chkconfig cloudera-scm-agent on

         • Debian/Ubuntu

           $ cp tarball_root/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent
           $ update-rc.d cloudera-scm-agent defaults

      2. On each Agent, open the tarball_root/etc/init.d/cloudera-scm-agent file and change the value of CMF_DEFAULTS from ${CMF_DEFAULTS:-/etc/default} to tarball_root/etc/default.
12. Log in to the Cloudera Manager Admin Console.
13. Restart all services:
    1. From the Home > Status tab click next to the cluster name and select Restart.
    2. In the confirmation dialog box that displays, click Restart.
14. Upgrade any required Cloudera Navigator components:
    • Upgrading the Cloudera Navigator Data Management Component
    • Upgrading Cloudera Navigator Key Trustee Server
    • Upgrading Cloudera Navigator Key HSM
    • Upgrading Key Trustee KMS
    • Upgrading Cloudera Navigator Encrypt
15. (Optional) Upgrade CDH.

    See Step 10: (Optional) Upgrade CDH.