Database Considerations for Cloudera Manager Upgrades
Cloudera Manager uses databases to store information about system configurations and tasks. Before upgrading, complete the pre-upgrade database tasks that apply in your environment.
When upgrading from a previous CDH version, Cloudera Manager automatically creates new Hive service(s) to capture the previous implicit Hive dependency from Hue and Impala. Your previous services will continue to function without impact.
Note that if Hue was using a Hive metastore of type Derby, then the newly created Hive service will also use Derby. But since Derby does not allow concurrent connections, although Hue will continue to work, the new Hive Metastore Server will fail to run. The failure is harmless (because nothing uses this new Hive Metastore Server at this point) and intentional, to preserve the set of cluster functionality as it was before upgrade. Cloudera discourages the use of a Derby metastore due to its limitations. You should consider switching to a different supported database type (PostgreSQL, MySQL, Oracle).
After you have completed these steps, the upgrade processes automatically complete any additional updates to database schemas and service data stored. You do not need to complete any data migration.
Back up Databases
Before beginning the upgrade process, shut down the services that are using databases. This includes Cloudera Manager Server, Activity Monitor, Service Monitor, Report Manager, Host Monitor, Cloudera Navigator, and Hive Metastore. Cloudera recommends that you then back up all databases. This is especially important if you are upgrading from 3.7.x and there is any possibility you may want to revert to using 3.7.x. For information on backing up databases:
- For MySQL, see Backing up the MySQL Database.
- For PostgreSQL, see Backing up the PostgreSQL Database.
- For Oracle, work with your database administrator to ensure databases are properly backed up.
If any additional database will be required as a result of the upgrade, complete any required preparatory work to install and configure those databases. For example, Cloudera Manager 4.0 offers a Host Monitoring service that requires a database. To enable the Host Monitoring service, you must install a database. The upgrade instructions assume all required databases have been prepared. For more information on using databases, see Installing and Configuring Databases.
Modify Databases to Support UTF-8
Cloudera Manager 4.0 adds support for UTF-8 character sets. Update any existing databases in your environment that are not configured to support UTF-8.
Modifying MySQL to Support UTF-8
To modify a MySQL database to support UTF-8, the default character set must be changed and then you must restart the mysql service. Use the following commands to complete these tasks:
mysql> alter database default character set utf8; mysql> quit $ sudo service mysql restart
Modifying PostgreSQL to Support UTF-8
There is no single command available to modify an existing PostgreSQL database to support UTF-8. As a result, you must complete the following process:
- Use pg_dump to export the database to a file. This creates a backup of the database that you will import into a new, empty database that supports UTF-8.
- Drop the existing database. This deletes the existing database.
- Create a new database that supports Unicode encoding and that has the
same name as the old database. Use a command of the following form,
replacing the database name and user name with values that match your
CREATE DATABASE scm_database WITH OWNER scm_user ENCODING 'UTF8'
- Review the contents of the exported database for non-standard characters. If you find unexpected characters, modify these so the database backup file contains the expected data.
- Import the database backup to the newly created database.
Modifying Oracle to Support UTF-8
Work with your Oracle database administrator to ensure any Oracle databases support UTF-8.
Modify Databases to Support Appropriate Maximum Connections
Check existing databases configurations to ensure the proper maximum number of connections is supported. Update the maximum configuration values, as required.
Modifying the Maximum Number of MySQL Connections
Allow 100 maximum connections for each database and then add 50 extra connections. For example, for two databases set the maximum connections to 250. If you store seven databases on one host (the databases for Activity Monitor, Service Monitor, Report Manager, Host Monitor, Cloudera Manager Server, Cloudera Navigator, and Hive Metastore), set the maximum connections to 750.
Modifying the Maximum Number of PostgreSQL Connections
Update the max_connection parameter in the /etc/postgresql.conf file.
You may have to increase the system resources available to PostgreSQL, as described at http://www.postgresql.org/docs/9.1/static/kernel-resources.html.
Modifying the Maximum Number of Oracle Connections
Work with your Oracle database administrator to ensure appropriate values are applied for your Oracle database settings. You must determine the number of connections, transactions, and sessions to be allowed. Allow 100 maximum connections for each database and then add 50 extra connections. For example, for two databases set the maximum connections to 250. If you store seven databases on one host (the databases for Activity Monitor, Service Monitor, Report Manager, Host Monitor, Cloudera Manager Server, Cloudera Navigator, and Hive Metastore), set the maximum connections to 750.
From the maximum number of connections, you can determine the number of anticipated sessions using the following formula:
sessions = (1.1 * maximum_connections) + 5
For example, if a host has two databases, you anticipate 250 maximum connections. If you anticipate a maximum of 250 connections, plan for 280 sessions.
Once you know the number of sessions, you can determine the number of anticipated transactions using the following formula:
transactions = 1.1 * sessions
Continuing with the previous example, if you anticipate 280 sessions, you can plan for 308 transactions.
Work with your Oracle database administrator to apply these derived values to your system.
Using the sample values above, Oracle attributes would be set as follows:
alter system set processes=250; alter system set transactions=308; alter system set sessions=280;
After you have completed any required database preparatory tasks, continue to the upgrade path that is appropriate for your environment. Supported paths include: