Known Issues and Limitations in Cloudera Manager 6.1.0

Cloudera Manager 6.x issue with the service role Resume

If a selected service role on a node is restarted and fails, and the customer clicks the "Resume" button in Cloudera Manager, the service role on all of the nodes will be restarted concurrently.

Products affected: Cloudera Manager

Releases affected:
  • Cloudera Manager 5.5 and later
  • Cloudera Manager 6.0 until 6.3.3
  • Cloudera Manager 7.1.x

Users affected: Users with admin role in Cloudera Manager can impact end users of the service.

Impact:In production clusters this can result in a cluster-wide service outage; Already observed for the YARN service and the HDFS service in a few clusters.

Severity: High

Action required:
  • A workaround exists where instead of performing a restart we recommend performing a stop/start of the services.
  • Issue is fixed in CM-6.3.4, CM-7.2.1 and above.

Knowledge article: For the latest update on this issue see the corresponding Knowledge article: Cloudera Customer Advisory: Cloudera Manager 6.x issue with service role Resume

Cloudera Manager installation fails on MariaDB 10.2.8 and later

When installing Cloudera Manager using MariaDB 10.2.8 or later, the Cloudera Manager web server doesn't come up and the install process ends with a failed status. The cloudera-scm-server.log includes the following SQL error:

2019-08-28 04:37:10,171 FATAL main:org.hsqldb.cmdline.SqlFile: SQL Error at 'UTF-8' line 57:
  "alter table ROLE_CONFIG_GROUPS
  drop column REVISION_ID"
  Key column 'REVISION_ID' doesn't exist in table
2019-08-28 04:37:10,171 FATAL main:org.hsqldb.cmdline.SqlFile: Rolling back SQL transaction.
2019-08-28 04:37:10,172 ERROR main:com.cloudera.enterprise.dbutil.SqlFileRunner: Exception while executing ddl scripts.
  com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Key column 'REVISION_ID' doesn't exist in table

Note that MariaDB 10.2.8 is provided by default in some operating systems, including SLES 12 SP4.

Workaround: Replace the default MariaDB 10.2.x version with MariaDB 10.2.7.

Affected Versions: MariaDB 10.2.8 and later

Cloudera Issue: OPSAPS-52340

HBase Failure on Upgrade to Cloudera Manager 6.1

After upgrading from Cloudera Manager 5.14 (and ….??) to Cloudera Manager 6.1 and restarting the Hive service, running an HBase query cause the following error in beeline:
Error: java.io.IOException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Workaround:
  1. In the Cloudera Manager Admin console, go to the Hive service.
  2. Select the Configuration tab.
  3. Search for the following property: HiveServer2 Environment Advanced Configuration Snippet (Safety Valve).
  4. Add the following property and value:
    HADOOP_CLASSPATH=/etc/hbase/conf
  5. Click Save Changes.
  6. Restart HiveServer2:
    1. Go to the Hive Service.
    2. Click the Instances tab.
    3. Click the HiveServer2 link in the table.
    4. Click Actions > Restart this HiveServer2.

Cloudera Issue: OPSAPS 49330

BDR - Hive restore failing during import

When the table filter used during hive cloud restore is different from the table filter used to create the hive cloud backup, the import step fails with the table not found error. Currently it impacts only the cloud restore scenario.

Products affected: Cloudera Manager

Releases affected:
  • Cloudera Manager 5.15, 5.16
  • Cloudera Manager 6.1.x
  • Cloudera Manager 6.2.x
  • Cloudera Manager 6.3.x

Users affected: BDR, Hive cloud restore, where restore uses a subset of tables from the exported tables

Impact:
  • Limited, the hive cloud restore all tables works properly.
  • The hive cloud restore from the hive cloud backup created prior to Cloudera Manager 5.15 would work without any problem.
  • No other BDR functionality is affected.
Immediate action required:
  • Workaround: Not available. Importing specific tables would fail. Impoting ALL tables would continue to work properly.
  • Upgrade: Upgrade to a Cloudera Manager version containing the fix.

Addressed in release/refresh/patch: Cloudera Manager 7.0 and higher versions

Cloudera Manager Server and Agents encounter TLS issues when the Agent config.ini file is misconfigured

The Cloudera Manager Server and Agents incorrectly thinks that TLS is enabled for the Agent if you set use_tls=1 in the Agent config.ini but do not provide a key, certificate, or truststore. This causes several issues, including log messages that indicate TLS is enabled when it is not, missing files from diagnostic bundles, and unavailable logs.

Workaround: Ensure that you specify a key, certificate, and truststore when you configure use_tls=1 in the Agent config.ini file.

Affected Versions: Cloudera Manager 6.1.x

Cloudera Issue: OPSAPS-48898, OPSAPS-48897

The restart Cloudera Manager Agent command does not restart the Agent Listener

When you restart the Cloudera Manager Agent, the Agent Listener (the status_server process) does not restart. This can cause issues if you make changes to TLS for Agents after you have installed Cloudera Manager since the Agent Listener needs to be restarted for TLS changes to take effect.

Workaround: Run the following command on every Agent host with supervisorctl:
/opt/cloudera/cm-agent/bin/supervisorctl -c /var/run/cloudera-scm-agent/supervisor/supervisord.conf restart status_server

This command requires root access to the host. supervisorctl is owned by the cloudera-scm user, but supervisord.conf is owned by root.

Affected Versions: Cloudera Manager 6.0.x, 6.1.x

Cloudera Issue: OPSAPS-48886

TSB-359 Backup and Disaster Recovery (BDR) HDFS and Hive Replications will fail on clusters running Cloudera Manager 6.1.0

Backup and Disaster Recovery (BDR) HDFS and Hive Replications will fail when replicating from secured (Kerberized) source clusters to destination clusters that have been upgraded to Cloudera Manager 6.1.0.

This also affects new installations of Cloudera Manager 6.1.0 on the destination cluster if an admin restarts the Cloudera Manager service.

Products affected: Cloudera Manager Backup and Disaster Recovery in a secure (Kerberized) environment

Releases affected: Cloudera Manager 6.1.0 (when used as the destination cluster of HDFS and/or Hive replication)

Users affected: Customers using HDFS or Hive Replication

Severity (Low/Medium/High): High

Root Cause and Impact:

In HDFS and Hive Replication, Cloudera Manager first runs a process on the destination cluster to verify if the replication is possible. Due to a bug, the source cluster is treated as an insecure (non-kerberized) cluster. As a result, replication fails.

You will see the exception javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)] in the process stderr logs.

Immediate action required: If you use BDR, do not upgrade a destination cluster to Cloudera Manager 6.1.0. Upgrade to Cloudera Manager 6.1.1 or higher when it becomes available.

If you have already upgraded your destination cluster to Cloudera Manager to 6.1.0, use the following workaround:

  1. For an existing HDFS or Hive replication schedule, select Actions > Edit Configuration.
  2. Save the schedule.

Please note that you will need to edit only one schedule even if you have multiple schedules.

Note: This workaround is not persistent. That is, if you restart the Cloudera Manager service, you must repeat the above workaround.

Cloudera Issue: OPSAPS-48865

Fixed in Cloudera Manager 6.1.1

BDR Job Mapper shows the following warning: AuthenticationException: GSSException: No valid credentials provided

A BDR Job Mapper might succeed with the following stack trace in the log message:
2018-12-05 13:57:03,475 WARN [main] org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider: KMS provider at [https://src-3.example.com:16000/kms/v1/] threw an IOException:
java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:492)
at org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:793) 

The warning appears if you try to replicate from an encrypted source cluster that has multiple KMS instances.

Workaround: You can safely ignore this message because the client succeeds upon fail over.

Affected Versions: CDH 6.1.0

Cloudera Issue: CDH-76053

Diff Format changed for Configuration History page

Color formatting for the diff display omits the red and green colors that indicate what was removed and added.

Affected Versions: Cloudera Manager 6.1.0

Cloudera Issue:OPSAPS-48544

Backup and Disaster Recovery (BDR) performance regression after upgrading to CDH 6.0.0

Hive replication with BDR experiences a performance regression when comparing CDH 6.0.0 and CDH 5.14.4. The slowdown occurs during the import step. For example, the performance regression may only be 10% for 4 million partitions. As the number of partitions goes down though, the performance impact becomes more visible. For example, 100,000 partitions may experience a 20% performance regression.

Affected Versions: Cloudera Manager 6.0.0, 6.01, 6.1.0; CDH 6.0.0, 6.0.1, 6.1.0

Cloudera Issue: OPSAPS-47520

Cloudera Manager allows more than a single space in YARN Admin ACLs

When adding a YARN Admin ACL in Cloudera Manager, you are allowed to enter multiple spaces in the entry. The space is the separator between the user and group lists, and only a single space should be allowed in the entry. All entries that appear after a second single space in a YARN Admin ACL will be ignored.

Affected Versions: Cloudera Manager 6.0.0, 6.0.1, 6.1.0

Cloudera Issue: OPSAPS-47688

Integer data types map to Float in Swagger API client

Integer data types show up as floating point numbers when using the Cloudera Manager API Python client.

Affected Versions: Cloudera Manager 6.0.0, 6.0.1, 6.1.0

Cloudera Issue: OPSAPS-45689

Package Installation of CDH Fails

When you install CDH with packages from a custom repository, ensure that the version of CDH you select for Select the version of CDH matches the version of CDH for the custom repository. Selecting the CDH version and specifying a custom repository are done during the Select Repository stage of installation.

If the versions do not match, installation fails.

Affected Versions: Cloudera Manager 6.x

Fixed Versions: N/A

Apache Issue: N/A

Cloudera Issue: OPSAPS-45703

Backup and Disaster Recovery replication to/from Cloudera Manager 6 clusters require Cloudera Manager 5.14.0 or higher

You can only use BDR to replicate to/from clusters managed by Cloudera Manager 6 with Cloudera Manager 5.14.0 or higher.

Affected versions: Cloudera Manager 6.x

Cloudera Issue: OPSAPS-42207