Known Issues and Workarounds in Cloudera Director

The process of bootstrapping a deployment can hang indefinitely waiting for Cloudera Manager to start, even after Cloudera Manager is up and reachable.

Workaround: Restart Cloudera Director server. If this does not work, the deployment must be terminated.

When shrinking or repairing an instance that has been terminated outside of Cloudera Director, Cloudera Director may fail to decommission and delete the host from Cloudera Manager.

Workaround: Once Cloudera Director's shrink or repair operation has completed, the user can decommission and delete the host manually from the Cloudera Manager instance.

Cloudera Director automatically installs JDBC drivers on an instance for Cloudera Manager and the CDH clusters it provisions. However, when you use an existing MySQL database with Cloudera Manager, Cloudera Director does not install the JDBC driver, which can result in database connection failures.

sudo yum install -y mysql-connector-java
sudo yum install -y mysql-devel

On CentOS/RHEL 7 operating systems, the tuned service overwrites the swappiness settings that Cloudera Director configures on instances.

sudo systemctl disable tuned.service 
sudo systemctl stop tuned.service

nscd, a tool which caches common name service requests, is installed on Cloudera Director-managed instances, but is not enabled on CentOS and RHEL. This can reduce the performance of the bootstrapping process.

Workaround: Enable nscd by making a bootstrap script for all instances on the cluster with the contents shown below.

sudo yum install -y nscd
sudo systemctl enable nscd.service
sudo systemctl start nscd.service

sudo yum install -y nscd
sudo chkconfig nscd on
sudo service nscd start

If a deployment or cluster is terminated or updated at the same time that a refresh of instance metadata is running, on rare occasions the refresh will prevent the terminate or update operation from completing properly.

Workaround: Resubmit the terminate or update request when the refresh is done.

If duplicate services are specified for a cluster, for example two Hive services or two Impala services, Cloudera Director infinitely retries to create them during cluster creation.

Workaround: Cancel the cluster bootstrap, terminate the cluster, and recreate without duplicate services.

Cloudera Director requests that Cloudera Manager perform automatic configuration for a cluster prior to applying any custom configurations. Automatic configuration may sometimes create multiple groups of instances within Cloudera Manager for a single corresponding group requested by Cloudera Director. When this occurs, custom configurations for the instances will only be applied to instances in one of the Cloudera Manager groups.

Workaround: Check for extra instance groups in Cloudera Manager after cluster bootstrap and manually apply missing custom configurations.

Cloudera Director 2.3 does not support deployment and management of Kudu.

Workaround: None.

Standing up a cluster with a large number of EBS volumes might trigger rate limiting on EBS allocation requests. The effect can spread to other calls from Cloudera Director to AWS.

Workaround: No more than 10 EBS volumes should be attached at a time.

Cloudera Director sometimes fails to detect stale services properly when restarting a cluster.

Workaround: If the cluster includes stale services after a restart, restart the cluster again manually through Cloudera Manager.

External databases are not configured for Hue and Oozie in clusters created through the Cloudera Director web UI.

Workaround: External databases can be specified for Hue and Oozie when creating a cluster using the CLI or API.

Cloudera Navigator Key Trustee Server cannot be one of the services deployed by Cloudera Director.

Workaround: Contact Cloudera Support if you need to add Cloudera Navigator Key Trustee Server.

Cloudera Director's web UI does not allow creation of clusters with instance groups that should not have CDH roles deployed on them.

Workaround: Use the CLI or API to create clusters with instance groups that should not have CDH roles deployed on them. At least one instance group in the cluster must specify roles. Otherwise roles will be automatically allocated across all instances.

Cloudera Director's web UI does not allow modification of clusters with instance groups that should not have CDH roles deployed on them, even if they were created using the API.

Workaround: Use the API to modify clusters with instance groups that should not have CDH roles deployed on them.

Cloudera Director is unable to resize XFS partitions, which makes creating an instance that uses the XFS filesystem fail during bootstrap.

Workaround: Use an image with an ext filesystem such as ext2, ext3, or ext4.

Cloudera Director cannot set up external databases for Sqoop2.

Workaround: Set up databases for this service as described in Cloudera Manager and Managed Service Databases.

Clusters deployed in Cloudera Manager version 5.4 and lower might not have metrics displayed in the web UI if these clusters share the same name as previously deleted clusters.

Workaround: Use Cloudera Manager 5.5 and higher.

If the Cloudera Manager username and password are changed directly in Cloudera Manager, Cloudera Director can no longer add new instances or authenticate with Cloudera Manager. Username and password changes must be implemented in Cloudera Director as well. For more information on keeping Cloudera Director and Cloudera Manager in sync, see CDH Cluster Management Tasks.

Workaround: Use the Cloudera Director web UI to update the Cloudera Manager username and password.

Modifying a cluster in Cloudera Manager after it is bootstrapped does not cause the cluster state to be synchronized with Cloudera Director. Services that have been added or removed in Cloudera Manager do not show up in Cloudera Director when growing the cluster. For more information on keeping Cloudera Director and Cloudera Manager in sync, see CDH Cluster Management Tasks.

Workaround: None.

Cloudera Director Server uses the AWS credentials from a configured Environment, as defined in a client configuration file or through the Cloudera Director web UI. If the Environment is not configured with credentials in Cloudera Director, the Cloudera Director server instead uses the AWS credentials that are configured on the instance on which the Cloudera Director server is running. When those credentials differ from the intended ones, EC2 instances may be allocated under unexpected accounts. Ensure that the Cloudera Director server instance is not configured with AWS credentials.

Severity: Medium

Workaround: Ensure that the Cloudera Director Environment has correct values for the keys. Alternatively, use IAM profiles for the Cloudera Director server instance.

Cloudera Director cannot resize the root partition on Centos 6.5 HVM AMIs. This is caused by a bug in the AMIs. For more information, see the CentOS Bug Tracker.

Workaround: None.

Terminating a cluster that is bootstrapping stops ongoing processes but keeps the cluster in the bootstrapping phase.

Severity: Low

Workaround: To transition the cluster to the Terminated phase, terminate the cluster again.

If you include Hive in your clusters and configure the Hive metastore to be installed on MySQL, Cloudera Manager may report, "The Hive Metastore canary failed to create a database." This is caused by a MySQL bug in MySQL 5.6.5 or higher that is exposed when used with the MySQL JDBC driver (used by Cloudera Director) version 5.1.19 or lower. For information on the MySQL bug, see the MySQL bug description.

Workaround: Depending on the driver version installed by Cloudera Director from your platform's software repositories, select an older MySQL version that does not have this bug.