Your browser is out of date!

Update your browser to view this website correctly. Update my browser now

×

Sign in or complete our product interest form to continue.

Please Read and Accept our Terms


 

Take advantage of CDH in the cloud.

Cloudera Director provides a single administration interface for central IT to deliver agility and for end-users to self-service provision and elastically scale clusters, all while ensuring auditability.

 

Cloudera Director runs as a web application. You can download and install the Cloudera Director server and client by selecting Standard Installation in the dropdown above. If you are new to Cloudera Director, you can get started quickly by selecting AWS Quick Start and following the wizard.

 

Cloudera Director has native support for Amazon Web Services (AWS), Google Cloud Platform, and Microsoft Azure.
 

Each Cloudera Director release embeds the current plug-in for supported cloud providers, but a newer plug-in may have been posted on the Cloudera GitHub site subsequent to the Cloudera Director release. To check for the latest version, click the appropriate link:

Selected tab: CloudProviders

The Cloudera Director SPI defines an open source Java interface that plug-ins implement to add support for additional cloud providers to Cloudera Director. For more information, see the README.md file in the SPI Cloudera Director GitHub repository.

Selected tab: ClouderaDirectorServiceProviderInterfaceSPI

The table below lists software requirements, recommendations, and supported versions for resources used with Cloudera Director.

  Cloudera Director Cloudera Manager and CDH
Operating Systems (64-bit only)

RHEL and CentOS 6.5, 6.7, 6.8, 7.1, 7.2, and 7.3

Ubuntu 14.04

For AWS and Google Cloud Platform: RHEL and CentOS 6.5, 6.7, 6.8, 7.1, 7.2, and 7.3

For Microsoft Azure: RHEL and CentOS 6.7, 6.8, and 7.2

RHEL 7.2 is supported only for Cloudera Manager and CDH 5.7 and higher, not for lower versions of Cloudera Manager and CDH.

To use Amazon EC2 D2 instances, you must run a minimum version of RHEL 6.7 or CentOS 6.7. Earlier versions of RHEL and CentOS do not support these instance types.

Oracle Java SE Development Kit (JDK)

Oracle JDK version 7 or 8

For download and installation information, see Java SE Downloads.

Oracle JDK version 7 or 8

Default Database Embedded H2 database Embedded PostgreSQL Database
Supported Databases

MySQL 5.5, 5.6, 5.7

MariaDB 5.5

MySQL 5.5, 5.6, 5.7

MariaDB 5.5

 

Note: To run Kafka and Sentry on the same cluster, you must use Kafka 2.1 with Cloudera Manager and CDH 5.9 or 5.10.

 

Note: For the latest information on operating system versions supported on Microsoft Azure, refer to the Cloudera Reference Architecture for Microsoft Azure Deployments.

 

Note: By default, Cloudera Director stores its environment and cluster data in an embedded H2 database located at /var/lib/cloudera-director-server/state.h2.db. Back up this file to avoid losing the data. For information on using an external MySQL database in place of the H2 embedded database, see Using MySQL for Cloudera Director Server. Cloudera recommends using an external database for both Cloudera Director and Cloudera Manager for production environments.

Selected tab: SupportedSoftwareandDistributions

The table below lists requirements for resources used with Cloudera Director.

  Cloudera Director Cloudera Manager and CDH
CPU 2 4
RAM 3.75 GB 64 GB
Disk 8 GB 500 GB
Recommended AWS instance c3.large or c4.large

Cloudera Manager: m4.xlarge or m4.4xlarge

Recommended Google Cloud Platform instance n1-standard-2 n1-highmem-4 or n1-highmem-8
Recommended Microsoft Azure instance Standard_D3 or larger

The following Azure instance types are supported:

  • STANDARD_DS12_v2
  • STANDARD_DS13_v2
  • STANDARD_DS14_v2
  • STANDARD_DS13
  • STANDARD_DS14
  • STANDARD_DS15_v2
  • STANDARD_GS4
  • STANDARD_GS5

 

Note: For the latest information on instance types supported on Microsoft Azure, refer to the Cloudera Reference Architecture for Microsoft Azure Deployments.

 

Note: The recommended instance for Cloudera Manager depends on the workload. Some instance types may not be available in every region. Cloudera Director does not dynamically validate instance type by region. Contact your Cloudera account representative for more information.

Selected tab: ResourceRequirements

Cloudera Director 2.4 can install any version of Cloudera Manager 5 with any CDH 5 parcels. Use of CDH packages is not supported.

 

If you are using Cloudera Director 2.4 to deploy Cloudera Manager and CDH, the latest released version of Cloudera Manager 5.11 and CDH 5.11 is installed by default. To use any other version of Cloudera Manager or CDH, follow the instructions for installing non-default versions of Cloudera Manager and CDH in the Getting Started section for your cloud provider:

Selected tab: SupportedClouderaManagerandCDHVersions

Cloudera Director recommends the following inbound ports to be open:

  • TCP ports 22: These ports allow SSH to Cloudera Director instance.
  • All traffic across all ports within the security group: This rule allows connectivity with all the components within the Hadoop cluster. This rule avoids numerous individual ports to be opened in the security group.
     
Type Protocol Port Range Source
SSH (22) TCP (6) 22 0.0.0.0/0
ALL Traffic ALL ALL security_group_id

See note paragraph below.


Note: In AWS, the All traffic rule above requires the security group ID. If you create a security group from scratch, create the security group with the SSH rule and then go back and edit the security group to allow all traffic within the security group.


To connect to the AWS network, Cloudera recommends that you open only these ports and set up a SOCKS proxy. Unless your network has direct connection to AWS, you must set this up to access the Cloudera Director instance. This is done in a later step.

In a restricted network environment, you may want to enable minimal network traffic between instances and keep open ports to a minimum rather than enabling all network traffic between cluster instances. For information about minimal port requirements, see Ports Used by Cloudera Director.

Selected tab: NetworkingandSecurityRequirements

Cloudera Director supports the following browsers:

  • Mozilla Firefox 11 and higher
  • Google Chrome
  • Internet Explorer 9 and higher
  • Safari 5 and higher
Selected tab: SupportedBrowsers
Selected tab: SystemRequirements

Issues Fixed in Cloudera Director 2.4.1

 

Errors when using MySQL 5.7 for the Cloudera Director database

The defaults related to TIMESTAMP field handling changed drastically in MySQL 5.7.4 and later, which is documented in SQL Mode Changes in MySQL 5.7 in the MySQL documentation. One of the tables created by Cloudera Director, SERVER_CONFIGS, conflicts with these new defaults, which were valid in previous versions of MySQL. This is further complicated by the fact that MySQL 5.7.x will allow upgrades from MySQL 5.6 with tables that violate these defaults.

The result is that any modifications attempted on the SERVER_CONFIGS table in MySQL 5.7.x will fail. Cloudera Director 2.4 introduced a change to this table, triggering this problem. Additionally, new installs have been observed failing on MySQL 5.7.x due to the SERVER_CONFIGS table violating the expected defaults.

This issue has been fixed in Cloudera Director 2.4.1 with database changes that:

  • Adjust the creation of the SERVER_CONFIGS table on new installations
  • Correct SERVER_CONFIGS for users upgrading to Cloudera Director 2.4
  • Correct SERVER_CONFIGS for users who have already upgraded to Cloudera Director 2.4

For fresh installations on MySQL 5.7.x, this may affect any version of Cloudera Director starting with version 2.0. For existing installations that are now running on MySQL 5.7.x, this may affect users attempting to upgrade to Cloudera Director 2.4 from Cloudera Director versions 2.0 to 2.3. Running on MySQL 5.5.x or 5.6.x will behave as expected without any database failures.

Workarounds:

  • Cloudera recommends contacting Cloudera Support in order to fix this issue. However, if that is not an option, the following steps can be used to address the issue.
  • For a fresh install of Cloudera Director, the simplest workaround is to disable strict mode on MySQL. For more information about strict mode and how to disable it, see SQL Server Modes in the MySQL documentation. Using Cloudera Director 2.4.1 will avoid this issue.
  • For existing installs, manually modify the MySQL database to avoid this issue:
    • Upgrading from versions lower than 2.0.0: In this case, Cloudera Director will fail when trying to create the SERVER_CONFIGS table. In the database housing the Cloudera Director tables, examine the core_schema_version table and remove the line with the script value V3_2.0.0_1__init_serverconfig.sql.

      delete from core_schema_version where script = 'V3_2.0.0_1__init_serverconfig.sql';

      You should see a response like the following:

      Query OK, 1 row affected (0.02 sec)

      After this, retry the upgrade using Cloudera Director 2.4.1, or disable strict mode.
    • Upgrading from versions 2.0.0 to 2.4.0: In this case, Cloudera Director will fail when trying to modify several tables to remove the VERSION column. You must complete the migration manually and fix the TIMESTAMP issue.

      ALTER TABLE SERVER_CONFIGS MODIFY UPDATED_AT TIMESTAMP NULL, MODIFY CREATED_AT TIMESTAMP NULL;

      ALTER TABLE AUTHORITIES DROP COLUMN VERSION;
      ALTER TABLE CLUSTERS DROP COLUMN VERSION;
      ALTER TABLE DEPLOYMENTS DROP COLUMN VERSION;
      ALTER TABLE ENVIRONMENTS DROP COLUMN VERSION;
      ALTER TABLE EXTERNAL_DATABASE_SERVERS DROP COLUMN VERSION;
      ALTER TABLE INSTANCE_TEMPLATES DROP COLUMN VERSION;
      ALTER TABLE SERVER_CONFIGS DROP COLUMN VERSION;
      ALTER TABLE USERS DROP COLUMN VERSION;

      UPDATE core_schema_version set success = 1 where script = 'V3_2.4.0_1__remove_versions.sql'

      One or more of the ALTER TABLE statements may fail with an error that looks like the following:

      ERROR 1091 (42000): Can't DROP 'VERSION'; check that column/key exists

      This can be ignored because it was correctly deleted as part of the initial attempt to upgrade to Cloudera Director 2.4.

      After this, retry the migration. Cloudera recommends upgrading to Cloudera Director 2.4.1 as soon as possible, although these manual corrections should alleviate the issue.

 

Bootstrap fails because of empty parcel list

Cloudera Director fails in the middle of bootstrap with IllegalArgumentException: Parcel validation failed. This can happen when Cloudera Manager instances take longer than usual to refresh the list of parcels.

 

Unhealthy host causes "apply host template" to fail when growing the cluster

When growing an existing cluster the update operation may fail to add the instances. If the server log indicates "API call to Cloudera Manager failed. Method=HostTemplatesResource.applyHostTemplate," the user should enable Cloudera Manager API Debugging and check the server logs in Cloudera Manager to get more information on the failure. See Cloudera Manager API Call Fails in Troubleshooting Cloudera Director for information about checking Cloudera Manager logs.

One of the reasons for failure could be that the CDH parcel wasn't activated by the time Cloudera Director attempted to apply the host template. This specific scenario is likely to happen if newly added instances show up as unhealthy in Cloudera Manager.

Workaround: The best course of action is to try and figure out why the newly added instances comes up as unhealthy. This can sometimes be fixed by using a different AMI or instance type. If that doesn't work, Cloudera Director's lp.update.sleepTimeForAddInstanceSeconds server property can be increased to add additional time for the host to come back as healthy so that parcel gets distributed and activated before the API call to apply host template.

 

Azure VMs with manually attached Public IPs from different Resource Groups are marked as "not found"

An Azure VM with a manually attached Public IP from a different Resource Group will no longer be marked as "not found" and excluded from the the list of active cluster nodes. As of 2.4.1 the VMs will not report Public IPs from different Resource Groups but they will function as expected otherwise.

Workaround: Create the Public IP for manually attaching to the VM in the same Resource Group as the VM itself.

 

NullPointerException when Cloudera Director retrieves the private FQDN of a VM instance

In some rare cases the OS Profile metadata of an Azure VM can be empty. This can be confirmed by inspecting the VM metadata on Azure Resource Explorer ("osProfile" JSON block will be missing from the VM properties block). The OS Profile contains information such as the VM's private FQDN. An empty OS Profile can be related to Azure VM agents not running correctly on the VM. Cloudera recommends contacting Microsoft Azure support to resolve the issue where OS Profile is empty for an Azure VM. As of Cloudera Director 2.4.1, VM with missing OS Profile will no longer cause NullPointerException.

 

Add D series instances to Azure instance type list

The following instance types are added to to the Azure instance type list:

  • STANDARD_D15_v2
  • STANDARD_D14_v2
  • STANDARD_D13_v2
  • STANDARD_D12_v2

 

Expand Error Log for Unsupported Azure VMs

When deploying an unsupported Azure VM type the error message now contains actionable information for how to get and use the latest supported VM types.

 

Shorten Azure VMs Instance ID field in Cloudera Director UI

Azure VMs in Cloudera Director reported their instance IDs as a full Resource ID with Subscription ID and Resource Group name included. As of 2.4.1 the instance ID field is shortened to just the VM name.

 

Use static private IP address assignment option instead of dynamic

To guarantee the private IP address does not change after the VM is deallocated and restarted, the private IP allocation method must be Static. As of 2.4.1 the default private IP address allocation method is changed to Static.

Workaround: Manually change the private IP address assignment option to "Static" for each VM in the cluster via Azure portal.

Selected tab: WhatsNew

Want to Get Involved or Learn More?

Check out our other resources

Cloudera Community

Collaborate with your peers, industry experts, and Clouderans to make the most of your investment in Hadoop.

Cloudera University

Receive expert Hadoop training through Cloudera University, the industry's only truly dynamic Hadoop training curriculum that’s updated regularly to reflect the state of the art in big data.