Configuring TLS Encryption for Cloudera Manager and CDH Using Auto-TLS

Auto-TLS, first introduced in Cloudera Manager 5.13 on Cloudera Director 2.6, is now available for on-premises clusters in Cloudera Enterprise 6.

Auto-TLS greatly simplifies the process of enabling and managing TLS encryption on your cluster. It automates the creation of an internal certificate authority (CA) and deployment of certificates across all cluster hosts. It can also automate the distribution of existing certificates, such as those signed by a public CA. Adding new cluster hosts or services to a cluster with auto-TLS enabled automatically creates and deploys the required certificates.

To configure TLS encryption manually, see Configuring TLS Encryption for Cloudera Manager.

Auto-TLS Requirements and Limitations

  • You must enable auto-TLS as part of a new installation. You cannot enable auto-TLS on an existing cluster. For instructions on how to enable TLS on an existing cluster manually, see Configuring TLS Encryption for Cloudera Manager.
  • You must install the Cloudera Manager Agent software on the Cloudera Manager Server host.
  • You must configure and enable auto-TLS before adding any hosts to Cloudera Manager.
  • You can enable auto-TLS using certificates created and managed by a Cloudera Manager certificate authority (CA), or certificates signed by a trusted public CA or your own internal CA. If you want to use a trusted public CA or your own internal CA, you must obtain all of the host certificates before enabling auto-TLS. For instructions on obtaining certificates from a CA, see Generate TLS Certificates.
  • The following Cloudera Enterprise services support auto-TLS. For unlisted services, you must enable TLS manually:
    Auto-TLS Supported Components
    Cloudera Enterprise Components that Support Auto-TLS
    HDFS Client Configuration Hue Client Impala Server Cloudera Navigator Metadata Server
    HDFS NameNode Web UI Hue Load Balancer Impala StateStore Oozie
    HiveServer2 Hue Server Kafka Broker Server Spark History Server
    HttpFS Impala Catalog Server Cloudera Navigator Audit Server YARN Web UI

The certmanager Utility

Auto-TLS is managed using the certmanager utility, which is included in the Cloudera Manager Agent software, and not the Cloudera Manager Server software. You must install the Cloudera Manager Agent software on the Cloudera Manager Server host to be able to use the utility.

The certmanager syntax is as follows:

/opt/cloudera/cm-agent/bin/certmanager [OPTIONS] COMMAND [ARGS]...
  • Options:
    • --location <certmanager-dir-root>

      The directory where certmanager stores all of its files on the Cloudera Manager Server host. If omitted, defaults to /var/lib/cloudera-scm-server/certmanager. This directory is created automatically, and must not exist before running the command. The agent host certificates and other files are stored elsewhere on each agent (see Auto-TLS Agent File Locations).

    • --help

      Displays the help message.

  • Commands:
    • add_custom_cert

      Adds a custom certificate and key for a host. Use this command only if you have configured a custom certificate directory (using the setup_custom_certdir command). You must run this command before adding a host in Cloudera Manager.

    • export_ca_cert

      Displays the Cloudera Manager internal CA certificate. You can export the certificate to a file using a redirect operator (> or >>).

    • setup

      Initializes the certificate manager and the internal CA, and configures Cloudera Manager Server to enable auto-TLS.

      • --configure-services

        Configures Cloudera Manager Server to enable automatic configuration of TLS for supported components, such as HDFS, YARN, and so on. If you omit this option, auto-TLS will only be configured for Cloudera Manager agent/server communication.

    • setup_custom_certdir

      Initializes the certificate manager using a custom certificate directory. Use this command if you are using existing certificates signed by a trusted public CA or your own internal CA.

      • --configure-services

        Configures Cloudera Manager Server to enable automatic configuration of TLS for supported components, such as HDFS, YARN, and so on. If you omit this option, auto-TLS will only be configured for Cloudera Manager agent/server communication.

Enabling Auto-TLS to Automatically Create Certificates

To allow auto-TLS to automatically create and manage certificates:

  1. Install Cloudera Manager Server and Cloudera Manager Agent on the Cloudera Manager Server host.
  2. On the Cloudera Manager Server host, initialize the certificate manager and configure Cloudera Manager Server to enable auto-TLS by running the following command:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager setup --configure-services

    Replace the JDK version with the correct path for your environment. If you want to store the files in a directory other than the default (/var/lib/cloudera-scm-server/certmanager), add the --location option as follows:

    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services

    Replace /opt/cloudera/CMCA with path to the directory you want to use. This directory is created automatically, and must not exist before running the command. Check the /var/log/cloudera-scm-agent/certmanager.log log file to confirm that the directories were created.

  3. If Cloudera Manager Server is running, restart it:
    RHEL 7 Compatible, Ubuntu 16, SLES 12
    sudo systemctl restart cloudera-scm-server
    RHEL 6 Compatible
    sudo service cloudera-scm-server restart
  4. Continue with the installation.

Enabling Auto-TLS with Existing Certificates

To use auto-TLS with existing certificates:

  1. Make sure that you have host certificates for all cluster hosts. For instructions on obtaining certificates from a certificate authority (CA), see Generate TLS Certificates.
  2. Install Cloudera Manager Server and Cloudera Manager Agent on the Cloudera Manager Server host.
  3. Initialize the certificate manager and configure Cloudera Manager Server to enable auto-TLS, using a custom certificate directory:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup_custom_certdir --configure-services --host-cert /path/to/hostcert.pem --host-key /path/to/hostkey.pem --ca-cert /path/to/cacert.pem --keystore-pw-file /path/to/keystore_pw_file --truststore-pw-file /path/to/truststore_pw_file

    Replace the JDK version with the correct path for your environment. Replace /path/to/hostcert.pem and /path/to/hostkey.pem with the certificate and key, respectively, of the Cloudera Manager Server host.

    The directory specified by the --location parameter is created automatically, and must not exist before running the command. Check the /var/log/cloudera-scm-agent/certmanager.log log file to confirm that the directories were created.

  4. For each cluster host (with unique certificates and keys), run the following command:
    sudo JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA add_custom_cert --host-cert <host_cert> --host-key <host_key> <hostname>

    Replace the JDK version with the correct path for your environment.

  5. If Cloudera Manager Server is running, restart it:
    RHEL 7 Compatible, Ubuntu 16, SLES 12
    sudo systemctl restart cloudera-scm-server
    RHEL 6 Compatible
    sudo service cloudera-scm-server restart
  6. Continue with the installation.

Auto-TLS Agent File Locations

The certificates, keystores, and password files generated by auto-TLS are stored in /var/lib/cloudera-scm-agent/agent-cert on each Cloudera Manager Agent. The filenames are as follows:

Auto-TLS Agent Files
Filename Description
cm-auto-global_cacerts.pem CA certificate and other trusted certificates in PEM format
cm-auto-global_truststore.jks CA certificate and other trusted certificates in JKS format
cm-auto-in_cluster_ca_cert.pem CA certificate in PEM format
cm-auto-in_cluster_truststore.jks CA certificate in JKS format
cm-auto-host_key_cert_chain.pem Agent host certificate and private key in PEM format
cm-auto-host_cert_chain.pem Agent host certificate in PEM format
cm-auto-host_key.pem Agent host private key in PEM format
cm-auto-host_keystore.jks Agent host private key in JKS format
cm-auto-host_key.pw Agent host private key password file