Level 3: Configuring the Cluster to Authenticate Agent Certificates
Required Role: Cluster Administrator or Full Administrator
Level 3 TLS configuration ensures that each Cloudera Manager Agent host presents a signed certificate to Cloudera Manager Server prior to encryption, thus preventing possible introduction of a rogue host into the cluster. The steps in this section include obtaining signed certificates from a public CA.
Preliminary Task: Obtaining Certificates for the Agent
Check with the specific CA you plan to use before generating CSRs and follow their specific process.
You need to obtain certificates for each Cloudera Manager Agent host in the cluster.
Step 1: Create Directories for Security Artifacts
$ sudo mkdir -p /opt/cloudera/security/pki
Use the same directory on all cluster hosts to simplify management and maintenance.
Step 2: Generate Keys and CSRs for Cloudera Manager Agent Hosts
Generate keys and CSRs for each Cloudera Manager Agent host in the cluster using Java keytool. The command-line syntax in the steps below uses the $(hostname -f) variable to pass in the name of the host on which the command is run, to simplify the process.
- Create a Java keystore and private key for use by the agent host. Use the same password for the key and the keystore (Cloudera Manager requires they be the same) as shown here:
$ keytool -genkeypair -alias $(hostname -f)-agent -keyalg RSA -keystore \ /opt/cloudera/security/pki/$(hostname -f)-agent.jks -keysize 2048 -dname \ "CN=$(hostname -f),OU=Dept,O=Example,L=City,ST=State,C=US" \ -storepass password -keypass password
- Generate the CSR for certificates that can authenticate as both a client and a server, using the extended attributes serverAuth and clientAuth, as shown here:
$ keytool -certreq -alias $(hostname -f)-agent \ -keystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks \ -file /opt/cloudera/security/pki/$(hostname -f)-agent.csr \ -ext EKU=serverAuth,clientAuth \ -storepass password -keypass password
Step 3: Submit the CSR to the CA
- Submit the CSR file to your certificate authority using the process and means required by the CA, for example, email or web submission.
- The public CA will request specific details from you, to verify that you own the domain name contained in the CSR, before they issue the certificate.
- When you receive the signed certificate from the CA, you can proceed with to Step 4.
Step 4: Verify the Certificate
Use OpenSSL tool to confirm that each signed certificate includes both server and client authentication options:
$ openssl x509 -in /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem -noout -text
The response should include both TLS Web Server Authentication and TLS Web Client Authentication, as shown below:
X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication
- Check your CSR and make sure it was created using both extended attribute options,
- Re-submit the CSR to the CA and re-iterate that the signed certificate must include both options, so that it can authenticate as a server and a client.
Assuming you have properly constructed certificates, you can now distributed the certificates to the Cloudera Manager Agent host comprising the cluster.
Concatenated Certificates and Multiple Intermediate Certificates
If the CA provided you with the root CA and intermediate file concatenated into a single file, split the file along the END CERTIFICATE/BEGIN CERTIFICATE boundary into individual files.
If the CA provided you with multiple intermediate CA certificates, use unique file names for each, such as intca-1.cert.pem, intca-2.cert.pem, and so on.
Use these certificates as needed in Step 5.
Step 5: Distribute the Certificates to the Cloudera Manager Agent Hosts
On each Cloudera Manager Agent host:
- Copy each signed certificate to the appropriate host in the path shown below. Add a suffix -agent.cert to each for easy identification (as to functionality) as
- Copy any root and intermediate CA certificates to the following paths:
- Append the intermediate CA certificate to the signed certificate. Note the append operator (>>) in the command below, not the overwrite operator
$ sudo cat /opt/cloudera/security/pki/intca.cert.pem >> /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem
- Import the signed certificate into the keystore:
$ sudo keytool -importcert -alias $(hostname -f)-agent \ -file /opt/cloudera/security/pki/$(hostname -f)-agent.cert.pem \ -keystore /opt/cloudera/security/pki/$(hostname -f)-agent.jksThe command should return a message such as that shown below. Enter yes at the prompt to continue:
... is not trusted. Install reply anyway? [no]: yesYou should see a message confirming successful import of the signed certificate into the keystore, certifying the private key:
Certificate reply was installed in keystore
Assuming the certificate imported successfully, you can continue.
Configuring the Cloudera Manager Agent Hosts for Certificate Authentication
The steps below assume that the Preliminary Tasks (obtaining and distributing the signed certificates) are complete.
Step 7: Export the Private Key to a File
Repeat these steps each Cloudera Manager Agent host system.
On each Cloudera Manager Agent host, use the keytool utility to export the private key and certificate to a PKCS12 file, which can then be split up into individual key and certificate files using the openssl command:
- Create a PKCS12 version of the Java keystore:
$ keytool -importkeystore -srckeystore /opt/cloudera/security/pki/$(hostname -f)-agent.jks \ -srcstorepass password -srckeypass password \ -destkeystore /opt/cloudera/security/pki/$(hostname -f)-agent.p12 \ -deststoretype PKCS12 -srcalias $(hostname -f)-agent -deststorepass \ password -destkeypass password
- Use the openssl command to export the private key into its own file:
$ openssl pkcs12 -in /opt/cloudera/security/pki/$(hostname -f)-agent.p12 \ -passin pass:password -nocerts -out \ /opt/cloudera/security/pki/$(hostname -f)-agent.key -passout pass:password
- Create a symbolic link for the .key filename:
$ ln -s /opt/cloudera/security/pki/$(hostname -f)-agent.key /opt/cloudera/security/pki/agent.key
With the symlink pointing to agent.key on each Cloudera Manager Agent host in the cluster, the same /etc/cloudera-scm-agent/config.ini file can also be used for all agent hosts rather than maintaining individual files. (The configuration changes are made in Step 8, below.)
Step 8: Create a Password File
- Use a text editor to create a file called agentkey.pw that contains the password. Save the file in the /etc/cloudera-scm-agent directory.
- Change ownership of the file to root:
$ sudo chown root:root /etc/cloudera-scm-agent/agentkey.pw
- Change the permissions of the file:
$ sudo chmod 440 /etc/cloudera-scm-agent/agentkey.pw
- The password used in the echo command is the private key that was created when TLS was initially configured for the cluster (see Export the Private Key to a File).
- The path to the agentkey.pw should match the path specified by the client_keypw_file property in the Agent's config.ini file.
Step 9: Configure the Agent to Use Private Keys and Certificates
- Use a text editor to open the configuration file, located in this path:
- In the [Security] section in the file (just above the [Hadoop] section, find the PEM certificates section details, specifically:
- client_key_file, which identifies the path to the private key file.
- client_keypw_file, which identifies the path to the private key password file.
- client_cert_file, which identifies the path to the client certificate file.
- Remove the hash mark (#) at the start of the line and enter the values for your system, as shown below (in bold) using the settings created so far:
[security] ... # PEM file containing client private key client_key_file=/opt/cloudera/security/pki/agent.key ... # If client_keypw_cmd isn't specified, instead a text file contining # the client private key password can be used. client_keypw_file=/etc/cloudera-scm-agent/agentkey.pw # PEM file containing client certificate. client_cert_file=/opt/cloudera/security/pki/agent.cert.pem ...
Copy the file to all other cluster hosts. If you have modified properties such as listening_hostname or listening_ip address in config.ini, you must edit the file individually on each host.
Step 10: Enable Agent Certificate Authentication
- Log in to the Cloudera Manager Admin Console.
- Select .
- Click the Security category.
- Configure the following TLS settings:
Setting Description Use TLS Authentication of Agents to Server Click the box to enable TLS authentication for Cloudera Manager Agent hosts. By default, the box is unchecked. Cloudera Manager TLS/SSL Certificate Trust Store File Enter the full filesystem path to the jssecacerts file (created in Step 2: Create the Java Truststore) on the Cloudera Manager Server host. For example: /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts Cloudera Manager TLS/SSL Certificate Trust Store Password Specify the password for the jssecacerts truststore.
- Click Save Changes to save the settings.
Step 11: Restart Cloudera Manager Server and Agents
- On the Cloudera Manager server host, restart the Cloudera Manager server:
$ sudo service cloudera-scm-server restart
- On every agent host, restart the Cloudera Manager agent:
$ sudo service cloudera-scm-agent restart
Step 12: Verify Cloudera Manager Server and Agent Communications
In the Cloudera Manager Admin Console, go to Last Heartbeat column after restarting the agents and server, TLS certificate authentication is working properly. If not, check the agent log (/var/log/cloudera-scm-agent/cloudera-scm-agent.log) for errors. See TLS Level 2, Step 4 for details about accessing the log.. If you see successful heartbeats reported in the