Known Issues and Workarounds in Key Trustee KMS

New Key Trustee KMS may fail after being added to an environment that previously had a single Key Trustee KMS instance

When adding a new Navigator Key Trustee KMS instance to an environment that previously only had a single Key Trustee KMS instance, the new Key Trustee KMS may fail to start and return the following message:

"Unable to verify private key match between KMS hosts. If the system has been recently upgraded, DO NOT TAKE FURTHER ACTION and contact your support representative as soon as possible. If this is a new installation, verify private key files have been synced between all KMS hosts. Aborting to prevent data inconsistency."

This message correctly appears if GPG private keys have not been synced across all Key Trustee KMS instances. However, it may erroneously appear when the GPG private keys have been synced, but the original Key Trustee KMS instance has not been restarted to pick up the new configuration.

Affected Versions: 5.15.0

Cloudera Bug: DOCS-3156

Workaround: Restart the original Key Trustee KMS instance, and then start the new Key Trustee KMS instance.

Incorrect default KMS ACL values allow remote access to purge and undelete API calls on encryption zone keys

The Navigator Key Trustee KMS includes two API calls in addition to those in Apache Hadoop KMS: purge and undelete. The KMS ACL values for these commands are keytrustee.kms.acl.PURGE and keytrustee.kms.acl.UNDELETE, respectively. The default value for the ACLs in Key Trustee KMS 5.12.0 and 5.13.0 is "*", which allows anyone who knows the name of an encryption zone key and has network access to the Key Trustee KMS to make those calls against known encryption zone keys. The UNDELETE command will result in the recovery of a previously deleted, but not purged, key. The key will still be protected by normal ACLs and is not exposed. The PURGE command will permanently delete a key in active use, resulting in loss of access to encrypted HDFS data.

Affected Versions:
  • Navigator Key Trustee KMS 5.12.0, 5.13.0
  • Cloudera Manager 5.12.0, 5.12.1, 5.12.2
  • Cloudera Manager 5.13.0, 5.13.1

Fixed in Version: 5.14.0

Cloudera Bug: DOCS-2910

Workaround: Use Cloudera Manager to set the values of the ACLs for purge (keytrustee.kms.acl.PURGE) and undelete (keytrustee.kms.acl.UNDELETE) to be empty; this denies access to all calls to those functions.

To set the values:

  1. Log into Cloudera Manager as an administrative user.
  2. Click on the Key Trustee KMS service.
  3. Click on Configuration.
  4. In the Search box, type “ACL”.
  5. At the bottom of the section Key Management Server Proxy Advanced Configuration Snippet (Safety Valve) for kms-acls.xml add a new ACL by clicking the + icon.
  6. In the Name field add “keytrustee.kms.acl.PURGE”.
  7. Repeat step 5, and in the Name field add “keytrustee.kms.acl.UNDELETE”.
  8. Scroll to the bottom of the page and click Save.
  9. Use Cloudera Manager to perform a rolling restart of the Navigator Key Trustee KMS service.

Cannot re-encrypt an encryption zone if a previous re-encryption on it was canceled

When canceling a re-encryption on an encryption zone, the status of the re-encryption may continue to show "Processing". When this occurs, future re-encrypt commands for this encryption zone will fail inside the NameNode, and the re-encryption will never complete.

Affected Version: 5.13.0

Fixed Version: 5.13.1

Cloudera Bug: CDH-59073

Workaround: To halt, or remove the "Processing" status for the encryption zone, re-issue the cancel re-encryption command on the encryption zone. If a new re-encryption command is required for this encryption zone, restart the NameNode before issuing the command.

Adding Key Trustee KMS 5.4 to Cloudera Manager 5.5 displays warning

Adding the Key Trustee KMS service to a CDH 5.4 cluster managed by Cloudera Manager 5.5 displays the following message, even if Key Trustee KMS is installed:

"The following selected services cannot be used due to missing components: keytrustee-keyprovider. Are you sure you wish to continue with them?"

Affected Version: 5.4

Workaround: Verify that the Key Trustee KMS parcel or package is installed and click OK to continue adding the service.

KMS and Key Trustee ACLs do not work in Cloudera Manager

ACLs configured for the KMS (File) and KMS (Navigator Key Trustee) services do not work since these services do not receive the values for hadoop.security.group.mapping and related group mapping configuration properties.

Affected Version: 5.3 up to and including 5.14.0

Cloudera Bug: OPSAPS-24483

Workaround:

KMS (File): Add all configuration properties starting with hadoop.security.group.mapping from the NameNode core-site.xml to the KMS (File) property, Key Management Server Advanced Configuration Snippet (Safety Valve) for core-site.xml

KMS (Navigator Key Trustee): Add all configuration properties starting with hadoop.security.group.mapping from the NameNode core-site.xml to the KMS (Navigator Key Trustee) property, Key Management Server Proxy Advanced Configuration Snippet (Safety Valve) for core-site.xml.

The Key Trustee KMS service fails to start if the Trust Store is configured without also configuring the Keystore

If you configure the Key Trustee KMS service Key Management Server Proxy TLS/SSL Certificate Trust Store File and Key Management Server Proxy TLS/SSL Certificate Trust Store Password parameters without also configuring the Key Management Server Proxy TLS/SSL Server JKS Keystore File Location and Key Management Server Proxy TLS/SSL Server JKS Keystore File Password parameters, the Key Trustee KMS service does not start.

Workaround: Configure all Trust Store and Keystore parameters.

Key Trustee KMS backup script fails if PostgreSQL versions lower than 9.3 are installed

If PostgreSQL versions lower than 9.3 are installed on the Key Trustee KMS host, the ktbackup.sh script fails with an error similar to the following:

pg_dump: server version: 9.3.11; pg_dump version: 9.2.14
pg_dump: aborting because of server version mismatch 

Workaround: Uninstall the lower PostgreSQL version.