Troubleshooting HDFS Encryption
This topic contains HDFS Encryption-specific troubleshooting information in the form of issues you might face when encrypting HDFS files/directories and their workarounds.
Retrieval of encryption keys fails
DescriptionYou see the following error when trying to list encryption keys
user1@example-sles-4:~> hadoop key list Cannot list keys for KeyProvider: KMSClientProvider[https: //example-sles-2.example.com:16000/kms/v1/]: Retrieval of all keys failed.
DistCp between unencrypted and encrypted locations fails
DescriptionBy default, DistCp compares checksums provided by the filesystem to verify that data was successfully copied to the destination. However, when copying between unencrypted and encrypted locations, the filesystem checksums will not match since the underlying block data is different.
Cannot move encrypted files to trash
NameNode - KMS communication fails after long periods of inactivity
Encrypted files and encryption zones cannot be created if a long period of time (by default, 20 hours) has passed since the last time the KMS and NameNode communicated.
- You can increase the KMS authentication token validity period to a very high number. Since the default value is 10 hours, this bug will only be encountered after 20 hours of no
communication between the NameNode and the KMS. Add the following property to the kms-site.xmlSafety Valve:
<property> <name>hadoop.kms.authentication.token.validity</name> <value>SOME VERY HIGH NUMBER</value> </property>
- You can switch the KMS signature secret provider to the string secret provider by adding the following property to the kms-site.xml Safety Valve:
<property> <name>hadoop.kms.authentication.signature.secret</name> <value>SOME VERY SECRET STRING</value> </property>