Operating System Known Issues

Oracle Linux 6.8 rejects TLS weak ciphers

In Oracle Linux 6.8 configurations, the curl command cannot connect to certain CDH services that run on Apache Tomcat when the cluster has been configured for TLS/SSL. Specifically, HttpFS, KMS, Oozie, and Solr services reject connection attempts because the default cipher configuration uses weak temporary server keys (based on Diffie-Hellman key exchange protocol).

Workaround: Override the default cipher configuration using all the ciphers from the list below. Only commas separate each cipher, and there are no line endings or other characters in this list.

  • For clusters not managed by Cloudera Manager, enter this list in the appropriate configuration file for the service (see the "Not managed" column in the table below). Restart the service.
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA
    
  • For clusters managed by Cloudera Manager, follow the steps below.
  Cluster managed by Cloudera Manager Not managed
Service Configuration property Environment variable Configuration file
HttpFS HttpFS Environment Advanced Configuration Snippet (Safety Valve) HTTPFS_SSL_CIPHERS /etc/hadoop-httpfs/conf/httpfs-env.sh
KMS Key Management Server Environment Advanced Configuration Snippet (Safety Valve) KMS_SSL_CIPHERS /etc/hadoop-kms/conf/kms-env.sh
Oozie Oozie Server Environment Advanced Configuration Snippet OOZIE_HTTPS_CIPHERS /etc/oozie/conf/oozie-env.sh
Solr Solr Server Environment Advanced Configuration Snippet SOLR_CIPHERS_CONFIG /etc/default/solr

To override the ciphers on Cloudera Manager clusters:

  1. Log into Cloudera Manager Admin Console.
  2. Search for the advanced configuration snippet name ("Configuration property") for each respective service role.
  3. Enter the appropriate environment variable in the entry field.
  4. Copy the list from here:
    TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA
    
  5. Paste as the setting for the environment variable. For example, overriding HttpFS is as follows:



  6. Restart the service.

SUSE Linux Enterprise Server 12 (SLES12) and Kerberos Credential Cache Issue

Cloudera has become aware of a conflict in the handling of the Kerberos credential cache in SLES12 that can result in the following error:
Credential cache directory /run/user/uid does not exist

The issue is caused by a change in SLES12 systemd and the location of Kerberos credentials cache for user accounts. This issue may emerge when passwordless accounts, such as impala or flume, for example, are executed using sudo -u user or su - user and the system attempts to retrieve the credential from the cache.

Workaround:

To work around this issue, add the following line to your system's Kerberos configuration file (/etc/krb5.conf), in the [libdefaults] section:
[libdefaults]
default_ccache_name = FILE:/tmp/krb5cc_%{uid}
...

Random JVM Hangs on RHEL 6.6 (Kernels 3.14-3.17)

User Java processes—HBase RegionServers, HDFS DataNodes—can deadlock and hang for no apparent reason.

Impact: A Java application or daemon hangs. A jstack, or attaching a debugger, can unblock the process.

Workaround:

Upgrade the operating system to one with kernel version 3.18 or later; for example, RHEL 6.6z and later. For information on checking your RHEL kernel version, see http://www.cyberciti.biz/faq/centsos-redhat-rhel-6-kernel-version/. In RHEL, the issue has been fixed recently in kernel-2.6.32-504.16.2.el6 update (April 21, https://rhn.redhat.com/errata/RHSA-2015-0864.html). The following example shows how to check for the fix:

% rpm -qp --changelog kernel-2.6.32-504.16.2.el6.x86_64.rpm | grep 'Ensure get_futex_key_refs() always implies a barrier'

Other distributions running kernel versions 3.14-3.17 may be affected.

For more information, see https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64

Leap-Second Events

Impact: After a leap-second event, Java applications (including CDH services) using older Java and Linux kernel versions, may consume almost 100% CPU. See https://access.redhat.com/articles/15145.

Leap-second events are tied to the time synchronization methods of the Linux kernel, the Linux distribution and version, and the Java version used by applications running on affected kernels.

Although Java is increasingly agnostic to system clock progression (and less susceptible to a kernel's mishandling of a leap-second event), using JDK 7 or 8 should prevent issues at the CDH level (for CDH components that use the Java Virtual Machine).

Immediate action required:

(1) Ensure that the kernel is up to date.

  • RHEL5/6/7, CentOS 5/6/7 - 2.6.32-298 or higher

  • Ubuntu Trusty 14.04 LTS - No action required

  • Ubuntu Precise 12.04 LTS - 3.2.0-29.46 or higher

  • Debian Jessie (8.x)- No action required

  • Debian Wheezy (7.x) - 3.2.29-1 or higher

  • Oracle Enterprise Linux (OEL) - Kernels built in 2013 or later

  • SLES11 (SP2, SP3 and SP4)

    • SP2 - 3.0.38-0.5 or higher

    • SP3 and SP4 - No action required

  • SLES12 - No action required.

(2) Ensure that your Java JDKs are current (especially if the kernel is not up to date and cannot be upgraded).
  • Java 7 - At least jdk-1.7.0_65

  • Java 8 - No action required.

(3) Ensure that your systems use either NTP or PTP synchronization.

For systems not using time synchronization, update both the OS tzdata and Java tzdata packages to the tzdata-2016g version, at a minimum. For OS tzdata package updates, contact OS support or check updated OS repositories. For Java tzdata package updates, see Oracle's Timezone Updater Tool.