What's New in CDH 5.16.x

See below for new features in CDH 5.16.x, grouped by component:

Apache Hive

The following new features have been added to Apache Hive in CDH 5.16.1:

Fine Grained Privileges

Sentry and Hive introduced fine grained privileges to provide object-level privileges to roles.

Fine grained privileges adds the CREATE privilege, which allow users to create databases and tables. See the Sentry Privileges documentation for more information about the new privileges.

Object Ownership

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the equivalent of the ALL privilege on the object. See the Object Ownership documentation for information about enabling object ownership.

Because of the new object ownership feature, HMS stores the user that creates a table or database in Hive as the object owner. If object ownership is enabled, Sentry grants the user the OWNER privilege. Whether or not object ownership is enabled, HMS stores the user that creates the object as the object owner. Previously, HMS stored the hive user as the object owner.

The following statements were added to Hive to support object ownership via Sentry:

  • ALTER DATABASE SET OWNER
  • ALTER TABLE SET OWNER
  • SHOW GRANT USER

Apache Impala

The following new features have been added to Apache Impala in CDH 5.16.1:

Fine Grained Privileges

Sentry and Impala introduced fine grained privileges to provide object-level privileges to roles.

Fine grained privileges include the REFRESH and CREATE privileges, which allow users to create databases and tables, and to execute commands that update metadata information on Impala databases and tables. See Impala Sentry documentation for the new privileges and the scopes of the objects that you can grant the new privileges on.

The following new privileges were added:

  • The REFRESH privilege
  • The CREATE privilege
  • The SELECT and INSERT privileges on SERVER

If a role has SELECT or INSERT privilege on an object in Impala before upgrading to CDH 5.16.1, that role will automatically get the REFRESH privilege during the upgrade.

Object Ownership

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the OWNER privilege which is the equivalent of the ALL privilege on the object. See the Object Ownership documentation for information about enabling object ownership.

If the object ownership feature is enabled, Sentry grants the user the OWNER privilege. Whether or not object ownership is enabled, HMS stores the object creator as the default object owner. Previously, HMS stored the Kerberos user as the object owner.

The following statements were added to Impala to support object ownership via Sentry:

  • ALTER DATABASE SET OWNER
  • ALTER TABLE SET OWNER
  • ALTER VIEW SET OWNER
  • SHOW GRANT USER

Admission Control Enhancement

A new query option, MAX_MEM_ESTIMATE_FOR_ADMISSION, was added. Use the new option to set an upper limit on the memory estimates of a query as a workaround for over-estimates precluding a query from being admitted.

General Performance Improvements

A new query option, SHUFFLE_DISTINCT_EXPRS, controls the shuffling behavior when a query has both grouping and distinct expressions.

Metadata Performance Improvements

  • Incremental Stats

    The following enhancements improve Impala stability. The features reduce chances of having catalogd and impalad crash due to be out of memory when using incremental stats.

    • Incremental stats are now compressed in memory in catalogd, reducing memory footprint in catalogd.
    • Incremental stats are fetched on demand from catalogd by impalad coordinators. This enhancement reduces memory footprint for impalad coordinators and statestored and also reduces network requirements to broadcast metadata.

    See Loading Incremental Statistics from Catalogd for details.

  • Automatic Invalidation of Metadata

    To keep the size of metadata bounded and to reduce the chances of catalogd cache running out of memory, this release introduces an automatic metadata invalidation feature with time-based and memory-based invalidation.

    Automatic invalidation of metadata provides more stability with lower chances of running out of memory, but could potentially cause performance risks. The feature is turned off by default.

    See Startup Options for Automatic Invalidation of Metadata for details.

Compatibility and Usability Enhancements

  • Additional separators are supported between date and time in default TIMESTAMP format, specifically, the multi-space separator and the 'T' separator. See TIMESTAMP Data Type for more information on TIMESTAMP format.

  • New hint placement is supported for INSERT statements. See Optimizer Hints in Impala for detail.

  • The REGEX_ESCAPE() function was implemented for escaping special characters to treat them literally in string literals.

  • SHOW CREATE VIEW was implemented with the same functionality as SHOW CREATE TABLE.

  • Impala Shell can connect directly to impalad when configured with proxy load balancer and Kerberos. See Configuration Options for the new flag that enables the direct connection.

Apache Sentry

The following new features have been added to Apache Sentry in CDH 5.16.1:

Fine Grained Privileges

The CREATE and REFRESH (Impala only) privileges have been introduced to allow users to create databases, tables and functions, and to execute commands that update metadata information on Impala databases and tables.

For more information about the new privileges, see Sentry Privileges.

Object Ownership

Object ownership designates an owner for a database, table, or view in Sentry. The owner of an object has the equivalent of the ALL privilege on the object.

In CDH 5.16.1, object ownership is disabled by default. For information about enabling and using object ownership, see Object Ownership.

No Group Name Case Restrictions

Sentry no longer normalizes group name characters to be lowercase. Therefore, operating system group names do not need to be treated as case insensitive. In previous versions, Sentry modified capital letters in operating system group names to be lowercase.

OpenJDK Support

OpenJDK is now supported with Cloudera Manager and CDH 5.16.1.

For information about migrating your Oracle JDK to OpenJDK, see Upgrading the JDK. To use OpenJDK in a new deployment, see Install Java Development Kit.

Cloudera Bug: CDH-60334

Apache Kudu

The following new features have been added to Apache Hive in CDH 5.16.1:

Starting with Apache Kudu 1.5.0 / CDH 5.13.x, Kudu has been fully integrated into CDH. Kudu now ships as part of the CDH parcel and packages. For a complete list of new features and changes introduced in Kudu 1.7 / CDH 5.16.1, see What's New in Apache Kudu.

Apache Phoenix

Apache Phoenix is available with CDH 5.16.2:

Starting with Apache Phoenix 4.14.1/ CDH 5.16.2, Phoenix parcel can be installed and used with CDH. You can now download and install the Phoenix parcel on CDH 5.16.2. For more information about installing and using Phoenix, see the Phoenix Guide.