New Features in Cloudera Manager 6.0.0

The following sections describe new and changed features for Cloudera Manager 6.0.0:

New Features in Cloudera Manager 6.0.0

API

You can now access the Cloudera Manager Swagger API user interface through the Cloudera Manager Admin Console. Navigate to Support > API Explorer to open Swagger.

Cloudera Manager 6.0 introduces new Python and Java API clients based on the Swagger API. These new API clients support all Cloudera Manager API versions.

Compatibility with Older Versions

The Older Python client Older Java client are still supported when the API version is lower than 30. Therefore, older Python and Java API clients can still be used with Cloudera Manager version 6.0 and higher as long as API version 19 or lower is used.

For example, a customer can use old Cloudera Manager API client version 5.14 with Cloudera Manager version 6.0, which by default invokes API version 19. If a customer wants to use new features that were introduced in Cloudera Manager 6.0, (API version 30) then customer must use the new API clients.

Older Python and Java clients and new Swagger-based Python and Java clients can co-exist in an application to allow for incremental transition to new the Swagger-based Python and Java clients.

Auto-TLS

Auto-TLS simplifies configuring TLS for clusters managed by Cloudera Manager. Cloudera Manager can now add hosts with TLS certificates automatically generated. Additionally, when you add new CDH services, Cloudera Manager populates the TLS configuration for the service.

You can use self-signed certificates created by Cloudera Manager's internal certificate authority, or you can use certificates you already have from a trusted public CA or your own internal CA.

An Enterprise or Trial license is required to enable Auto-TLS.

For more information, see Configuring TLS Encryption for Cloudera Manager and CDH Using Auto-TLS.

Cluster-specific User Roles

You can now assign privileges for specific clusters to the following user roles: Cluster Administrator, Operator, Limited Operator, Configurator, and Read-Only.

For example, the user account lucy has the Cluster Administrator role with privileges for a cluster named Cluster1. lucy can only perform the Cluster Administrator actions on Cluster1. She cannot perform actions on any other cluster managed by Cloudera Manager.

For more information, see Cloudera Manager User Roles.

Common Service Descriptors (CSD)

  • CSDs can now specify more than one repository for parcels. A list of URLs can be specified via the new property additionalRepoUrls of the parcel descriptor.
  • CSD authors can declare a set of invalid values for numeric parameters for the following types: long, double, port, and memory.

HBase

New command to create the HBase HDFS WAL directory in a separate location. By default the HBase HDFS WAL directory is created in /hbase/WALs. New command allows the HBase HDFS WAL directory to be created a different location. After setting the separate WAL directory, the Hbase service needs to be restarted.

Cell-Level ACL Checks

Newly deployed Cloudera Manager managed clusters will now optimize for HBase Cell-Level ACL checks by default with the hbase.security.access.early_out property. The property controls whether certain checks can be skipped for performance reasons

Impala

Cloudera Manager now collects more metrics that are helpful to Impala administrators for monitoring Catalog size and Impala Daemon health. Metrics from Impala Daemons help track the amount of memory used by the Java Virtual Machine (JVM) embedded in the Impala Daemon process. Use the metrics to understand memory consumption, particularly the memory consumption of the Catalog cache stored in coordinator Impala Daemons. The new metrics are:
  • impala_jvm_heap_committed_usage_byte
  • impala_jvm_heap_current_usage_bytes
  • impala_jvm_heap_init_usage_bytes
  • impala_jvm_heap_max_usage_bytes

Kafka

  • Added num.network.threads as a configuration setting for Kafka brokers.
  • Kafka's broker heap size is now configurable in the Cloudera Manager Admin Console wizard for Kafka.

Security

Cloudera Manager now passes SSL keystore credentials to SOLR through the Hadoop Credential Store.

Scalability

  • The performance for the cluster status table loading time on the Cloudera Manager Admin Console home page for large clusters has been improved.
  • The cluster restart command/operation can be retried now. This is especially helpful in a large cluster where the user has an opportunity to fix the cluster restart failure and retry the failed command.
  • When Cloudera Manager sees multiple agents with the same hostname or IP appearing, it will ask users to adjust the UUID on the host and remove the agent that has no roles running.

Upgrade

Cloudera Manager has the following upgrade improvements:

  • New upgrade wizard and documentation.

    The Cloudera Enterprise Upgrade Guide allows you to create a customized version of the guide that only includes the steps required for your upgrade. You can use a form at the top of pages in the guide to select your Cloudera Enterprise versions, operating system versions, databases, and other information about your upgrade. The information you enter is retained on each page in the guide.

  • You can now run the Host and Service inspectors up to two days prior to an upgrade. This allows a long running inspection to complete prior to starting the upgrade.
  • Rolling upgrades for CDH have been improved. YARN jobs running MapReduce2 are now configured to read MapReduce JARs from HDFS instead of from local disk. This makes jobs more robust during rolling upgrade when the local binaries are modified while a job is executing. Clusters created in or upgrading to CDH 6.0 will use this new behavior.

Changes in Cloudera Manager 6.0.0

Agents

Because of changes to Cloudera Manager, the commands used for a hard stop and hard restart have changed.

For more information, see Starting, Stopping, and Restarting Cloudera Manager Agents.

API names

The following API names have changed to fix typos:
  • hiverserver2_load_balancer has been changed to hiveserver2_load_balancer
  • hbase_client_java_opts has been changed to hdfs_client_java_opts
  • hbase_active_master_detecton_window has been changed to hbase_active_master_detection_window
  • hdfs_active_namenode_detecton_window has been changed to hdfs_active_namenode_detection_window
  • mapreduce_active_jobtracker_detecton_window has been changed to mapreduce_active_jobtracker_detection_window
  • yarn_active_resourcemanager_detecton_window has been changed to yarn_active_resourcemanager_detection_window

The hiverserver2_load_balancer change affects Hive services when HiveServer 2 is configured for High Availability.

The hdfs_client_java_opts parameter configures the Client Java Configuration Options, found under the HDFS Gateway role configuration.

The other parameters tune the behavior of health test checking for the HBase Master, HDFS NameNode, MapReduce JobTracker, and YARN ResourceManager respectively.

Any API scripts or cluster templates referencing the old names will need to be updated to use the new names.

Cloudera Issue: OPSAPS-33266, OPSAPS-39223, and OPSAPS-24569

Client Configurations

Downloading the client configuration for a service now requires a user account that meets the following requirements: the user account must be assigned a user role that has permission to perform the action and has privileges for the specific cluster because of the new cluster-specific user role feature.

External Authentication

Previously, Cloudera Manager, by default, mapped specific values from an external authentication method to Cloudera Manager user roles.

For example, if the authentication method for Cloudera Manager is a SAML Script, Cloudera Manager automatically mapped exit codes 0 to 11 to the user roles that Cloudera Manager ships with. If you upgrade to Cloudera Manager 6, these mappings are preserved. You can continue using these default mappings, create additional ones, or map different values.

If you perform a fresh installation of Cloudera Manager, values must be mapped to user roles manually in Cloudera Manager.

Additionally, LDAP Group, SAML Attribute, and External Program to user role mappings are no longer done through the Administration > Settings page. Instead, like the exit codes for SAML Scripts s, they are configured on the new Administration > Users & Roles (previously Users) > <Authentication Method> page.

For more information, see Mapping External Authentication to a Role.

HBase

  • Updated property values:
    Updated the default values for the following properties to match the upstream defaults:
    • hbase.snapshot.region.timeout
    • hbase.snapshot.master.timeout.millis
    • hbase.client.retries.number (all roles)
    • hbase.hstore.blockingStoreFiles (regionserver)
    Removed the following values:
    • hbase.snapshot.master.timeoutMillis
    • hbase.fs.tmp.dir (all roles)
    • hbase.bucketcache.combinedcache.enabled (regionserver)
    • hbase.bulkload.staging.dir (regionserver)
    • hbase.regionserver.hlog.blocksize (regionserver)
  • The HBase Thrift Server now turns on Framed Transport and Compact Protocols by default out of the box for safety reasons. This may require code changes to any custom client-side thrift programs that connect to the HBase Thrift Server to continue working.

Kafka

  • Data Retention Hours property - The Kafka Broker parameter Data Retention Hours (data.retention.hours) was removed from the Cloudera ManagerAdmin console. Use Data Retention Time (data.retention.ms) parameter instead.
  • Default Kafka minimum heap - The default minimum allowed heap for Kafka has been increased to 256 MB. The recommended minimum heap is 512 MB. If your Kafka broker heap size is set to a value less than 256 MB, increase it to 256 MB or higher.

New Users

If you create a user and do not assign a role to it, the user defaults to no access. The user cannot perform any actions on the cluster.

Reporting

The default HDFS block count reporting threshold for Cloudera Manager has been changed to 1000000 from 500000. When you upgrade, the configuration will be updated to the new default if you are using the default.

User Roles

The Dashboard and Auditor user roles can now view the Solr Collection Statistics and the HBase Table Statistics pages.