Issues Fixed in CDH 5.16.x

The following topics describe issues fixed in CDH 5.16.x, from newest to oldest release. You can also review What's new in CDH in 5.16.x or Known issues in CDH 5.

Issues Fixed in CDH 5.16.1

Apache Tomcat Vulnerability CVE-2018-11784

Fixed a vulnerability in Apache Tomcat where specially-crafted URLs could be used to redirect to any given URI. CVE-2018-11784.

Cloudera Issue: CDH-73885

Cloudera Search restore operation puts shard replicas on same host

Restoring an Apache Solr collection sometimes places all shard replicas on the same host.

Cloudera Issue: CDH-68828

Missing authorization in Apache Impala may allow data injection

A malicious user who is authenticated with Kerberos may have unauthorized access to internal services used by Impala to transfer intermediate data during query execution. If details of a running query (e.g. query ID, query plan) are available, a user can craft some RPC requests with custom software to inject data into a running query or end query execution prematurely, leading to wrong results of the query.

Cloudera Issue: CDH-72373 / TSB-338

CVE: CVE-2018-11785

Upstream Issues Fixed

Apache Hadoop

  • HADOOP-13426 - Improved IPC performance.
  • HADOOP-13483 - Fixed an issue where file-create overwrote directories instead of throwing error messages
  • HADOOP-15473 - Configured the serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997
  • HADOOP-15655 - Enhanced KMS client retry behavior. Previously, the KMS did not retry upon SocketTimeoutException.
  • HDFS-8229 - Fixed an issue where the LAZY_PERSIST file gets deleted after NameNode restart.
  • HDFS-10240 - Fixed a race between close/recoverLease leads to missing block
  • HDFS-12299 - Fixed a race between update pipeline and DN Re-Registration
  • HDFS-13051 - Fixed a dead lock during async editlog rolling if the edit queue is full.
  • HDFS-13322 - Fixed an issue where the UID persists when switching between ticket caches.
  • HDFS-13486 - Fixed an issue where a faulty node can cause a lease leak and NPE on accessing data.
  • HDFS-13601 - Optimized ByteString conversions in PBHelper.
  • HDFS-13611 - Fixed an issue where text was used as a ConcurrentHashMap key in PBHelperClient.
  • HDFS-13813 - Added a check to see if a child inode exists in the global FSDirectory directory when saving (serializing) INodeDirectorySection.
  • MAPREDUCE-7053 - Fixed an issue where Timed out tasks can fail to produce thread dump
  • YARN-6966 - Fixed an issue where NodeManager metrics may return wrong negative values when NM restart.
  • YARN-6967 - Fixed an issue where the limit for diagnostic message size was not honored
  • YARN-8436 - Fixed an issue where the ResourceManager can fail while sorting queues if an update comes in

Apache HBase

  • HBASE-15232 - Handle region location cache mgmt in AsyncProcess for multi()'s
  • HBASE-15390 - Unnecessary MetaCache evictions cause elevated number of requests to meta
  • HBASE-18891 - Upgrade to netty-all 4.0.50.Final
  • HBASE-19924 - hbase rpc throttling does not work for multi() with request count rater.
  • HBASE-20493 - Port HBASE-19994 (Create a new class for RPC throttling exception, make it retryable) to branch-1
  • HBASE-20651 - Master, prevents hbck or shell command to reassign the split parent region
  • HBASE-20723 - Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them
  • HBASE-20997 - rebuildUserRegions() does not build ReplicaMapping during master switchover

Apache Hive

Code Changes Should Not Be Required

The following fixes should not require code changes, but they contain improvements that might enhance your deployment:

  • HIVE-6980 - Drop table by using direct sql
  • HIVE-10296 - Cast exception observed when hive runs a multi join query on metastore (postgres), since postgres pushes the filter into the join, and ignores the condition before applying cast
  • HIVE-12981 - ThriftCLIService uses incompatible getShortName() implementation
  • HIVE-15237 - Propagate Spark job failure to Hive
  • HIVE-15860 - RemoteSparkJobMonitor may hang when RemoteDriver exits abnormally
  • HIVE-16483 - HoS should populate split related configurations to HiveConf
  • HIVE-17213 - HoS file merging doesn't work for union all
  • HIVE-18031 - Support replication for Alter Database operation
  • HIVE-18283 - Better error message and error code for HoS exceptions
  • HIVE-18765 - SparkClientImpl swallows exception messages from the RemoteDriver
  • HIVE-18916 - SparkClientImpl doesn't error out if spark-submit fails
  • HIVE-19259 - Create view on tables having union all fail with 'Table not found'
  • HIVE-19310 - Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
  • HIVE-19371 - Add table ownerType to HMS thrift API
  • HIVE-19372 - Add table ownerType to JDO/SQL and ObjectStore
  • HIVE-19374 - Parse and process ALTER TABLE SET OWNER command syntax
  • HIVE-19605 - TAB_COL_STATS table has no index on db/table name
  • HIVE-19668 - Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
  • HIVE-19783 - Retrieve only locations in HiveMetaStore.dropPartitionsAndGetLocations
  • HIVE-20183 - Inserting from bucketed table can cause data loss, if the source table contains empty bucket
  • HIVE-20345 - Drop database may hang if the tables get deleted from a different call

Hue

  • HUE-8118 - [core] Fine grain tracking of the memory usage
  • HUE-8118 - [core] The duration of the request is always shown even when instrumentation flag is off
  • HUE-8128 - [backend] Force debug logging in server logs does not get all debug
  • HUE-8162 - [core] Add delete operation to the right document assist
  • HUE-8177 - [oozie] Add a config check for /user/hue/oozie/workspaces
  • HUE-8377 - [security] Support new Sentry finer grain privileges
  • HUE-8377 - [security] Correctly apply the new permissions to the database scope
  • HUE-8451 - [notebook] Many "codec can't decode byte" errors on pig execution if browser language=jp
  • HUE-8464 - [core] Fix SAML encryption missing key file passphrase
  • HUE-8467 - [jobbrowser] Support impala digest auth for queries
  • HUE-8475 - [report] Protect against pivot conflicting with nested facets
  • HUE-8476 - [frontend] Fix jQuery Hive autocomplete column mapping
  • HUE-8487 - [useradmin] Fix Add Sync LDAP user fails when using DN with special character
  • HUE-8505 - [core] Close impala session on logout
  • HUE-8519 - [jb] Impala API can now directly return json
  • HUE-8558 - [jb] Add tracking URL to Spark Jobs and remove url and killUrl
  • HUE-8564 - [useradmin] Fix last activity update for jobbrowser/api/jobs requests
  • HUE-8564 - [useradmin] Fix last activity update for notebook/api/check_status
  • HUE-8571 - [sentry] navigator_api ERROR for PRIVILEGE_HIERARCHY[hierarchy[server][SENTRY_PRIVILEGE_KEY]['action']]
  • HUE-8602 - [sentry] Remove ALTER and DROP in the Hive section

Apache Impala

  • IMPALA-6086 - Require the SELECT privilege on the database for built-in function calls.
  • IMPALA-6451 - Fixed the AuthorizationException in CTAS for Kudu tables.
  • IMPALA-6479 - DESCRIBE now respects column level privileges and only shows the columns that the user has the privilege to view.
  • IMPALA-6571 - Fixed the NullPointerException in SHOW CREATE TABLE for HBase tables.
  • IMPALA-6917 - Limit impalad mem-limit to 12GB.
  • IMPALA-7225 - REFRESH..PARTITION no longer reset the number of rows in a partition.
  • IMPALA-7272 - Fixed the crash in StringMinMaxFilter.
  • IMPALA-7360 - Fixed an issue where Avro scanner sometimes skipped blocks when skip marker was on HDFS block boundary.
  • IMPALA-7419 - Fixed the NullPointerException in SimplifyConditionalsRule.
  • IMPALA-7483 - impalad/catalogd on JVM deadlock now get aborted.
  • IMPALA-7520 - Fixed the NullPointerException in SentryProxy.

Apache Kudu

  • KUDU-2260 - Fixed a rare issue where system failure could leave unexpected null bytes at the end of metadata files, causing Kudu to be unable to restart.
  • KUDU-2364 - Fixed an issue when a tablet server was wiped and recreated with the same RPC address, ksck listed it twice, both as healthy, even though only one of them was there.
  • KUDU-2412 - The kudu-python client can now compile in environments where __int128 is not supported. This was most commonly el6 environments.
  • KUDU-2509 - Fixed an issue that might result in a crash of a tablet server in case of a WAL replay error while bootstrapping a tablet.
  • KUDU-2580 - Fixed authentication token reacquisition in the C++ client.
  • Fixed an issue that caused the kudu CLI tool to unexpectedly exit when the connection to the master or tserver was abruptly closed.

Apache Oozie

  • OOZIE-2457 - Oozie log parsing regex consume more than 90% cpu
  • OOZIE-3193 - Applications are not killed when submitted via subworkflow
  • OOZIE-3354 - [core] [SSH action] SSH action gets hung
  • OOZIE-3370 - Property filtering is not consistent across job submission

Apache Sentry

  • SENTRY-1272 - Enable ALTERVIEW_RENAME and ALTERVIEW_AS operation in hive binding
  • SENTRY-2194 - Upgrade Sentry hadoop-version dependency to 2.7.5
  • SENTRY-2210 - AUTHZ_PATH should have index on the foreign key AUTHZ_OBJ_ID
  • SENTRY-2214 - Sentry should not allow URI grants to EMPTY or NULL locations
  • SENTRY-2219 - Create index AUTHZ_PATH_FK_IDX at table AUTHZ_PATH only when it does not exist for Oracle
  • SENTRY-2238 - Explicitly set Database on SentryHivePrivilegeObjectDesc
  • SENTRY-2299 - NPE In Sentry HDFS Sync Plugin
  • SENTRY-2310 - Sentry is not be able to fetch full update subsequently, when there is HMS restart in the snapshot process.
  • SENTRY-2332 - Load hadoop default configuration when starting sentry service
  • SENTRY-2333 - Create index AUTHZ_PATH_FK_IDX at table AUTHZ_PATH for Postgres only when it does not exist
  • SENTRY-2403 - Incorrect naming in RollingFileWithoutDeleteAppender
  • SENTRY-2406 - Make sure inputHierarchy and outputHierarchy have unique values

Apache Solr

  • SOLR-12290 - Do not close any servlet streams and improve our servlet stream closing prevention code for users and devs.
  • SOLR-12293 - Updates need to use their own connection pool to maintain connection reuse and prevent spurious recoveries.

Apache Spark

  • SPARK-22864 - [CORE] Disable allocation schedule in ExecutorAllocationManagerSuite.
  • SPARK-25253 - [PYSPARK] Refactor local connection & auth code
  • SPARK-25318 - Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

Apache Zookeeper

  • ZOOKEEPER-706 - Large numbers of watches can cause session re-establishment to fail
  • ZOOKEEPER-1382 - Zookeeper server holds onto dead/expired session ids in the watch data structures