Issues Fixed in CDH 5.11.x

The following topics describe issues fixed in CDH 5.11.x, from newest to oldest release. You can also review What's New in CDH 5.11.x or Known Issues in CDH 5.

Issues Fixed in CDH 5.11.2

CDH 5.11.2 fixes the following issues:

Upstream Issues Fixed

The following upstream issues are fixed in CDH 5.11.2:

  • FLUME-2752 - Fix AvroSource startup resource leaks
  • FLUME-2905 - Fix NetcatSource file descriptor leak if startup fails
  • HADOOP-12751 - While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple .
  • HADOOP-14141 - Store KMS SSL keystore password in catalina.properties
  • HADOOP-14242 - Make KMS Tomcat SSL property sslEnabledProtocols and clientAuth configurable
  • HADOOP-14511 - WritableRpcEngine.Invocation#toString NPE on null parameters
  • HDFS-6757 - Simplify lease manager with INodeID
  • HDFS-8856 - Make LeaseManager#countPath O(1).
  • HDFS-10220 - A large number of expired leases can make namenode unresponsive and cause failover
  • HDFS-10506 - OIV's ReverseXML processor cannot reconstruct some snapshot details
  • HDFS-11579 - Make HttpFS Tomcat SSL property sslEnabledProtocols and clientAuth configurable
  • HDFS-11708 - Positional read will fail if replicas moved to different DNs after stream is opened
  • HDFS-11741 - Long running balancer may fail due to expired DataEncryptionKey
  • HDFS-11861 - ipc.Client.Connection#sendRpcRequest should log request name
  • HDFS-11881 - NameNode consumes a lot of memory for snapshot diff report generation
  • HDFS-11960 - Successfully closed files can stay under-replicated
  • HDFS-12042 - Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption
  • HDFS-12139 - HTTPFS liststatus returns incorrect pathSuffix for path of file
  • YARN-2780 - Log aggregated resource allocation in rm-appsummary.log
  • YARN-6368 - Decommissioning an NM results in a -1 exit code
  • YARN-6615 - AmIpFilter drops query parameters on redirect
  • HBASE-15720 - Print row locks at the debug dump page
  • HBASE-15837 - Memstore size accounting is wrong if postBatchMutate() throws exception
  • HBASE-16033 - Add more details in logging of responseTooSlow/TooLarge
  • HBASE-16630 - Fragmentation in long running Bucket Cache
  • HBASE-16739 - Timed out exception message should include encoded region name
  • HBASE-16977 - VerifyReplication should log a printable representation of the row keys
  • HBASE-17131 - Avoid livelock caused by HRegion#processRowsWithLocks
  • HBASE-17501 - guard against NPE while reading FileTrailer and HFileBlock
  • HBASE-17587 - Do not Rethrow DoNotRetryIOException as UnknownScannerException
  • HBASE-17673 - Monitored RPC Handler not shown in the WebUI
  • HBASE-17688 - MultiRowRangeFilter not working correctly if given same start and stop RowKey
  • HBASE-17710 - HBase in standalone mode creates directories with 777 permission
  • HBASE-17731 - Fractional latency reporting in MultiThreadedAction
  • HBASE-17798 - RpcServer.Listener.Reader can abort due to CancelledKeyException
  • HBASE-17970 - Set yarn.app.mapreduce.am.staging-dir when starting MiniMRCluster
  • HBASE-18096 - Limit HFileUtil visibility and add missing annotations
  • HIVE-9567 - JSON SerDe not escaping special chars when writing char/varchar data
  • HIVE-10209 - FetchTask with VC may fail because ExecMapper.done is true
  • HIVE-11418 - Dropping a database in an encryption zone with CASCADE and trash enabled fails
  • HIVE-11592 - ORC metadata section can sometimes exceed protobuf message size limit
  • HIVE-11878 - ClassNotFoundException can possibly occur if multiple jars are registered one at a time in Hive
  • HIVE-12274 - Increase width of columns used for general configuration in the metastore.
  • HIVE-12551 - Fix several kryo exceptions in branch-1
  • HIVE-12762 - Common join on parquet tables returns incorrect result when hive.optimize.index.filter set to true
  • HIVE-13330 - ORC vectorized string dictionary reader does not differentiate null vs empty string dictionary
  • HIVE-13947 - HoS prints wrong number for hash table size in map join scenario
  • HIVE-14178 - Hive::needsToCopy should reuse FileUtils::equalsFileSystem
  • HIVE-14564 - Column Pruning generates out of order columns in SelectOperator which cause ArrayIndexOutOfBoundsException.
  • HIVE-15122 - Hive: Upcasting types should not obscure stats (min/max/ndv)
  • HIVE-16004 - OutOfMemory in SparkReduceRecordHandler with vectorization mode
  • HIVE-16060 - GenericUDTFJSONTuple's json cache could overgrow beyond its limit
  • HIVE-16291 - Hive fails when unions a parquet table with itself
  • HIVE-16413 - Create table as select does not check ownership of the location
  • HIVE-16559 - Parquet schema evolution for partitioned tables may break if table and partition serdes differ
  • HIVE-16593 - SparkClientFactory.stop may prevent JVM from exiting
  • HIVE-16647 - Improve the validation output to make the output to stderr and stdout more consistent
  • HIVE-16660 - Not able to add partition for views in hive when sentry is enabled
  • HIVE-16665 - Race condition in Utilities.GetInputPathsCallable --> createDummyFileForEmptyPartition
  • HIVE-16693 - beeline "source" command freezes if you have a comment in it.
  • HIVE-16697 - Schema table validator should return a sorted list of missing tables
  • HIVE-16869 - Hive returns wrong result when predicates on non-existing columns are pushed down to Parquet reader
  • HIVE-16930 - HoS should verify the value of Kerberos principal and keytab file before adding them to spark-submit command parameters
  • HIVE-16935 - Hive should strip comments from input before choosing which CommandProcessor to run.
  • HIVE-17050 - Multiline queries that have comment in middle fail when executed via "beeline -e"
  • HIVE-17052 - Remove logging of predicate filters
  • HIVE-17149 - Hdfs directory is not cleared if partition creation failed on HMS
  • HUE-5504 - [oozie] Only use JDBC URL from hive2 action when hardcoded
  • HUE-6370 - [jobsub] Attempting to create and save any JobSub fails
  • HUE-6398 - [jobsub] Fix loading edit design page
  • HUE-6407 - [pig] Play button doesn't come back after killing the running pig job
  • HUE-6446 - [oozie] User cant edit shared coordinator or bundle
  • HUE-6604 - [oozie] Fix timestamp conversion to server timezone
  • HUE-6694 - [aws] Gracefully handle bucket creation error for non-DNS compliant names
  • HUE-6695 - [aws] Provide user-friendly message when accessing forbidden paths
  • HUE-6696 - [aws] Display user friendly message when accessing non-DNS compliant bucket
  • HUE-6702 - [about] About page is accessible without authentication
  • HUE-6710 - [notebook] Application reachable directly by users without granted access
  • HUE-6791 - [search] Protect against pivot facets conflicting with nested facets
  • HUE-6813 - [metastore] Fix database comment last character truncation
  • HUE-6814 - [search] Only return distinct usernames in security impersonate dropdown
  • HUE-6819 - [oozie] Set generic widget for generic actions in oozie graph
  • HUE-6856 - [search] Protect against reflected XSS in search query parameters
  • HUE-6950 - [saml] Create home directory for new user login
  • HUE-6995 - [oozie] Set minimum width of a workflow node
  • IMPALA-4276 - Profile displays non-default query options set by planner
  • IMPALA-4546 - Fix Moscow timezone conversion after 2014
  • IMPALA-4631 - don't use floating point operations for time unit conversions
  • IMPALA-4716 - Expr rewrite causes IllegalStateException
  • IMPALA-4738 - STDDEV_SAMP should return NULL for single record input
  • IMPALA-4962 - Fix SHOW COLUMN STATS for HS2
  • IMPALA-5021 - Fix count(*) remaining rows overflow in Parquet.
  • IMPALA-5056 - Ensure analysis uses 'fresh' catalog after metadata loading
  • IMPALA-5154 - Handle 'unpartitioned' Kudu tables
  • IMPALA-5172 - Buffer overrun for Snappy decompression
  • IMPALA-5187 - Bump breakpad version to include the fix for Breakpad #681, re-enable the strict check that was disabled in IMPALA-3794.
  • IMPALA-5189 - Pin version of setuptools-scm
  • IMPALA-5197 - Erroneous corrupted Parquet file message
  • IMPALA-5198 - Error messages are sometimes dropped before reaching client
  • IMPALA-5217 - KuduTableSink checks null constraints incorrectly
  • IMPALA-5223 - Add waiting for HBase Zookeeper nodes to retry loop
  • IMPALA-5301 - Set Kudu minicluster memory limit
  • IMPALA-5318 - Generate access events with fully qualified table names
  • IMPALA-5355 - Fix the order of Sentry roles and privileges
  • IMPALA-5363 - Reset probe_batch_ after reaching limit
  • IMPALA-5419 - Check for cancellation when building hash tables
  • IMPALA-5469 - Fix exception when processing catalog update
  • IMPALA-5487 - Fix race in RuntimeProfile::toThrift()
  • IMPALA-5524 - Fixes NPE during planning with DISABLE_UNSAFE_SPILLS=1
  • IMPALA-5554 - sorter DCHECK on null column
  • IMPALA-5580 - fix Java UDFs that return NULL strings
  • IMPALA-5615 - Fix compute incremental stats for general partition exprs
  • IMPALA-5623 - Fix lag() on STRING cols to release UDF mem
  • IMPALA-5638 - Fix Kudu table set tblproperties inconsistencies
  • IMPALA-5657 - Fix a couple of bugs with FunctionCallExpr and IGNORE NULLS
  • IMPALA-5172 - fix incorrect cast in call to LZO decompress
  • KITE-1155 - Deleting an already deleted empty path should not fail the job
  • OOZIE-2816 - Strip out the first command word from Sqoop action if its "sqoop"
  • OOZIE-2844 - Increase stability of Oozie actions when log4j.properties is missing or not readable
  • OOZIE-2872 - Address backward compatibility issue introduced by OOZIE-2748
  • OOZIE-2908 - Fix typo in oozie.actions.null.args.allowed property in oozie-default.xml
  • OOZIE-2923 - Improve Spark options parsing
  • OOZIE-2984 - Parse spark-defaults.conf values with spaces without needing the quotes
  • PARQUET-389 - Support predicate push down on missing columns.
  • PIG-3567 - LogicalPlanPrinter throws OOM for large scripts
  • PIG-3655 - BinStorage and InterStorage approach to record markers is broken
  • SENTRY-1644 - Partition ACLs disappear after renaming Hive table with partitions.
  • SENTRY-1646 - Unable to truncate table <database>.<tablename>; from "default" databases
  • SENTRY-1759 - UpdatableCache leaks connections
  • SENTRY-1811 - Optimize data structures used in HDFS sync
  • SENTRY-1827 - Minimize TPathsDump thrift message used in HDFS sync
  • SOLR-6673 - MDC-based logging of collection, shard, etc.
  • SOLR-8836 - /update should return BAD REQUEST when invalid JSON provided.
  • SOLR-9153 - Update beanutils version to 1.9.2
  • SOLR-9527 - Solr RESTORE API doesn't distribute the replicas uniformly
  • SOLR-10076 - Hiding keystore and truststore passwords from /admin/info/* outputs.
  • SOLR-10889 - Stale zookeeper information is used during failover check.
  • SPARK-13278 - Launcher fails to start with JDK 9 EA
  • SPARK-15067 - YARN executors are launched with fixed perm gen size
  • SPARK-16845 - org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificOrdering `grows beyond 64 KB
  • SPARK-19019 - PySpark does not work with Python 3.6.0
  • SPARK-19688 - Spark on YARN Credentials File set to different application directory.
  • SPARK-20393 - Strengthen Spark to prevent XSS vulnerabilities
  • SPARK-20904 - Task failures during shutdown cause problems with preempted executors.
  • SPARK-20922 - Unsafe deserialization in Spark LauncherConnection.
  • ZOOKEEPER-1653 - zookeeper fails to start because of inconsistent epoch.
  • ZOOKEEPER-2040 - Server to log underlying cause of SASL connection problems.

Issues Fixed in CDH 5.11.1

CDH 5.11.1 fixes the following issues:

Upstream Issues Fixed

The following upstream issues are fixed in CDH 5.11.1:

  • FLUME-3080 - Close failure in HDFS Sink might cause data loss
  • FLUME-3085 - HDFS Sink can skip flushing some BucketWriters, might lead to data loss
  • HADOOP-11400 - GraphiteSink does not reconnect to Graphite after 'broken pipe'
  • HADOOP-11599 - Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured
  • HADOOP-12672 - RPC timeout should not override IPC ping interval
  • HADOOP-13503 - Improve SaslRpcClient failure logging
  • HADOOP-13926 - S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore
  • HADOOP-14019 - Fix some typos in the s3a docs
  • HADOOP-14028 - S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures
  • HADOOP-14051 - S3Guard: link docs from index, fix typos
  • HADOOP-14059 - typo in s3a rename(self, subdir) error message
  • HADOOP-14092 - Typo in hadoop-aws index.md
  • HADOOP-14104 - Client should always ask namenode for kms provider path
  • HADOOP-14144 - s3guard: CLI diff non-empty after import on new table
  • HADOOP-14172 - S3Guard: import does not import empty directory
  • HADOOP-14195 - CredentialProviderFactory$getProviders is not thread-safe
  • HADOOP-14204 - S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
  • HADOOP-14215 - DynamoDB client should waitForActive on existing tables
  • HADOOP-14236 - S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries
  • HADOOP-14255 - S3A to delete unnecessary fake directory objects in mkdirs()
  • HADOOP-14256 - [S3A DOC] Correct the format for "Seoul" example
  • HADOOP-14268 - Fix markdown itemization in hadoop-aws documents
  • HADOOP-14282 - S3Guard: DynamoDBMetadata::prune() should self interrupt correctly
  • HADOOP-14417 - Update default SSL cipher list for KMS
  • HDFS-10715 - NPE when applying AvailableSpaceBlockPlacementPolicy
  • HDFS-10797 - Revert "Disk usage summary of snapshots causes renamed blocks to get counted twice
  • HDFS-11499 - Decommissioning stuck because of failing recovery
  • HDFS-11515 - -du throws ConcurrentModificationException
  • HDFS-11689 - New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code
  • HDFS-11816 - Update default SSL cipher list for HttpFS
  • YARN-3251 - Fixed a deadlock in CapacityScheduler when computing absoluteMaxAvailableCapacity in LeafQueue
  • YARN-6042 - Dump scheduler and queue state information into FairScheduler DEBUG log.
  • YARN-6360 - Prevent FS state dump logger from cramming other log files
  • YARN-6432 - FairScheduler: Reserve preempted resources for corresponding applications.
  • YARN-6433 - Only accessible cgroup mount directories should be selected for a controller.
  • YARN-6453 - fairscheduler-statedump.log gets generated regardless of service
  • YARN-6500 - Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler.
  • HBASE-15125 - BackportHBaseFsck's adoptHdfsOrphan function creates region with wrong end key boundary
  • HBASE-15941 - HBCK repair should not unsplit healthy splitted region
  • HBASE-15955 - Disable action in CatalogJanitor#setEnabled should wait for active cleanup scan to finish
  • HBASE-16032 - Possible memory leak in StoreScanner
  • HBASE-16238 - It's useless to catch SESSIONEXPIRED exception and retry in RecoverableZooKeeper
  • HBASE-16350 - Undo server abort from HBASE-14968
  • HBASE-16429 - FSHLog: deadlock if rollWriter called when ring buffer filled with appends
  • HBASE-16663 - JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
  • HBASE-16721 - Concurrency issue in WAL unflushed seqId tracking
  • HBASE-17265 - Region left unassigned in master failover when region failed to open
  • HBASE-17328 - Properly dispose of looped replication peers
  • HBASE-17460 - enable_table_replication can not perform cyclic replication of a table
  • HBASE-17717 - Explicitly use "sasl" ACL scheme for hbase superuser
  • HBASE-17779 - disable_table_replication returns misleading message and does not turn off replication
  • HBASE-17792 - Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees
  • HIVE-11141 - Improve RuleRegExp when the Expression node stack gets huge
  • HIVE-11428 - Performance: Struct IN() clauses are extremely slow
  • HIVE-11671 - Optimize RuleRegExp in DPP codepath
  • HIVE-11842 - Improve RuleRegExp by caching some internal data structures
  • HIVE-12179 - Add option to not add spark-assembly.jar to Hive classpath
  • HIVE-12768 - Thread safety: binary sortable serde decimal deserialization
  • HIVE-14210 - ExecDriver should call jobclient.close() to trigger cleanup
  • HIVE-14380 - Queries on tables with remote HDFS paths fail in "encryption" checks.
  • HIVE-15282 - Different modification times are used when an index is built and when its staleness is checked
  • HIVE-15782 - query on parquet table returns incorrect result when hive.optimize.index.filter is set to true
  • HIVE-15879 - Fix HiveMetaStoreChecker.checkPartitionDirs method
  • HIVE-15997 - Resource leaks when query is cancelled
  • HIVE-16024 - MSCK Repair Requires nonstrict hive.mapred.mode
  • HIVE-16047 - Shouldn't try to get KeyProvider unless encryption is enabled
  • HIVE-16156 - FileSinkOperator should delete existing output target when renaming
  • HIVE-16175 - Possible race condition in InstanceCache
  • HIVE-16205 - Improving type safety in Objectstore
  • HIVE-16297 - Improving hive logging configuration variables
  • HIVE-16394 - HoS does not support queue name change in middle of session
  • HIVE-16459 - Forward channelInactive to RpcDispatcher
  • HIVE-16646 - Alias in transform ... as clause shouldn't be case sensitive
  • HUE-6109 - [core] Remove the restriction on Document2 invalid chars
  • HUE-6144 - [oozie] Add generic XSL template to workflow graph parser
  • HUE-6154 - [core] Ace paste event shouldn't reset the cursor position
  • HUE-6155 - [core] Remove table extender header throttling for Firefox
  • HUE-6158 - [autocomplete] The autocompleter eats characters to the right of the cursor on insert
  • HUE-6193 - [converter] Retain last_executed time when creating doc2 object
  • HUE-6197 - [impala] Fix XSS Vulnerability in the old editors' error messages
  • HUE-6207 - [editor] Avoid to always show the horizontal scroll bar when there's no scrolling needed
  • HUE-6208 - [core] The scroll left anchor should reset the horizontal scrollbar position
  • HUE-6212 - [oozie] Prevent XSS injection in coordinator cron frequency field
  • HUE-6223 - [autocomplete] Fix issue where tables from subsequent statements appear in the autocomplete results
  • HUE-6228 - [core] Disable touchscreen detection on Nicescroll
  • HUE-6251 - [editor] Log warnings and continue on failed bulk delete and copy actions
  • IMPALA-3641 - Fix catalogd RPC responses to DROP IF EXISTS.
  • IMPALA-4088 - Assign fix values to the minicluster server ports
  • IMPALA-4293 - query profile should include error log
  • IMPALA-4544 - ASAN should ignore SEGV and leaks
  • IMPALA-4615 - Fix create_table.sql command order
  • IMPALA-4733 - Change HBase ports to non-ephemeral
  • IMPALA-4787 - Optimize APPX_MEDIAN() memory usage
  • IMPALA-4822 - Implement dynamic log level changes
  • IMPALA-4899 - Fix parquet table writer dictionary leak
  • IMPALA-4902 - Copy parameters map in HdfsPartition.toThrift().
  • IMPALA-4998 - Fix missing table lock acquisition.
  • IMPALA-5028 - Lock table in /catalog_objects endpoint.
  • IMPALA-5055 - Fix DCHECK in parquet-column-readers.cc ReadPageHeader()
  • IMPALA-5088 - Fix heap buffer overflow
  • IMPALA-5115 - Handle status from HdfsTableSink::WriteClusteredRowBatch
  • IMPALA-5145 - Do not constant fold null in CastExprs
  • IMPALA-5156 - Drop VLOG level passed into Kudu client
  • IMPALA-5186 - Handle failed CreateAndOpenScanner() in MT scan.
  • IMPALA-5193 - Initialize decompressor before finding first tuple
  • IMPALA-5251 - Fix propagation of input exprs' types in 2-phase agg
  • IMPALA-5252 - Fix crash in HiveUdfCall::GetStringVal() when mem_limit exceeded
  • IMPALA-5253 - Use appropriate transport for StatestoreSubscriber
  • IMPALA-5322 - Fix a potential crash in Frontend & Catalog JNI startup
  • OOZIE-2739 - Remove property expansion pattern from ShellMain's log4j properties content
  • OOZIE-2818 - Can't overwrite oozie.action.max.output.data on a per-workflow basis
  • OOZIE-2819 - Make Oozie REST API accept multibyte characters for script Actions
  • SENTRY-1508 - Revert ""REVERT: MetastorePlugin.java does not handle properly initialization failure
  • SENTRY-1605 - SENTRY-1508 need to be fixed because of Kerberos initialization issue
  • SENTRY-1683 - MetastoreCacheInitializer has a race condition in handling results list
  • SENTRY-1714 - MetastorePlugin.java should quetly return from renameAuthzObject() when both paths are null
  • SOLR-9836 - Add ability to recover from leader when index corruption is detected on SolrCore creation.
  • SOLR-9848 - Lower solr.cloud.wait-for-updates-with-stale-state-pause back down from 7 seconds.
  • SOLR-10360 - Remove an extra space from Hadoop distcp cmd used by Solr backup/restore
  • SOLR-10430 - Add ls command to ZkCLI for listing sub-dirs
  • SPARK-14930 - [SPARK-13693] Fix race condition in CheckpointWriter.stop()
  • SPARK-19178 - [SQL][Backport-to-1.6] convert string of large numbers to int should return null
  • SPARK-19263 - DAGScheduler should avoid sending conflicting task set.
  • SPARK-19537 - Move pendingPartitions to ShuffleMapStage.
  • SPARK-20435 - [CORE] More thorough redaction of sensitive information
  • SQOOP-3123 - Introduce escaping logic for column mapping parameters (same what Sqoop already uses for the DB column names), thus special column names (e.g. containing '#' character) and mappings realted to those columns can be in the same format (thus not confusing the end users), and also eliminates the related AVRO format clashing issues.
  • SQOOP-3140 - Removing deprecated mapred.map.max.attempts, mapred.reduce.max.attempts entries and using the new constants directly from Hadoop instead
  • SQOOP-3159 - Sqoop (export + --table) with Oracle table_name having '$' fails with error

Issues Fixed in CDH 5.11.0

CDH 5.11.0 fixes the following issues.

Upstream Issues Fixed

The following upstream issues are fixed in CDH 5.11.0:

  • AVRO-1973 - Upgrade netty to the latest 3.x release
  • FLUME-2318 - Make SpoolingDirectorySource able to handle empty files
  • FLUME-2514 - Speed up TestFileChannelRestart tests
  • FLUME-2567 - Remove external Maven repositories
  • FLUME-2593 - ResettableFileInputStream returns negate values from read() method
  • FLUME-2619 - Spooldir source should log channel exceptions
  • FLUME-2704 - Configurable poll delay for spooling directory source
  • FLUME-2718 - HTTP Source to support generic Stream Handler
  • FLUME-2725 - HDFS Sink does not use configured timezone for rounding
  • FLUME-2761 - Move Hive sink out of preview mode
  • FLUME-2798 - Malformed Syslog messages can lead to OutOfMemoryException
  • FLUME-2801 - Performance improvement on TailDir source
  • FLUME-2806 - flume-ng.ps1 Error running script to start an agent on Windows
  • FLUME-2810 - Add static Schema URL to AvroEventSerializer configuration
  • FLUME-2812 - Fix semaphore leak causing java.lang.Error: Maximum permit count exceeded in MemoryChannel
  • FLUME-2835 - Hive Sink tests need to create table with transactional property set
  • FLUME-2857 - Make Kafka Source/Channel/Sink restore default values when live updating config
  • FLUME-2875 - Allow RollingFileSink to specify a file prefix and a file extension.
  • FLUME-2881 - Windows Launch Script fails in plugins dir code
  • FLUME-2889 - Fixes to DateTime computations
  • FLUME-2890 - Typo in Twitter source warning
  • FLUME-2911 - Add include pattern option in SpoolDir source
  • FLUME-2936 - KafkaSource tests: Use loopback instead of host address
  • FLUME-2937 - Integrate checkstyle for non-test classes
  • FLUME-2939 - Update recursive SpoolDir source to use Java 7 APIs
  • FLUME-2941 - Integrate checkstyle for test classes
  • FLUME-2949 - checkstyle: Make suppressions compatible with Windows
  • FLUME-2950 - ReliableSpoolingFileEventReader.rollCurrentFile is broken
  • FLUME-2952 - SyslogAgent: Avoid NPE on stop()
  • FLUME-2955 - Add file path to the header in TaildirSource
  • FLUME-2959 - Fix issues with flume-checkstyle module
  • FLUME-2966 - Fix NPE in JMS Source
  • FLUME-2971 - Add secure Kafka Sink/Source/Channel setup to the User Guide
  • FLUME-2974 - Fix some broken SpoolDirSource unit tests
  • FLUME-2998 - Add a missing configuration parameter to SequenceSource docs
  • FLUME-2999 - Kafka channel and sink should enable statically assigned partition per event via header
  • FLUME-3027 - Change Kafka Channel to clear offsets map after commit
  • FLUME-3049 - Make HDFS sink rotate more reliably in secure mode
  • HADOOP-10823 - TestReloadingX509TrustManager is flaky
  • HADOOP-11447 - Add a more meaningful toString method to SampleStat and MutableStat.
  • HADOOP-11619 - FTPFileSystem should override getDefaultPort.
  • HADOOP-11859 - PseudoAuthenticationHandler fails with httpcomponents v4.4
  • HADOOP-11878 - FileContext#fixRelativePart should check for not null for a more informative exception.
  • HADOOP-12009 - Clarify FileSystem.listStatus() sorting order & fix FileSystemContractBaseTest:testListStatus.
  • HADOOP-12655 - TestHttpServer.testBindAddress bind port range is wider than expected.
  • HADOOP-12667 - s3a to support createNonRecursive API
  • HADOOP-12774 - s3a should use UGI.getCurrentUser.getShortname() for username
  • HADOOP-12977 - s3a to handle delete("/", true) robustly
  • HADOOP-13037 - Refactor Azure Data Lake Store as an independent FileSystem
  • HADOOP-13119 - Add ability to secure log servlet using proxy users. Contribute by Yuanbo Liu.
  • HADOOP-13164 - Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories
  • HADOOP-13204 - Add support for SSE-KMS and SSE-C in s3a filesystem.
  • HADOOP-13207 - Specify FileSystem listStatus, listFiles and RemoteIterator
  • HADOOP-13257 - Improve Azure Data Lake contract tests
  • HADOOP-13309 - Document S3A known limitations in file ownership and permission model
  • HADOOP-13336 - S3A to support per-bucket configuration (S3Guard specific changes)
  • HADOOP-13345 - S3Guard: Ensure GenericOptionsParser is used for S3Guard CLI
  • HADOOP-13375 - o.a.h.security.TestGroupsCaching.testBackgroundRefreshCounters seems flaky.
  • HADOOP-13433 - Race in UGI.reloginFromKeytab
  • HADOOP-13446 - Support running isolated unit tests separate from AWS integration tests
  • HADOOP-13448 - S3Guard: Define MetadataStore interface.
  • HADOOP-13449 - S3Guard: Implement DynamoDBMetadataStore
  • HADOOP-13452 - S3Guard: Implement access policy for intra-client consistency with in-memory metadata store
  • HADOOP-13455 - S3Guard: Write end user docs, change table autocreate default.
  • HADOOP-13496 - Include file lengths in Mismatch in length error for distcp
  • HADOOP-13502 - Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests
  • HADOOP-13508 - FsPermission string constructor does not recognize sticky bit
  • HADOOP-13540 - improve section on troubleshooting s3a auth problems
  • HADOOP-13560 - S3ABlockOutputStream to support huge (many GB) file writes
  • HADOOP-13573 - S3Guard: create basic contract tests for MetadataStore implementations
  • HADOOP-13589 - S3Guard: Allow execution of all S3A integration tests with S3Guard enabled
  • HADOOP-13599 - s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown
  • HADOOP-13614 - Purge some superfluous/obsolete S3 FS tests that are slowing test runs down
  • HADOOP-13627 - Have an explicit KerberosAuthException for UGI to throw, text from public constants
  • HADOOP-13631 - S3Guard: implement move() for LocalMetadataStore, add unit tests
  • HADOOP-13650 - S3Guard: Provide command line tools to manipulate metadata store.
  • HADOOP-13651 - S3Guard: S3AFileSystem Integration with MetadataStore
  • HADOOP-13674 - S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint
  • HADOOP-13680 - fs.s3a.readahead.range to use getLongBytes
  • HADOOP-13692 - hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts
  • HADOOP-13727 - S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider
  • HADOOP-13735 - ITestS3AFileContextStatistics.testStatistics() failing
  • HADOOP-13742 - Expose NumOpenConnectionsPerUser as a metric.
  • HADOOP-13749 - KMSClientProvider combined with KeyProviderCache can result in wrong UGI being used
  • HADOOP-13782 - Make MutableRates metrics thread-local write, aggregate-on-read
  • HADOOP-13793 - S3guard: add inconsistency injection, integration tests
  • HADOOP-13801 - regression: ITestS3AMiniYarnCluster failing
  • HADOOP-13804 - MutableStat mean loses accuracy if add(long, long) is used
  • HADOOP-13805 - UGI.getCurrentUser() fails if user does not have a keytab associated
  • HADOOP-13823 - s3a rename: fail if dest file exists
  • HADOOP-13826 - S3A Deadlock in multipart copy due to thread pool limits
  • HADOOP-13830 - Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string"
  • HADOOP-13847 - KMSWebApp should close KeyProviderCryptoExtension
  • HADOOP-13850 - s3guard to log choice of metadata store at debug
  • HADOOP-13855 - Fix a couple of the s3a statistic names to be consistent with the rest
  • HADOOP-13857 - S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs
  • HADOOP-13876 - S3Guard: better support for multi-bucket access
  • HADOOP-13877 - S3Guard: fix TestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is set
  • HADOOP-13886 - s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure
  • HADOOP-13893 - dynamodb dependency -> compile
  • HADOOP-13899 - S3Guard: tune dynamodb client & tests
  • HADOOP-13900 - Remove snapshot version of SDK dependency from Azure Data Lake Store File System
  • HADOOP-13903 - Improvements to KMS logging to help debug authorization errors.
  • HADOOP-13904 - DynamoDBMetadataStore to handle DDB throttling
  • HADOOP-13908 - S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore
  • HADOOP-13911 - Remove TRUSTSTORE_PASSWORD related scripts from KMS
  • HADOOP-13914 - S3guard: Improve S3AFileStatus#isEmptyDirectory handling
  • HADOOP-13928 - TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error.
  • HADOOP-13929 - ADLS connector should not check in contract-test-options.xml.
  • HADOOP-13931 - S3AGuard: Use BatchWriteItem in DynamoDBMetadataStore#put()
  • HADOOP-13934 - S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to BatchWriteItem limits
  • HADOOP-13937 - S3Guard: Mock bucket locations in MockS3ClientFactory
  • HADOOP-13953 - Make FTPFileSystem's data connection mode and transfer mode configurable
  • HADOOP-13956 - Read ADLS credentials from Credential Provider.
  • HADOOP-13958 - Bump up release year to 2017
  • HADOOP-13960 - Initialize DynamoDBMetadataStore without associated S3AFileSystem.
  • HADOOP-13962 - Update ADLS SDK to 2.1.4.
  • HADOOP-13976 - Path globbing does not match newlines
  • HADOOP-13985 - s3guard: add a version marker to every table
  • HADOOP-13988 - KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser
  • HADOOP-13995 - s3guard cli: make tests easier to run and address failure
  • HADOOP-14001 - Improve delegation token validity checking.
  • HADOOP-14003 - Make additional KMS tomcat settings configurable
  • HADOOP-14013 - S3Guard: fix multi-bucket integration tests
  • HADOOP-14017 - User friendly name for ADLS user and group
  • HADOOP-14020 - Optimize dirListingUnion
  • HADOOP-14027 - S3Guard: Implicitly creating DynamoDB table ignores endpoint config
  • HADOOP-14029 - Fix KMSClientProvider for non-secure proxyuser use case
  • HADOOP-14040 - Use shaded aws-sdk uber-JAR 1.11.86
  • HADOOP-14041 - CLI command to prune old metadata
  • HADOOP-14044 - Synchronization issue in delegation token cancel functionality
  • HADOOP-14046 - Metastore destruction test creates table w/o version
  • HADOOP-14049 - Honour AclBit flag associated to file/folder permission for Azure datalake account
  • HADOOP-14050 - Add process name to kms process
  • HADOOP-14079 - s3guard: fix breaking link in s3guard.md
  • HADOOP-14083 - KMS should support old SSL clients.
  • HADOOP-14085 - s3guard: drop unnecessary type assertion and cast.
  • HADOOP-14094 - Rethink S3GuardTool options
  • HADOOP-14096 - s3guard: regression in dirListingUnion
  • HADOOP-14102 - Relax error message assertion in S3A test ITestS3AEncryptionSSEC
  • HADOOP-14110 - In S3AFileSystem, make getAmazonClient() package private; export getBucketLocation()
  • HADOOP-14113 - Review ADL Docs
  • HADOOP-14114 - S3A can no longer handle unencoded + in URIs
  • HADOOP-14125 - s3guard tool tests aren't isolated; can't run in parallel
  • HADOOP-14129 - ITestS3ACredentialsInURL sometimes fails
  • HADOOP-14130 - Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients
  • HADOOP-14131 - kms.sh creates bogus dir for tomcat logs
  • HADOOP-14168 - S3GuardTool tests should not run if S3Guard is not set up
  • HADOOP-14181 - S3Guard: Add validation of DynamoDB region
  • HDFS-3716 - Purger should remove stale fsimage ckpt files
  • HDFS-4366 - Block Replication Policy Implementation May Skip Higher-Priority Blocks for Lower-Priority Blocks.
  • HDFS-8307 - Spurious DNS Queries from hdfs shell
  • HDFS-8640 - Make reserved RBW space visible through JMX.
  • HDFS-8883 - NameNode Metrics : Add FSNameSystem lock Queue Length
  • HDFS-9145 - Tracking methods that hold FSNamesytemLock for too long
  • HDFS-9205 - Do not schedule corrupt blocks for replication.
  • HDFS-9390 - Block management for maintenance states.
  • HDFS-9391 - Update webUI/JMX to display maintenance state info.
  • HDFS-9467 - Fix data race accessing writeLockHeldTimeStamp in FSNamesystem
  • HDFS-10292 - Add block id when client got Unable to close file exception
  • HDFS-10534 - NameNode WebUI should display DataNode usage histogram
  • HDFS-10559 - DiskBalancer: Use SHA1 for Plan ID
  • HDFS-10713 - Throttle FsNameSystem lock warnings
  • HDFS-10733 - NameNode terminated after full GC thinking QJM is unresponsive
  • HDFS-10798 - Make the threshold of reporting FSNamesystem lock contention configurable
  • HDFS-10808 - DiskBalancer does not execute multi-steps plan-redux
  • HDFS-10817 - Add Logging for Long-held NN Read Locks
  • HDFS-10896 - Move lock logging logic from FSNamesystem into FSNamesystemLock
  • HDFS-10941 - Improve BlockManager#processMisReplicatesAsync log
  • HDFS-10966 - Enhance Dispatcher logic on deciding when to give up a source DataNode
  • HDFS-11087 - NamenodeFsck should check if the output writer is still writable
  • HDFS-11132 - Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing files
  • HDFS-11160 - VolumeScanner reports write-in-progress replicas as corrupt incorrectly
  • HDFS-11180 - Intermittent deadlock in NameNode when failover happens.
  • HDFS-11197 - Listing encryption zones fails when deleting a EZ that is on a snapshotted directory
  • HDFS-11259 - Update fsck to display maintenance state info.
  • HDFS-11263 - ClassCastException when we use Bzipcodec for Fsimage compression
  • HDFS-11265 - Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI.
  • HDFS-11271 - Typo in NameNode UI
  • HDFS-11275 - Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
  • HDFS-11280 - Allow WebHDFS to reuse HTTP connections to NN
  • HDFS-11292 - log lastWrittenTxId etc info in logSyncAll
  • HDFS-11296 - Maintenance state expiry should be an epoch time and not jvm monotonic.
  • HDFS-11306 - Print remaining edit logs from buffer if edit log can't be rolled
  • HDFS-11363 - Need more diagnosis info when seeing Slow waitForAckedSeqno.
  • HDFS-11378 - Verify multiple DataNodes can be decommissioned/maintenance at the same time.
  • HDFS-11379 - DFSInputStream may infinite loop requesting block locations
  • HDFS-11390 - Add process name to httpfs process
  • HDFS-11411 - Avoid OutOfMemoryError in TestMaintenanceState test runs.
  • HDFS-11412 - Maintenance minimum replication config value allowable range should be [0, DefaultReplication].
  • HDFS-11418 - HttpFS should support old SSL clients
  • HDFS-11441 - Add escaping to error message in KMS web UI
  • HDFS-11466 - Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms
  • MAPREDUCE-5155 - Race condition in test case TestFetchFailure cause it to fail
  • MAPREDUCE-5335 - Rename Job Tracker terminology in ShuffleSchedulerImpl
  • MAPREDUCE-5883 - "Total megabyte-seconds" in job counters is slightly misleading
  • MAPREDUCE-5981 - Log levels of certain MR logs can be changed to DEBUG
  • MAPREDUCE-6172 - TestDbClasses timeouts are too aggressive
  • MAPREDUCE-6571 - JobEndNotification info logs are missing in AM container syslog
  • MAPREDUCE-6639 - Process hangs in LocatedFileStatusFetcher if FileSystem.get throws.
  • MAPREDUCE-6715 - Fix Several Unsafe Practices
  • MAPREDUCE-6817 - The format of job start time in JHS is different from those of submit and finish time.
  • MAPREDUCE-6829 - Add peak memory usage counter for each task.
  • YARN-1047 - Expose # of pre-emptions as a queue counter
  • YARN-2301 - Improved yarn container command
  • YARN-2306 - Add test for leakage of reservation metrics in fair scheduler.
  • YARN-2336 - Fair scheduler's REST API returns a missing '[' bracket JSON for deep queue tree
  • YARN-2902 - Addendum. Killing a container that is localizing can orphan resources in the DOWNLOADING state
  • YARN-2988 - Graph#save() may leak file descriptors.
  • YARN-3269 - Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path
  • YARN-3396 - Handle URISyntaxException in ResourceLocalizationService.
  • YARN-3933 - FairScheduler: Multiple calls to completedContainer are not safe.
  • YARN-3957 - FairScheduler NPE In FairSchedulerQueueInfo causing scheduler page to return 500.
  • YARN-4232 - TopCLI console support for HA mode
  • YARN-4363 - In TestFairScheduler, testcase should not create FairScheduler redundantly.
  • YARN-4544 - All the log messages about rolling monitoring interval are shown with WARN level
  • YARN-4555 - TestDefaultContainerExecutor#testContainerLaunchError fails on non-english locale environment
  • YARN-4719 - Add a helper library to maintain node state and allows common queries.
  • YARN-4752 - Improved preemption in FairScheduler.
  • YARN-4805 - Don't go through all schedulers in ParameterizedTestBase.
  • YARN-4882 - Change the log level to DEBUG for recovering completed applications
  • YARN-5136 - Error in handling event type APP_ATTEMPT_REMOVED to the scheduler
  • YARN-5181 - ClusterNodeTracker: add method to get list of nodes matching a specific resourceName.
  • YARN-5182 - MockNodes.newNodes creates one more node per rack than requested.
  • YARN-5257 - Fix unreleased resources and null dereferences
  • YARN-5308 - FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling
  • YARN-5752 - TestLocalResourcesTrackerImpl#testLocalResourceCache times out
  • YARN-5798 - Set UncaughtExceptionHandler for all FairScheduler threads.
  • YARN-5830 - FairScheduler: Avoid preempting AM containers.
  • YARN-5859 - TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails
  • YARN-5890 - FairScheduler should log information about AM-resource-usage and max-AM-share for queues
  • YARN-5920 - Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang
  • YARN-6061 - Add an UncaughtExceptionHandler for critical threads in RM.
  • YARN-6112 - UpdateCallDuration is calculated only when debug logging is enabled.
  • YARN-6125 - The application attempt's diagnostic message should have a maximum size
  • YARN-6131 - FairScheduler: Lower update interval for faster tests.
  • YARN-6144 - FairScheduler: preempted resources can become negative.
  • YARN-6163 - FS Preemption is a trickle for severely starved applications.
  • YARN-6171 - ConcurrentModificationException on FSAppAttempt.containersToPreempt.
  • YARN-6172 - FSLeafQueue demand update needs to be atomic.
  • YARN-6193 - FairScheduler might not trigger preemption when using DRF.
  • YARN-6194 - Cluster capacity in SchedulingPolicy is updated only on allocation file reload.
  • YARN-6210 - FairScheduler: Node reservations can interfere with preemption.
  • YARN-6215 - Custom Cloudera version. FairScheduler preemption and update should not run concurrently.
  • YARN-6218 - Fix TestAMRMClient when using FairScheduler.
  • YARN-6264 - AM not launched when a single vcore is available on the cluster.
  • YARN-6448 - Continuous scheduling thread crashes while sorting nodes.
  • HBASE-6721 - RegionServer Group based Assignment. Adds new rsgroups feature which allows isol
  • HBASE-12949 - Scanner can be stuck in infinite loop if the HFile is corrupted
  • HBASE-15295 - MutateTableAccess.multiMutate() does not get high priority causing a deadlock
  • HBASE-15315 - Remove always set super user call as high priority
  • HBASE-15322 - Operations using Unsafe path broken for platforms not having sun.misc.Unsafe.
  • HBASE-15325 - ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests
  • HBASE-15328 - sanity check the redirect used to send master info requests to the embedded regionserver.
  • HBASE-15378 - Scanner cannot handle heartbeat message with no results
  • HBASE-15435 - Add WAL (in bytes) written metric
  • HBASE-15441 - Fix WAL splitting when region has moved multiple times
  • HBASE-15587 - FSTableDescriptors.getDescriptor() logs stack trace erronously
  • HBASE-15645 - hbase.rpc.timeout is not used in operations of HTable
  • HBASE-15931 - Add log for long-running tasks in AsyncProcess
  • HBASE-16062 - Improper error handling in WAL Reader/Writer creation
  • HBASE-16165 - Decrease RpcServer.callQueueSize before writeResponse causes OOM
  • HBASE-16237 - Blocks for hbase:meta table are not cached in L1 cache
  • HBASE-16266 - Do not throw ScannerTimeoutException when catch UnknownScannerException
  • HBASE-16304 - HRegion#RegionScannerImpl#handleFileNotFoundException may lead to deadlock when trying to obtain write lock on updatesLock
  • HBASE-16460 - Can't rebuild the BucketAllocator's data structures when BucketCache uses FileIOEngine
  • HBASE-16527 - IOExceptions from DFS client still can cause CatalogJanitor to delete referenced files
  • HBASE-16604 - Scanner retries on IOException can cause the scans to miss data
  • HBASE-16649 - Truncate table with splits preserved can cause both data loss and truncated data appeared again
  • HBASE-16662 - Fix open POODLE vulnerabilities
  • HBASE-16754 - All WALSplitter OutputSinks should process compaction events
  • HBASE-16765 - backportNew SteppingRegionSplitPolicy, avoid too aggressive spread of regions for small tables. Includesand the amendmentAmend: Make SteppingSplitPolicy the default.
  • HBASE-16807 - , RegionServer will fail to report new active Hmaster until HMaster/RegionServer failover.
  • HBASE-16841 - Data loss in MOB files after cloning a snapshot and deleting that snapshot
  • HBASE-16931 - Setting cell's seqId to zero in compaction flow might cause RS down.
  • HBASE-16960 - RegionServer hang when aborting
  • HBASE-16981 - Expand Mob Compaction Partition policy from daily to weekly, monthly
  • HBASE-17020 - keylen in midkey() dont computed correctly
  • HBASE-17023 - Region left unassigned due to AM and SSH each thinking others would do the assignment work
  • HBASE-17044 - Fix merge failed before creating merged region leaves meta inconsistent
  • HBASE-17069 - RegionServer writes invalid META entries for split daughters in some circumstances
  • HBASE-17206 - FSHLog may roll a new writer successfully with unflushed entries
  • HBASE-17238 - Wrong in-memory hbase:meta location causing SSH failure
  • HBASE-17241 - Avoid compacting already compacted mob files with _del files
  • HBASE-17243 - Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects
  • HBASE-17243 - Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects
  • HBASE-17275 - Assign timeout may cause region to be unassigned forever
  • HBASE-17309 - Fix connection leaks in TestAcidGuarantees
  • HBASE-17381 - ReplicationSourceWorkerThread can die due to unhandled exceptions
  • HBASE-17409 - Limit jsonp callback name to prevent xss
  • HBASE-17437 - Support specifying a WAL directory outside of the root directory
  • HBASE-17452 - Failed taking snapshot - region Manifest proto-message too large
  • HBASE-17522 - Handle JVM throwing runtime exceptions when we ask for details on heap usage the same as a correctly returned 'undefined'.
  • HBASE-17558 - ZK dumping jsp should escape HTML.
  • HBASE-17561 - table status page should escape values that may contain arbitrary characters.
  • HBASE-17624 - ADDENDUMincluding missing parts of TestWALObserver changes from
  • HBASE-17640 - Unittest error in TestMobCompactor with different timezone
  • HBASE-17653 - HBASE-17624 rsgroup synchronizations will (distributed) deadlock
  • HBASE-17654 - RSGroup refactoring.
  • HBASE-17675 - ReplicationEndpoint should choose new sinks if a SaslException occurs
  • HBASE-17677 - ServerName parsing from directory name should be more robust to errors from guava's HostAndPort
  • HIVE-7723 - Explain plan for complex query with lots of partitions is slow due to in-efficient collection used to find a matching ReadEntity
  • HIVE-9138 - Add some explain to PTF operator
  • HIVE-9481 - allow column list specification in INSERT statement
  • HIVE-10641 - create CRC32 UDF
  • HIVE-11147 - MetaTool doesn't update FS root location for partitions with space in name
  • HIVE-11293 - HiveConnection.setAutoCommit(true) throws exception
  • HIVE-11376 - Removes legacy code wrt skipping files with codec to HiveInputFormat from CombineHiveInputFormat
  • HIVE-11594 - Analyze Table for column names with embedded spaces
  • HIVE-11849 - NPE in HiveHBaseTableShapshotInputFormat in query with just count(*)
  • HIVE-11882 - Fetch optimizer should stop source files traversal once it exceeds the hive.fetch.task.conversion.threshold
  • HIVE-11920 - ADD JAR failing with URL schemes other than file/ivy/hdfs
  • HIVE-12222 - Define port range in property for RPCServer
  • HIVE-12236 - Enable SimpleFetchOptimizer for more query types
  • HIVE-12261 - schematool version info exit status should depend on compatibility, not equality
  • HIVE-12349 - NPE in ORC SARG for IS NULL queries on Timestamp and Date columns
  • HIVE-12465 - Hive might produce wrong results when (outer) joins are merged
  • HIVE-12780 - Fix the output of the history command in Beeline
  • HIVE-12789 - Fix output twice in the history command of Beeline
  • HIVE-12897 - Improve dynamic partition loading
  • HIVE-12976 - MetaStoreDirectSql doesn't batch IN lists in all cases
  • HIVE-13149 - Remove some unnecessary HMS connections from HS2
  • HIVE-13179 - Allow custom HiveConf to be passed to Authentication Providers
  • HIVE-13240 - GroupByOperator: Drop the hash aggregates when closing operator
  • HIVE-13463 - Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
  • HIVE-13510 - Dynamic partitioning doesn’t work when remote metastore is used
  • HIVE-13726 - Improve dynamic partition loading VI
  • HIVE-13864 - Beeline ignores the command that follows a semicolon and comment
  • HIVE-13866 - flatten callstack for directSQL errors
  • HIVE-13895 - HoS start-up overhead in yarn-client mode
  • HIVE-13901 - Hivemetastore add partitions can be slow depending on filesystems
  • HIVE-13902 - [Refactor] Minimize metastore jar dependencies on task nodes
  • HIVE-13904 - Ignore case when retrieving ColumnInfo from RowResolver
  • HIVE-13966 - DbNotificationListener: can loose DDL operation notifications
  • HIVE-14204 - Optimize loading dynamic partitions
  • HIVE-14423 - S3: Fetching partition sizes from FS can be expensive when stats are not available in metastore
  • HIVE-14511 - Improve MSCK for partitioned table to deal with special cases Updated changed q.out
  • HIVE-14693 - Some paritions will be left out when partition number is the multiple of the option hive.msck.repair.batch.size
  • HIVE-14713 - LDAP Authentication Provider should be covered with unit tests
  • HIVE-14764 - Enabling "hive.metastore.metrics.enabled" throws OOM in HiveMetastore
  • HIVE-14803 - S3: Stats gathering for insert queries can be expensive for partitioned dataset
  • HIVE-14819 - FunctionInfo for permanent functions shows TEMPORARY FunctionType
  • HIVE-14820 - RPC server for spark inside HS2 is not getting server address properly
  • HIVE-14873 - Add UDF for extraction of 'day of week'
  • HIVE-14920 - S3: Optimize SimpleFetchOptimizer::checkThreshold()
  • HIVE-15025 - Secure-Socket-Layer (SSL) support for HMS
  • HIVE-15060 - Remove the autoCommit warning from beeline
  • HIVE-15065 - SimpleFetchOptimizer should decide based on metastore stats when available
  • HIVE-15137 - metastore add partitions background thread should use current username
  • HIVE-15207 - Implement a capability to detect incorrect sequence numbers
  • HIVE-15072 - Schematool should recognize missing tables in metastore
  • HIVE-15073 - Schematool should detect malformed URIs
  • HIVE-12261 - Schematool version info exit status should depend on compatibility, not equality
  • HIVE-15074 - Schematool provides a way to detect invalid entries in VERSION table
  • HIVE-15263 - Detect the values for incorrect NULL values
  • HIVE-15391 - Location validation for table should ignore the values for view
  • HIVE-15392 - Refactoring the validate function of HiveSchemaTool to make the output consistent
  • HIVE-15232 - Add notification events for functions and indexes
  • HIVE-15338 - Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP
  • HIVE-15346 - "values temp table" should not be an input
  • HIVE-15359 - CDH=48042skip.footer.line.count doesnt work properly for certain situations
  • HIVE-15410 - WebHCat supports get/set table property with its name containing period and hyphen
  • HIVE-15430 - Change SchemaTool table validator to test based on the dbType
  • HIVE-15485 - Investigate the DoAs failure in HoS
  • HIVE-15517 - NOT (x <=> y) returns NULL if x or y is NULL
  • HIVE-15530 - Optimize the column stats update logic in table alteration
  • HIVE-15546 - Optimize Utilities.getInputPaths() so each listStatus of a partition is done in parallel
  • HIVE-15550 - fix arglist logging in schematool
  • HIVE-15551 - memory leak in directsql for mysql+bonecp specific initialization
  • HIVE-15572 - Improve the response time for query canceling when it happens during acquiring locks
  • HIVE-15735 - In some cases, view objects inside a view do not have parents.
  • HIVE-15754 - exchange partition is not generating notifications
  • HIVE-15766 - DBNotificationlistener leaks JDOPersistenceManager : Use same PersistenceManager for metadata and notification
  • HIVE-15778 - DROP INDEX (non-existent) throws NPE when using DbNotificationListener
  • HIVE-15803 - msck can hang when nested partitions are present
  • HIVE-15805 - Some minor improvement on the validation tool
  • HIVE-15815 - Allow to pass some Oozie properties to Spark in HoS
  • HIVE-15872 - The PERCENTILE_APPROX UDAF does not work with empty set
  • HIVE-15879 - Fix HiveMetaStoreChecker.checkPartitionDirs method
  • HIVE-15881 - Use hive.exec.input.listing.max.threads variable name instead of mapred.dfsclient.parallelism.max
  • HIVE-15895 - Use HDFS for stats collection temp dir on blob storage
  • HIVE-16014 - HiveMetastoreChecker should use hive.metastore.fshandler.threads instead of hive.mv.files.thread for pool size
  • HIVE-16019 - Query fails when group by/order by on same column with uppercase name
  • HIVE-16086 - Fix HiveMetaStoreChecker.checkPartitionDirsSingleThreaded method
  • HIVE-16176 - Fixing the non-zero exit code issue with HiveSchemaTool
  • HUE-5167 - Improve SASL error reporting in HUE
  • HUE-5482 - [home] Handle multiple home/trash directories by merging them into one
  • HUE-5594 - [editor] The new autocomplete dropdown should include popular group by and order by clauses
  • HUE-5659 - [home] Ignore history dependencies when importing document from different cluster
  • HUE-5701 - [security] Remove last references to $.cookie and migrate to $.totalStorage from FB
  • IMPALA-1430 - IMPALA-4878,IMPALA-4879: codegen native UDAs
  • IMPALA-2020 - 4915, 4936: Add rounding for decimal casts
  • IMPALA-2020 - IMPALA-4809: Codegen support for DECIMAL_V2
  • IMPALA-2605 - Omit the sort and mini stress tests
  • IMPALA-4055 - Speed up to_date() with custom implementation.
  • IMPALA-4263 - Fix wrong ommission of agg/analytic hash exchanges.
  • IMPALA-4282 - Remove max length check for type strings.
  • IMPALA-4370 - Divide and modulo result types for DECIMAL version V2
  • IMPALA-4449 - Revisit table locking pattern in the catalog
  • IMPALA-4675 - Case-insensitive matching of Parquet fields.
  • IMPALA-4702 - Fix command line help for webserver_private_key_file
  • IMPALA-4705 - IMPALA-4779, IMPALA-4780: Fix some Expr bugs with codegen
  • IMPALA-4725 - Query option to control Parquet array resolution.
  • IMPALA-4729 - Implement REPLACE()
  • IMPALA-4742 - Change "{}".format() to "{0}".format() for Py 2.6
  • IMPALA-4749 - hit DCHECK in sorter with scratch limit
  • IMPALA-4767 - Workaround for HIVE-15653 to preserve table stats.
  • IMPALA-4808 - old hash join can reference invalid memory
  • IMPALA-4809 - Enable support for DECIMAL_V2 in decimal_casting.py
  • IMPALA-4810 - fix incorrect expr-test decimal types
  • IMPALA-4810 - Make DECIMAL expr-test cases table driven
  • IMPALA-4810 - Add DECIMAL_V2 query option
  • IMPALA-4813 - Round on divide and multiply
  • IMPALA-4821 - Update AVG() for DECIMAL_V2
  • IMPALA-4828 - Alter Kudu schema outside Impala may crash on read
  • IMPALA-4854 - Fix incremental stats with complex types.
  • IMPALA-4916 - Fix maintenance of set of item sets in DisjointSet.
  • IMPALA-4929 - Safe concurrent access to IR function call graph
  • IMPALA-4933 - IMPALA-4931: Simplify SSL initialization on startup
  • IMPALA-4934 - Disable Kudu OpenSSL initialization
  • IMPALA-4981 - Re-enable spilling with MT_DOP.
  • IMPALA-4995 - Fix integer overflow in TopNNode::PrepareForOutput
  • IMPALA-4997 - Fix overflows in Sorter::TupleIterator
  • IMPALA-5005 - Don't allow server to send SASL COMPLETE msg out of order
  • IMPALA-5025 - Update binutils to 2.26.1
  • IMPALA-5027 - make udf headers buildable externally
  • OOZIE-2243 - Kill Command does not kill the child job for java action
  • OOZIE-2519 - Oozie HA with SSL info is slightly incorrect
  • OOZIE-2584 - Eliminate Thread.sleep() calls in TestMemoryLocks
  • OOZIE-2659 - TestPauseTransitService is flaky
  • OOZIE-2720 - Test failure - TestCoordMaterializeTriggerService#testMaxMatThrottleNotPicked
  • OOZIE-2742 - Unable to kill applications based on tag
  • OOZIE-2748 - NPE in LauncherMapper.printArgs()
  • OOZIE-2757 - Malformed xml is Spark action doc page
  • OOZIE-2777 - Config-default.xml longer than 64k results in java.io.UTFDataFormatException
  • OOZIE-2786 - Pass Oozie workflow ID and settings to Spark application configuration
  • OOZIE-2787 - Oozie distributes application jar twice making the spark job fail
  • OOZIE-2792 - Hive2 action is not parsing Spark application ID from log file properly when Hive is on Spark
  • OOZIE-2802 - Spark action failure on Spark 2.1.0 due to duplicate sharelibs
  • OOZIE-2803 - Mask passwords when printing out configs/args in MapReduceMain and SparkMain
  • OOZIE-2811 - Add support for filtering out properties from SparkConfigurationService
  • SENTRY-1476 - SentryStore is subject to JDQL injection. Sentry is subject to JDOQL injection
  • SENTRY-1504 - NPE in log4j.properties parsingSentryMain trips over missing log4j property
  • SENTRY-1505 - CommitContext isn't used by anything and should be removed
  • SENTRY-1507 - Sentry should use Datanucleus version of javax.jdo
  • SENTRY-1508 - MetastorePlugin.java does not handle properly initialization failure
  • SENTRY-1515 - Cleanup exception handling in SentryStore
  • SENTRY-1517 - SentryStore should actually use function getMSentryRole to get roles
  • SENTRY-1518 - Add metrics for SentryStore transactions
  • SENTRY-1520 - Provide mechanism for triggering HMS full snapshot
  • SENTRY-1533 - Sentry console metrics reporting interval should be configurable
  • SENTRY-1534 - Oracle supports serializable instead of repeatable-read
  • SENTRY-1557 - getRolesForGroups(),getRoleNamesForGroups() does too many trips to the DB
  • SENTRY-1564 - Improve error detection and reporting in MetastoreCacheInitializer.java
  • SENTRY-1577 - Support "create function using jar" for hive when Sentry is enabled
  • SENTRY-1581 - Provide Log4J metrics reporter
  • SENTRY-1582 - Additional comments to clarify the intent of string manipulation methods in SentryStore.java
  • SENTRY-1594 - TransactionBlock should become generic
  • SENTRY-1599 - CloseablePersistenceManager is no longer needed
  • SENTRY-1609 - DelegateSentryStore is subject to JDQL injection
  • SENTRY-1615 - SentryStore should not allocate empty objects that are immediately returned
  • SENTRY-1625 - PrivilegeOperatePersistence can use QueryParamBuilder Fix [SQL Injection: JDO] in [sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/provider/db/generic/service/persistent/PrivilegeOperatePersistence.java]
  • SENTRY-1636 - Remove thrift dependency on fb303
  • SOLR-9284 - The HDFS BlockDirectoryCache should not let it's keysToRelease or names maps grow indefinitely.
  • SOLR-9330 - Fix AlreadyClosedException on admin/mbeans?stats=true
  • SOLR-9389 - HDFS Transaction logs stay open for writes which leaks Xceivers.
  • SOLR-9506 - cache IndexFingerprint for each segment
  • SOLR-9699, SOLR-4668 - fix exception from core status in parallel with core reload
  • SOLR-9819 - Upgrade Apache commons-fileupload to 1.3.2, fixing a security vulnerability
  • SOLR-9859 - backport of replication.properties cannot be updated after being written and neither replication.properties nor index.properties are durable after a crash. Don't log error on NoSuchFileException
  • SOLR-9901 - backport of SOLR-9899 Implement move in HdfsDirectoryFactory. SOLR-9899: StandardDirectoryFactory should use optimizations for all FilterDirectorys not just NRTCachingDirectory.
  • SOLR-10031 - Validation of filename params in ReplicationHandler
  • SOLR-10114, SOLR-9941 - Reordered delete-by-query can delete or omit child documents
  • SOLR-10119 - TestReplicationHandler assertion fixes part of
  • SOLR-10121, SOLR-10116 - BlockCache corruption with high concurrency
  • SPARK-4105 - retry the fetch or stage if shuffle block is corrupt
  • SPARK-12241 - [YARN] Improve failure reporting in Yarn client obtainTokenForHBase()
  • SPARK-12523 - [YARN] Support long-running of the Spark On HBase and hive meta store.
  • SPARK-16654 - [CORE] Add UI coverage for Application Level Blacklisting. Automatically Kill Executors and Nodes when they are Blacklisted
  • SPARK-17874 - [CORE] Add SSL port configuration.
  • SPARK-18372 - [SQL][BRANCH-1.6] Staging directory fail to be removed
  • SPARK-18750 - [YARN] Avoid using "mapValues" when allocating containers.
  • SPARK-19033 - [CORE] Add admin acls for history server
  • SPARK-19220 - [UI] Make redirection to HTTPS apply to all URIs.
  • SPARK-19554 - [UI,YARN] Allow SHS URL to be used for tracking in YARN RM.
  • SPARK-19652 - [UI] Do auth checks for REST API access.
  • SPARK-19720 - [CORE] Redact sensitive information from SparkSubmit console
  • SQOOP-2349 - Add command line option for setting transaction isolation levels for metadata queries
  • SQOOP-2896 - Sqoop exec job fails with SQLException Access denied for user
  • SQOOP-2909 - Oracle related ImportTest fails after SQOOP-2737
  • SQOOP-2911 - Fix failing HCatalogExportTest caused by SQOOP-2863
  • SQOOP-2950 - Sqoop trunk has consistent UT failures - need fixing
  • SQOOP-3053 - Create a cmd line argument for sqoop.throwOnError and use it through SqoopOptions
  • SQOOP-3055 - Fixing MySQL tests failing due to ignored test inputs/configuration
  • SQOOP-3057 - Fixing 3rd party Oracle tests failing due to invalid case of column names
  • SQOOP-3066 - Introduce an option + env variable to enable/disable SQOOP-2737 feature
  • SQOOP-3068 - Enhance error (tool.ImportTool: Encountered IOException running import job: java.io.IOException: Expected schema) to suggest workaround
  • SQOOP-3069 - Get OracleExportTest#testUpsertTestExport in line with SQOOP-3066
  • SQOOP-3071 - Fix OracleManager to apply localTimeZone correctly in case of Date objects too
  • SQOOP-3072 - Reenable escaping in ImportTest#testProductWithWhiteSpaceImport for proper execution
  • SQOOP-3081 - use OracleEscapeUtils.escapeIdentifier in OracleUpsertOutputFormat instead of inline appending quotes
  • SQOOP-3124 - Fix ordering in column list query of PostgreSQL connector to reflect the logical order instead of adhoc ordering
  • ZOOKEEPER-1045 - Backport — Support Quorum Peer mutual authentication via SASL
  • ZOOKEEPER-2044 - Backport — Fix CancelledKeyException on ZooKeeper server (zookeeper branch-3.4)

Cloudera Search

Security vulnerability resolved

An attacker can download any file accessible to Solr superuser using the path manipulation vulnerability in Solr (CVE-2017-3163; see Cloudera Security Bulletins more information). This release provides the fix for it. In addition, attackers can leverage this vulnerability to manipulate files unrelated to Solr index, by launching a “man in the middle” attack between individual Solr servers. To avoid this issue, enable TLS and Kerberos authentication.