Your browser is out of date!

Update your browser to view this website correctly. Update my browser now

×

Please Read and Accept our Terms


Long term component architecture

As the main curator of open standards in Hadoop, Cloudera has a track record of bringing new open source solutions into its platform (such as Apache Spark, Apache HBase, and Apache Parquet) that are eventually adopted by the community at large. As standards, you can build longterm architecture on these components with confidence.

 

PLEASE NOTE:

With the exception of DSSD support, Cloudera Enterprise 5.6.0 is identical to CDH 5.5.2/Cloudera Manager 5.5.3  If you do not need DSSD support, you do not need to upgrade if you are already using the latest 5.5.x release.

 

Note: Mixed operating system type and version clusters are supported, however using the same version of the same operating system on all cluster hosts is strongly recommended.

CDH 5 provides 64-bit packages for RHEL-compatible, SLES, Ubuntu, and Debian systems as listed below.

 

Operating System Version Packages
Red Hat Enterprise Linux (RHEL)-compatible
RHEL (+ SELinux mode in available versions) 5.7 64-bit
  5.10 64-bit
  6.4 64-bit
  6.5 64-bit
  6.6 64-bit
  6.7 64-bit
  7.1 64-bit
  7.2 64-bit
CentOS (+ SELinux mode in available versions) 5.7 64-bit
  5.10 64-bit
  6.4 64-bit
  6.5 64-bit
  6.6 64-bit
  6.7 64-bit
  7.1 64-bit
  7.2 64-bit
Oracle Linux with default kernel and Unbreakable Enterprise Kernel 5.7 (UEK R2) 64-bit
  5.10 64-bit
  5.11 64-bit
  6.4 (UEK R2) 64-bit
  6.5 (UEK R2, UEK R3) 64-bit
  6.6 (UEK R3) 64-bit
  6.7 (UEK R3) 64-bit
  7.1 64-bit
  7.2 64-bit
SLES
SUSE Linux Enterprise Server (SLES) 11 with Service Pack 2 64-bit
SUSE Linux Enterprise Server (SLES) 11 with Service Pack 3 64-bit
SUSE Linux Enterprise Server (SLES) 11 with Service Pack 4 64-bit
Ubuntu/Debian
Ubuntu Precise 12.04 - Long-Term Support (LTS) 64-bit
  Trusty 14.04 - Long-Term Support (LTS) 64-bit
Debian Wheezy 7.0, 7.1, and 7.8 64-bit
 
Important: Cloudera supports RHEL 7 with the following limitations:
 
Note:
  • Cloudera Enterprise is supported on platforms with Security-Enhanced Linux (SELinux) enabled. Cloudera is not responsible for policy support nor policy enforcement. If you experience issues with SELinux, contact your OS provider.
  • CDH 5.7 DataNode hosts with EMC® DSSD™ D5™ are supported by RHEL 6.6, 7.1, and 7.2. CDH 5.6 DataNode hosts with EMC® DSSD™ D5™ are only supported by RHEL 6.6.
Selected tab: SupportedOperatingSystems
Component MariaDB MySQL SQLite PostgreSQL Oracle Derby - see Note 5
Oozie 5.5 5.1, 5.5, 5.6, 5.7 8.1, 8.3, 8.4, 9.1, 9.2, 9.3, 9.4

See Note 3

11gR2, 12c Default
Flume Default (for the JDBC Channel only)
Hue 5.5 5.1, 5.5, 5.6, 5.7

See Note 6

Default 8.1, 8.3, 8.4, 9.1, 9.2, 9.3, 9.4

See Note 3

11gR2, 12c
Hive/Impala 5.5 5.1, 5.5, 5.6, 5.7

See Note 1

8.1, 8.3, 8.4, 9.1, 9.2, 9.3, 9.4

See Note 3

11gR2, 12c Default
Sentry 5.5 5.1, 5.5, 5.6, 5.7

See Note 1

8.1, 8.3, 8.4, 9.1, 9.2, 9.3, 9.4

See Note 3

11gR2, 12c
Sqoop 1 5.5 See Note 4 See Note 4 See Note 4
Sqoop 2 5.5 Default
 
  Note:
  1. MySQL 5.5 is supported on CDH 5.1. MySQL 5.6 is supported on CDH 5.1 and higher. The InnoDB storage engine must be enabled in the MySQL server.
  2. Cloudera Manager installation fails if GTID-based replication is enabled in MySQL.
  3. PostgreSQL 9.2 is supported on CDH 5.1 and higher. PostgreSQL 9.3 is supported on CDH 5.2 and higher. PostgreSQL 9.4 is supported on CDH 5.5 and higher.
  4. For purposes of transferring data only, Sqoop 1 supports MySQL 5.0 and above, PostgreSQL 8.4 and above, Oracle 10.2 and above, Teradata 13.10 and above, and Netezza TwinFin 5.0 and above. The Sqoop metastore works only with HSQLDB (1.8.0 and higher 1.x versions; the metastore does not work with any HSQLDB 2.x versions).
  5. Derby is supported as shown in the table, but not always recommended. See the pages for individual components in the Cloudera Installation and Upgrade guide for recommendations.
  6. CDH 5 Hue requires the default MySQL version of the operating system on which it is being installed, which is usually MySQL 5.1, 5.5, or 5.6.
Selected tab: SupportedDatabases
  Important: JDK 1.6 is not supported on any CDH 5 release (even though the libraries of CDH 5.0-CDH 5.4 are compatible). Applications using CDH libraries must run a supported version of JDK 1.7 or higher, and one that also matches the JDK version of your CDH cluster.
 
CDH 5.7.x is supported with the versions shown in the following table:
Minimum Supported Version Recommended Version Exceptions
1.7.0_55 1.7.0_67, 1.7.0_75, 1.7.0_80 None
1.8.0_31 1.8.0_60 Cloudera recommends that you not use JDK 1.8.0_40.
Selected tab: SupportedJDKVersions

Hue

Hue works with the two most recent versions of the following browsers. Cookies and JavaScript must be on.

  • Chrome
  • Firefox
  • Safari (not supported on Windows)
  • Internet Explorer

Hue could display in older versions and even other browsers, but you might not have access to all of its features.

Selected tab: SupportedBrowsers

 

CDH requires IPv4. IPv6 is not supported.

See also Configuring Network Names.

 

Selected tab: SupportedInternetProtocol

The following components are supported by the indicated versions of Transport Layer Security (TLS):

 

Table 1. Components Supported by TLS

Component

Role Name Port Version
Flume   Avro Source/Sink   TLS 1.2
Flume   Flume HTTP Source/Sink   TLS 1.2
HBase Master HBase Master Web UI Port 60010 TLS 1.2
HDFS NameNode Secure NameNode Web UI Port 50470 TLS 1.2
HDFS Secondary NameNode Secure Secondary NameNode Web UI Port 50495 TLS 1.2
HDFS HttpFS REST Port 14000 TLS 1.1, TLS 1.2
Hive HiveServer2 HiveServer2 Port 10000 TLS 1.2
Hue Hue Server Hue HTTP Port 8888 TLS 1.2
Cloudera Impala Impala Daemon Impala Daemon Beeswax Port 21000 TLS 1.2
Cloudera Impala Impala Daemon Impala Daemon HiveServer2 Port 21050 TLS 1.2
Cloudera Impala Impala Daemon Impala Daemon Backend Port 22000 TLS 1.2
Cloudera Impala Impala Daemon Impala Daemon HTTP Server Port 25000 TLS 1.2
Cloudera Impala Impala StateStore StateStore Service Port 24000 TLS 1.2
Cloudera Impala Impala StateStore StateStore HTTP Server Port 25010 TLS 1.2
Cloudera Impala Impala Catalog Server Catalog Server HTTP Server Port 25020 TLS 1.2
Cloudera Impala Impala Catalog Server Catalog Server Service Port 26000 TLS 1.2
Oozie Oozie Server Oozie HTTPS Port 11443 TLS 1.1, TLS 1.2
Solr Solr Server Solr HTTP Port 8983 TLS 1.1, TLS 1.2
Solr Solr Server Solr HTTPS Port 8985 TLS 1.1, TLS 1.2
YARN ResourceManager ResourceManager Web Application HTTP Port 8090 TLS 1.2
YARN JobHistory Server MRv1 JobHistory Web Application HTTP Port 19890 TLS 1.2
Selected tab: SupportedTransportLayerSecurityVersions
Selected tab: SystemRequirements

Issues Fixed in CDH 5.7.6

Upstream Issues Fixed

The following upstream issues are fixed in CDH 5.7.6:

  • AVRO-1943 - Flaky test: TestNettyServerWithCompression.testConnectionsCount.
  • FLUME-2908 - NetcatSource - SocketChannel not closed when session is broken.
  • FLUME-2997 - Fix flaky test in SpillableMemoryChannel.
  • FLUME-3002 - Fix tests in TestBucketWriter.
  • FLUME-3003 - Fix flaky testSourceCounter in TestSyslogUdpSource.
  • FLUME-3049 - Make HDFS sink rotate more reliably in secure mode.
  • HADOOP-7930 - Kerberos relogin interval in UserGroupInformation should be configurable.
  • HADOOP-11031 - Design Document for Credential Provider API.
  • HADOOP-11619 - FTPFileSystem should override getDefaultPort.
  • HADOOP-12453 - Support decoding KMS Delegation Token with its own Identifier.
  • HADOOP-12537 - S3A to support Amazon STS temporary credentials.
  • HADOOP-12655 - TestHttpServer.testBindAddress bind port range is wider than expected.
  • HADOOP-12723 - S3A: Add ability to plug in any AWSCredentialsProvider.
  • HADOOP-13034 - Log message about input options in distcp lacks some items.
  • HADOOP-13433 - Race in UGI.reloginFromKeytab.
  • HADOOP-13590 - Retry until TGT expires even if the UGI renewal thread encountered exception.
  • HADOOP-13627 - Have an explicit KerberosAuthException for UGI to throw, text from public constants.
  • HADOOP-13641 - Update UGI#spawnAutoRenewalThreadForUserCreds to reduce indentation.
  • HADOOP-13838 - KMSTokenRenewer should close providers.
  • HADOOP-13953 - Make FTPFileSystem's data connection mode and transfer mode configurable.
  • HADOOP-14003 - Make additional KMS tomcat settings configurable.
  • HDFS-9428 - Fix intermittent failure of TestDNFencing.testQueueingWithAppend.
  • HDFS-9630 - DistCp minor refactoring and clean up.
  • HDFS-9638 - Improve DistCp Help and documentation.
  • HDFS-9764 - DistCp doesn't print value for several arguments including -numListstatusThreads.
  • HDFS-9804 - Allow long-running Balancer to login with keytab.
  • HDFS-9820 - Improve distcp to support efficient restore to an earlier snapshot.
  • HDFS-9888 - Allow reseting KerberosName in unit tests.
  • HDFS-10216 - Distcp -diff throws exception when handling relative path.
  • HDFS-10271 - Extra bytes are getting released from reservedSpace for append.
  • HDFS-10298 - Document the usage of distcp -diff option.
  • HDFS-10313 - Distcp need to enforce the order of snapshot names passed to -diff.
  • HDFS-10336 - TestBalancer failing intermittently because of not reseting UserGroupInformation completely
  • HDFS-10397 - Distcp should ignore -delete option if -diff option is provided instead of exiting.
  • HDFS-10556 - DistCpOptions should be validated automatically.
  • HDFS-10763 - Open files can leak permanently due to inconsistent lease update.
  • HDFS-11040 - Add documentation for HDFS-9820 distcp improvement.
  • HDFS-11056 - Concurrent append and read operations lead to checksum error.
  • HDFS-11160 - VolumeScanner reports write-in-progress replicas as corrupt incorrectly.
  • HDFS-11229 - HDFS-11056 failed to close meta file.
  • HDFS-11275 - Check groupEntryIndex and throw a helpful exception on failures when removing ACL.
  • HDFS-11292 - log lastWrittenTxId etc info in logSyncAll.
  • HDFS-11306 - Print remaining edit logs from buffer if edit log can't be rolled.
  • MAPREDUCE-6571 - JobEndNotification info logs are missing in AM container syslog.
  • MAPREDUCE-6763 - Shuffle server listen queue is too small.
  • MAPREDUCE-6798 - Fix intermittent failure of TestJobHistoryParsing.testJobHistoryMethods.
  • MAPREDUCE-6801 - Fix flaky TestKill.testKillJob.
  • MAPREDUCE-6817 - The format of job start time in JHS is different from those of submit and finish time.
  • MAPREDUCE-6831 - Flaky test TestJobImpl.testKilledDuringKillAbort
  • YARN-2306 - Add test for leakage of reservation metrics in fair scheduler.
  • YARN-3554 - Default value for maximum nodemanager connect wait time is too high.
  • YARN-4363 - In TestFairScheduler, testcase should not create FairScheduler redundantly.
  • YARN-4555 - TestDefaultContainerExecutor#testContainerLaunchError fails on non-english locale environment.
  • YARN-5752 - TestLocalResourcesTrackerImpl#testLocalResourceCache times out.
  • YARN-5837 - NPE when getting node status of a decommissioned node after an RM restart.
  • YARN-5859 - TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails
  • YARN-5862 - TestDiskFailures.testLocalDirsFailures failed.
  • YARN-5890 - FairScheduler should log information about AM-resource-usage and max-AM-share for queues.
  • YARN-5920 - Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang.
  • HBASE-15324 - Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split.
  • HBASE-15430 - Failed taking snapshot - Manifest proto-message too large.
  • HBASE-16172 - Unify the retry logic in ScannerCallableWithReplicas and RpcRetryingCallerWithReadReplicas.
  • HBASE-16270 - Handle duplicate clearing of snapshot in region replicas.
  • HBASE-16345 - RpcRetryingCallerWithReadReplicas#call() should catch some RegionServer Exceptions.
  • HBASE-16824 - Writer.flush() can be called on already closed streams in WAL roll.
  • HBASE-16841 - Data loss in MOB files after cloning a snapshot and deleting that snapshot.
  • HBASE-17058 - Lower epsilon used for jitter verification from HBASE-15324.
  • HBASE-17241 - Avoid compacting already compacted mob files with _del files.
  • HBASE-17452 - Failed taking snapshot - region Manifest proto-message too large.
  • HBASE-17522 - Handle JVM throwing runtime exceptions when we ask for details on heap usage the same as a correctly returned 'undefined'.
  • HIVE-10965 - direct SQL for stats fails in 0-column case.
  • HIVE-11849 - NPE in HiveHBaseTableShapshotInputFormat in query with just count(*).
  • HIVE-12083 - HIVE-10965 introduces thrift error if partNames or colNames are empty.
  • HIVE-12465 - Hive might produce wrong results when (outer) joins are merged.
  • HIVE-12619 - Switching the field order within an array of structs causes the query to fail.
  • HIVE-12780 - Fix the output of the history command in Beeline HIVE-12789: Fix output twice in the history command of Beeline.
  • HIVE-12891 - Hive fails when java.io.tmpdir is set to a relative location.
  • HIVE-12976 - MetaStoreDirectSql doesn't batch IN lists in all cases.
  • HIVE-13129 - CliService leaks HMS connection.
  • HIVE-13149 - Remove some unnecessary HMS connections from HS2.
  • HIVE-13240 - GroupByOperator: Drop the hash aggregates when closing operator.
  • HIVE-13539 - HiveHFileOutputFormat searching the wrong directory for HFiles.
  • HIVE-13696 - Modify FairSchedulerShim to dynamically reload changes to fair-scheduler.xml.
  • HIVE-13866 - flatten callstack for directSQL errors.
  • HIVE-14173 - NPE was thrown after enabling directsql in the middle of session.
  • HIVE-14764 - Enabling "hive.metastore.metrics.enabled" throws OOM in HiveMetastore.
  • HIVE-14820 - RPC server for spark inside HS2 is not getting server address properly.
  • HIVE-15054 - Hive insertion query execution fails on Hive on Spark.
  • HIVE-15090 - Temporary DB failure can stop ExpiredTokenRemover thread.
  • HIVE-15338 - Wrong result from non-vectorized DATEDIFF with scalar parameter of type DATE/TIMESTAMP.
  • HIVE-15410 - WebHCat supports get/set table property with its name containing period and hyphen.
  • HIVE-15551 - memory leak in directsql for mysql+bonecp specific initialization.
  • HUE-4466 - [security] deliver csrftoken cookie with secure bit set if possible.
  • HUE-5218 - [search] Validate dashboard sharing works.
  • IMPALA-2864 - Ensure that client connections are closed after a failed Open()
  • IMPALA-3167 - Fix assignment of WHERE conjunct through grouping agg + OJ.
  • IMPALA-3552 - Make incremental stats max serialized size configurable
  • IMPALA-3698 - Fix Isilon permissions test
  • IMPALA-3861 - Replace BetweenPredicates with their equivalent CompoundPredicate.
  • IMPALA-3875 - Thrift threaded server hang in some cases
  • IMPALA-3983 - /IMPALA-3974: Delete function jar resources after load
  • IMPALA-4153 - Return valid non-NULL pointer for 0-byte allocations
  • IMPALA-4223 - Handle truncated file read from HDFS cache
  • IMPALA-4336 - Cast exprs after unnesting union operands.
  • IMPALA-4363 - Add Parquet timestamp validation
  • IMPALA-4423 - Correct but conservative implementation of Subquery.equals().
  • IMPALA-4433 - Always generate testdata using the same time zone setting
  • IMPALA-4449 - Revisit table locking pattern in the catalog
  • IMPALA-4488 - HS2 GetOperationStatus() should keep session alive
  • IMPALA-4550 - Fix CastExpr analysis for substituted slots
  • IMPALA-4579 - SHOW CREATE VIEW fails for view containing a subquery
  • IMPALA-4765 - Avoid using several loading threads on one table.
  • OOZIE-2194 - oozie job -kill doesn't work with spark action.
  • OOZIE-2243 - Kill Command does not kill the child job for java action.
  • OOZIE-2584 - Eliminate Thread.sleep() calls in TestMemoryLocks.
  • OOZIE-2678 - Oozie job -kill doesn't work with tez jobs.
  • OOZIE-2742 - Unable to kill applications based on tag.
  • PIG-5025 - Fix flaky test failures in TestLoad.java.
  • SENTRY-1260 - Improve error handling - ArrayIndexOutOfBoundsException in PathsUpdate.parsePath can cause MetastoreCacheInitializer intialization to fail.
  • SENTRY-1270 - Improve error handling - Database with malformed URI causes NPE in HMS plugin during DDL.
  • SENTRY-1520 - Provide mechanism for triggering HMS full snapshot.
  • SENTRY-1564 - Improve error detection and reporting in MetastoreCacheInitializer.java.
  • SOLR-9284 - The HDFS BlockDirectoryCache should not let it's keysToRelease or names maps grow indefinitely.
  • SOLR-9330 - Fix AlreadyClosedException on admin/mbeans?stats=true.
  • SOLR-10031 - Validation of filename params in ReplicationHandler.
  • SPARK-12241 - [YARN] Improve failure reporting in Yarn client obtainTokenForHBase().
  • SPARK-12523 - [YARN] Support long-running of the Spark On HBase and hive meta store.
  • SPARK-12966 - [SQL] ArrayType(DecimalType) support in Postgres JDBC.
  • SPARK-13566 - [CORE] Avoid deadlock between BlockManager and Executor Thread.
  • SPARK-13958 - Executor OOM due to unbounded growth of pointer array in…
  • SPARK-14204 - [SQL] register driverClass rather than user-specified class.
  • SPARK-16044 - [SQL] Backport input_file_name() for data source based on NewHadoopRDD to branch 1.6.
  • SPARK-17245 - [SQL][BRANCH-1.6] Do not rely on Hive's session state to retrieve HiveConf.
  • SPARK-17465 - [SPARK CORE] Inappropriate memory management in `org.apache.spark.storage.MemoryStore` may lead to memory leak.
  • SPARK-18750 - [YARN] Follow up: move test to correct directory in 2.1 branch.
  • SPARK-18750 - [YARN] Avoid using "mapValues" when allocating containers.
  • SQOOP-2349 - Add command line option for setting transaction isolation levels for metadata queries.
  • SQOOP-2880 - Provide argument for overriding temporary directory.
  • SQOOP-2884 - Document --temporary-rootdir.
  • SQOOP-2915 - Fixing Oracle related unit tests.
  • SQOOP-2983 - OraOop export has degraded performance with wide tables.
  • SQOOP-3013 - Configuration "tmpjars" is not checked for empty strings before passing to MR.
  • SQOOP-3028 - Include stack trace in the logging of exceptions in ExportTool.
  • SQOOP-3034 - HBase import should fail fast if using anything other than as-textfile.
  • SQOOP-3053 - Create a cmd line argument for sqoop.throwOnError and use it through SqoopOptions.
  • SQOOP-3055 - Fixing MySQL tests failing due to ignored test inputs/configuration.
  • SQOOP-3057 - Fixing 3rd party Oracle tests failing due to invalid case of column names.
  • SQOOP-3071 - Fix OracleManager to apply localTimeZone correctly in case of Date objects too.
  • SQOOP-3124 - Fix ordering in column list query of PostgreSQL connector to reflect the logical order instead of adhoc ordering.
Selected tab: WhatsNew

Want to Get Involved or Learn More?

Check out our other resources

Cloudera Community

Collaborate with your peers, industry experts, and Clouderans to make the most of your investment in Hadoop.

Cloudera University

Receive expert Hadoop training through Cloudera University, the industry's only truly dynamic Hadoop training curriculum that’s updated regularly to reflect the state of the art in big data.