CDH 6 includes Apache Kafka as part of the core package. The documentation includes improved contents for how to set up, install, and administer your Kafka ecosystem. For more information, see the Cloudera Enterprise 6.0.x Apache Kafka Guide. We look forward to your feedback on both the existing and new documentation.

CDK Powered By Apache Kafka Requirements and Supported Versions

The following sections describe software requirements and supported versions of complementary software for CDK Powered By Apache Kafka:

Supported CDH and Cloudera Manager Releases

For the list of supported releases of CDH and Cloudera Manager, see CDH and Cloudera Manager Supported Versions.

Supported Integrations

Flume and Spark Connectors to Kafka

Flume and Spark connectors to Kafka are included with CDH 5.7.x and higher and only work with CDK 2.0.x Powered By Apache Kafka and higher.

Sentry Integration with Kafka

Sentry authorization integration with Kafka is available with CDK Powered By Apache Kafka version 2.1.x and higher on CDH 5.9.x and higher.

CDH 5.13.x introduces Sentry-HA; CDK 3.1 provides a Sentry-HA aware client. You can connect with an older version, but it won't be in high-availability mode: if the Sentry server that serves Kafka goes away, Kafka will not have access to authorization information.

Supported Operating Systems

For the list of supported operating systems, see CDH and Cloudera Manager Supported Operating Systems.

SUSE Linux Enterprise Server (SLES)

Unlike CentOS, SLES limits virtual memory by default. Changing this default requires adding the following entries to the /etc/security/limits.conf file:

* hard as unlimited
* soft as unlimited

Supported JDK Versions

CDK 3.0 and higher Powered By Apache Kafka require JDK 8, and do not support JDK 7.

For a list of supported and tested JDK versions for CDK 2.2.x and below Powered By Apache Kafka, see CDH and Cloudera Manager Supported JDK Versions.

Ports Used by Kafka

Kafka uses the TCP ports listed in the following table. Before deploying Kafka, ensure that these ports are open on each system.

Component Service Port Access Requirement Comment

Broker

TCP Port

9092

External/Internal

The primary communication port used by producers and consumers; also used for inter-broker communication.

Broker

TLS/SSL Port

9093

External/Internal

A secured communication port used by producers and consumers; also used for inter-broker communication.

Broker

JMX Port

9393

Internal

Internal use only. Used for administration via JMX.

MirrorMaker

JMX Port

9394

Internal

Internal use only. Used to administer the producer and consumer of the MirrorMaker.

Broker

HTTP Metric Report Port

24042

Internal

Internal use only. This is the port via which the HTTP metric reporter listens. It is used to retrieve metrics through HTTP instead of JMX.