CDH 6 includes Apache Kafka as part of the core package. The documentation includes improved contents for how to set up, install, and administer your Kafka ecosystem. For more information, see the Cloudera Enterprise 6.0.x Apache Kafka Guide. We look forward to your feedback on both the existing and new documentation.
CDK Powered By Apache Kafka Incompatible Changes and Limitations
This section describes incompatible changes and limitations:
Incompatible Changes and Limitations in CDK 4.0.0 Powered By Apache Kafka
Scala-based Clients API Removed
Properties for Exactly Once Semantics Not Available in Cloudera Manager
The configuration properties related the idempotent and transactional capabilities of the producer are not available for configuration via Cloudera Manager. These properties must be set through the Kafka Broker Advanced Configuration Snippet (Safety Valve) for kafka.properties safety valve. For more information regarding configuration using safety valves, see Custom Configuration.
- Broker Properties
- Producer Properties
- Consumer Properties
For more information, see the upstream Apache Kafka documentation.
Default Behaviour Changes in CDK 4.0.0 Powered by Apache Kafka
- Unclean leader election is automatically enabled by the controller when unclean.leader.election.enable config is dynamically updated by using per-topic config override.
- The default value for request.timeout.ms is decreased to 30 seconds. In addition, a new logic is added that makes the JoinGroup requests ignore this timeout.
Incompatible Changes and Limitations in CDK 3.1.0 Powered By Apache Kafka
Scala-based Clients API Deprecated
Incompatible Changes and Limitations in CDK 3.0.0 Powered By Apache Kafka
CDK 3.0 Requires CDH 5.13 when Co-located
Using version 3.0 and later of CDK Powered by Apache Kafka requires a newer version of Cloudera Manager and/or CDH when Kafka and CDH are in the same logical cluster in Cloudera Manager. For more information on compatibilities among versions, see Product Compatibility Matrix for CDK Powered By Apache Kafka.
Incompatible Changes and Limitations in CDK 2.0.0 Powered By Apache Kafka
Flume shipped with CDH 5.7 and lower can only send data to CDK 2.0 and higher Powered By Apache Kafka via unsecured transport.Security additions to CDK 2.0 Powered By Apache Kafka are not supported by Flume in CDH 5.7 (or lower versions).
Topic Blacklist Removed
The MirrorMaker Topic blacklist setting has been removed in CDK 2.0 and higher Powered By Apache Kafka.
Avoid Data Loss Option Removed
The Avoid Data Loss option from earlier releases has been removed in CDK 2.0 Powered By Apache Kafka in favor of automatically setting the following properties.
- Producer settings
- retries=max integer
- max.block.ms=max long
- Consumer setting
- MirrorMaker setting