Cloudera Distribution of Apache Kafka Known Issues

Flume Kafka Client Incompatible Changes in CDH 5.8

After an upgrade to CDH 5.8, Flume throws an error on startup and stops processing Kafka messages. This is caused by an incompatible configuration change: upgrade of the Kafka client from 0.8.2 to 0.9.0 in CDH 5.8.

Additionally, Flume may skip some Kafka messages because Kafka offset storage moves from ZooKeeper to Kafka with the client change. Without knowing about the previous offsets stored in ZooKeeper, the Flume Kafka client assumes the most recent offset, skipping over the existing messages that have not yet been read.

Products affected: Flume

Releases affected: CDH 5.8.0, CDH 5.8.1

Users affected: Users who have Flume configured with a Kafka source or channel. Flume Kafka sinks are not affected.

Severity (Low/Medium/High): High

Impact: Potential loss of messages consumed by the Flume agent from Kafka source, or loss of events in the Flume Kafka channel after upgrade

Immediate action required: If you are upgrading from a version lower than CDH 5.7.0, upgrade to CDH 5.7.x first. The Flume Kafka client version in lower versions of CDH does not support the migration properties required in the Flume configuration.

If you are currently running CDH 5.7.x, upgrade to CDH 5.8.2 or CDH 5.9.0 to resolve this issue. If you are upgrading to CDH 5.8.0 or CDH 5.8.1, see the following documentation to perform the steps required for Flume before upgrading: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_58_upgrade_kafka_flume_note.html

Addressed in release/refresh/patch: CDH 5.8.2 and higher, CDH 5.9.0 and higher

Workaround: See Upgrading to CDH 5.8 When Using the Flume Kafka Client