Managing Multiple Kafka Versions

Kafka Feature Support in Cloudera Manager and CDH

Using the latest features of Kafka sometimes requires a newer version of Cloudera Manager and/or CDH that supports the feature. For the purpose of understanding upgrades, the table below includes several earlier versions where Kafka was known as CDK Powered by Apache Kafka (CDK in the table).

Kafka and CM/CDH Supported Versions Matrix
CDH Kafka Version Apache Kafka Version Minimum Supported Version  
Cloudera Manager

CDH
(No Security)

CDH
(with Sentry)

Kafka Feature Notes
CDH 6.0.0 1.0.1 CM 6.0.0 CDH 6.0.0 CDH 6.0.0  
CDK 3.1.0 1.0.1 CM 5.13.0 CDH 5.13.0 CDH 5.13.0 Sentry-HA supported
CDK 3.0.0 0.11.0 CM 5.13.0 CDH 5.13.0 CDH 5.13.0 Sentry-HA not supported
CDK 2.2 0.10.2 CM 5.9.0 CDH 5.4.0 CDH 5.9.0  
CDK 2.1 0.10.0 CM 5.9.0 CDH 5.4.0 CDH 5.9.0 Sentry Authorization
CDK 2.0 0.9.0 CM5.5.3 CDH 5.4.0 CDH 5.4.0 Enhanced Security

Client/Broker Compatibility Across Kafka Versions

Maintaining compatibility across different Kafka clients and brokers is a common issue. Mismatches among client and broker versions can occur as part of any of the following scenarios:

  • Upgrading your Kafka cluster without upgrading your Kafka clients.
  • Using a third-party application that produces to or consumes from your Kafka cluster.
  • Having a client program communicating with two or more Kafka clusters (such as consuming from one cluster and producing to a different cluster).
  • Using Flume or Spark as a Kafka consumer.

In these cases, it is important to understand client/broker compatibility across Kafka versions. Here are general rules that apply:

  • Newer Kafka brokers can talk to older Kafka clients. The reverse is not true: older Kafka brokers cannot talk to newer Kafka clients.
  • Changes in either the major part or the minor part of the upstream version major.minor (highlighted in red in the table above) determines whether the client and broker are compatible. Differences among the maintenance versions are not considered when determining compatibility.

As a result, the general pattern for upgrading Kafka from version A to version B is:

  1. Change Kafka server.properties to refer to version A.
  2. Upgrade the brokers to version B and restart the cluster.
  3. Upgrade the clients to version B.
  4. After all the clients are upgraded, change the properties from the first step to version B and restart the cluster.

Upgrading your Kafka Cluster

At some point, you will want to upgrade your Kafka cluster. Or in some cases, when you upgrade to a newer CDH distribution, then Kafka will be upgraded along with the full CDH upgrade. In either case, you may wish to read the sections below before upgrading.

General Upgrade Information

Previously, Cloudera distributed Kafka as a parcel (CDK) separate from CDH. Installation of the separate Kafka required Cloudera Manager 5.4 or higher. For a list of available parcels and packages, see CDK Powered By Apache Kafka® Version and Packaging Information Kafka bundled along with CDH.

As of CDH 6, Cloudera distributes Kafka bundled along with CDH. Kafka is in the parcel itself. Installation requires Cloudera Manager 6.0 or higher. For installation instructions for Kafka using Cloudera Manager, see Cloudera Installation Guide.

Cloudera recommends that you deploy Kafka on dedicated hosts that are not used for other cluster roles.

Upgrading Kafka from CDH 6.0.0 to other CDH 6 versions

To ensure there is no downtime during an upgrade, these instructions describe performing a rolling upgrade.

Before upgrading, ensure that you set inter.broker.protocol.version and log.message.format.version to the current Kafka version (see Kafka and CM/CDH Supported Versions Matrix), and then unset them after the upgrade. This is a good practice because the newer broker versions can write log entries that the older brokers cannot read. If you need to rollback to the older version, and you have not set inter.broker.protocol.version and log.message.format.version, data loss can occur.

From Cloudera Manager on the cluster to upgrade:

  1. Explicitly set the Kafka protocol version to match what's being used currently among the brokers and clients.

    Update server.properties on all brokers as follows:

    1. Choose the Kafka service.
    2. Click Configuration.
    3. Use the Search field to find the Kafka Broker Advanced Configuration Snippet (Safety Valve) configuration property.
    4. Add the following properties to the safety valve:
      inter.broker.protocol.version = current_Kafka_version
      log.message.format.version = current_Kafka_version

      Make sure you enter full Kafka version numbers with three values, such as 0.10.0. Otherwise, you'll see an error similar to the following:

      2018-06-14 14:25:47,818 FATAL kafka.Kafka$:
      java.lang.IllegalArgumentException: Version `0.10` is not a valid version
              at kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:72)
              at kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:72)
              at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
                      
    5. Save your changes. The information is automatically copied to each broker.
  2. Upgrade CDH. See Upgrading CDH.

    Do not restart the Kafka service, select Activate Only and click OK.

  3. Perform a rolling restart.

    Select Rolling Restart or Restart based on the downtime that you can afford.

At this point, the brokers are running in compatibility mode with older clients. It can run in this mode indefinitely. If you do upgrade clients, after all clients are upgraded, remove the Safety Valve properties and restart the cluster.

Upstream Upgrade Instructions

The table below links to the upstream Apache Kafka documentation for detailed upgrade instructions. It is recommended that you read the instructions for your specific upgrade to identify any additional steps that apply to your specific upgrade path.

Version-Specific Upgrade Instructions
CDH Kafka Version Upstream Kafka Version Detailed Upstream Instructions
CDH 6.0.0 1.0.1 Upgrading from 0.8.x through 0.11.0.x to 1.0.0
Kafka 3.1.0 1.0.1 Upgrading from 0.8.x through 0.11.0.x to 1.0.0
Kafka 3.0.0 0.11.0 Upgrading from 0.8.x through 0.10.2.x to 0.11.0.0
Kafka 2.2 0.10.2 Upgrading from 0.8.x through 0.10.1.x to 0.10.2.0
Kafka 2.1 0.10.0 Upgrading from 0.8.x through 0.9.x.to 0.10.0.0
Kafka 2.0 0.9.0 Upgrading from 0.8.0 through 0.8.2.x to 0.9.0.0