CDH 6 includes Apache Kafka as part of the core package. The documentation includes improved contents for how to set up, install, and administer your Kafka ecosystem. For more information, see the Cloudera Enterprise 6.0.x Apache Kafka Guide. We look forward to your feedback on both the existing and new documentation.

Using Apache Kafka Command-line Tools

Apache Kafka command-line tools are located in /usr/bin:
  • kafka-topics
    Create, alter, list, and describe topics. For example: 
    $ /usr/bin/kafka-topics --zookeeper zk01.example.com:2181 --list
sink1
t1
t2
$ /usr/bin/kafka-topics --create --zookeeper hostname:2181/kafka --replication-factor 2 
  --partitions 4 --topic topicname
  • kafka-console-consumer
    Read data from a Kafka topic and write it to standard output. For example: 
    $ /usr/bin/kafka-console-consumer --zookeeper zk01.example.com:2181 --topic t1
  • kafka-console-producer
    Read data from standard output and write it to a Kafka topic. For example: 
    $ /usr/bin/kafka-console-producer --broker-list kafka02.example.com:9092,kafka03.example.com:9092 --topic t1
  • kafka-consumer-offset-checker (deprecated)
    Check the number of messages read and written, as well as the lag for each consumer in a specific consumer group. For example: 
    $ /usr/bin/kafka-consumer-offset-checker --group flume --topic t1 --zookeeper zk01.example.com:2181

  • kafka-consumer-groups

    To view offsets as in the previous example with the ConsumerOffsetChecker, you describe the consumer group using the following command: 
    $ /usr/bin/kafka-consumer-groups --zookeeper zk01.example.com:2181 --describe --group flume

GROUP   TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     OWNER
flume   t1     0          1               3               2       test-consumer-group_postamac.local-1456198719410-29ccd54f-0