The next release of CDH includes Apache Kafka as part of the core package. The documentation for the next release includes improved contents for how to set up, install, and administer your Kafka ecosystem. To see a pre-release of the new documentation, see the Cloudera Enterprise 6.0.x Apache Kafka Guide. We look forward to your feedback on both the existing and new documentation.

Using Apache Kafka Command-line Tools

Apache Kafka command-line tools are located in /usr/bin:
  • kafka-topics
    Create, alter, list, and describe topics. For example:
    $ /usr/bin/kafka-topics --zookeeper zk01.example.com:2181 --list
    sink1
    t1
    t2
    $ /usr/bin/kafka-topics --create --zookeeper hostname:2181/kafka --replication-factor 2 
      --partitions 4 --topic topicname 
                            
  • kafka-console-consumer
    Read data from a Kafka topic and write it to standard output. For example:
    $ /usr/bin/kafka-console-consumer --zookeeper zk01.example.com:2181 --topic t1
  • kafka-console-producer
    Read data from standard output and write it to a Kafka topic. For example:
    $ /usr/bin/kafka-console-producer --broker-list kafka02.example.com:9092,kafka03.example.com:9092 --topic t1
  • kafka-consumer-offset-checker (deprecated)
    Check the number of messages read and written, as well as the lag for each consumer in a specific consumer group. For example:
    $ /usr/bin/kafka-consumer-offset-checker --group flume --topic t1 --zookeeper zk01.example.com:2181
  • kafka-consumer-groups

    To view offsets as in the previous example with the ConsumerOffsetChecker, you describe the consumer group using the following command:
    $ /usr/bin/kafka-consumer-groups --zookeeper zk01.example.com:2181 --describe --group flume
    
    GROUP   TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG     OWNER
    flume   t1     0          1               3               2       test-consumer-group_postamac.local-1456198719410-29ccd54f-0