Kafka Administration Basics

Broker Log Management

As mentioned previously, Kafka brokers save their data as log segments in a directory. The logs are rotated depending on the size and time settings.

The most common log retention settings to adjust for your cluster are shown below. These are accessible in Cloudera Manager via the Kafka > Configuration tab.

  • log.dirs: The location for the Kafka data (that is, topic directories and log segments).
  • log.retention.{ms|minutes|hours}: The retention period for the entire log. Any older log segments are removed.
  • log.retention.bytes: The retention size for the entire log.

There are many more variables available for fine-tuning broker log management. For more detailed information, look at the relevant variables in the Apache Kafka documentation topic Broker Configs.

  • log.dirs
  • log.flush.*
  • log.retention.*
  • log.roll.*
  • log.segment.*

Record Management

As mentioned previously, Kafka brokers save their data as log segments in a directory. The logs are rotated depending on the size and time settings.

The most common log retention settings to adjust for your cluster are shown below. These are accessible in Cloudera Manager via the Kafka > Configuration tab.

  • log.dirs: The location for the Kafka data (that is, topic directories and log segments).
  • log.retention.{ms|minutes|hours}: The retention period for the entire log. Any older log segments are removed.
  • log.retention.bytes: The retention size for the entire log.

There are many more variables available for fine-tuning broker log management. For more detailed information, look at the relevant variables in the Apache Kafka documentation topic Broker Configs.

  • log.dirs
  • log.flush.*
  • log.retention.*
  • log.roll.*
  • log.segment.*

Broker Garbage Collection

To set the JVM Garbage Collection log path on the brokers:

  1. In Cloudera Manager, go to Kafka > Configurations.
  2. Find the property Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) and add the line:
    KAFKA_GC_LOG_OPTS="-Xloggc:/var/logs/kafka/gc-broker.log"
  3. Restart the Kafka service to apply the new configuration.

To set the JVM Garbage log rotation on the brokers:

  1. In Cloudera Manager, go to Kafka > Configurations.
  2. Find the property Kafka Broker Environment Advanced Configuration Snippet (Safety Valve) and add the line (modified as appropriate):
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=10
    -XX:GCLogFileSize=100M
                    
  3. Restart the Kafka service to apply the new configuration.

Adding Users as Kafka Administrators

In some cases, additional users besides the kafka account need administrator access. This can be done in Cloudera Manager by going to Kafka > Configuration > Super users.