Apache Kudu Administration

Starting and Stopping Kudu Processes

Start Kudu services using the following commands:
sudo service kudu-master start
sudo service kudu-tserver start
To stop Kudu services, use the following commands:
sudo service kudu-master stop
sudo service kudu-tserver stop
Configure the Kudu services to start automatically when the server starts, by adding them to the default runlevel.
sudo chkconfig kudu-master on            # RHEL / CentOS 
sudo chkconfig kudu-tserver on           # RHEL / CentOS 

sudo update-rc.d kudu-master defaults    # Ubuntu 
sudo update-rc.d kudu-tserver defaults   # Ubuntu

Kudu Web Interfaces

Kudu tablet servers and masters expose useful operational information on a built-in web interface.

Kudu Master Web Interface

Kudu master processes serve their web interface on port 8051. The interface exposes several pages with information about the state of the cluster.

  • A list of tablet servers, their host names, and the time of their last heartbeat.

  • A list of tables, including schema and tablet location information for each.

  • SQL code which you can paste into Impala Shell to add an existing table to Impala’s list of known data sources.

Kudu Tablet Server Web Interface

Each tablet server serves a web interface on port 8050. The interface exposes information about each tablet hosted on the server, its current state, and debugging information about maintenance background operations.

Common Web Interface Pages

Both Kudu masters and tablet servers expose the following information via their web interfaces:

  • HTTP access to server logs.

  • An /rpcz endpoint which lists currently running RPCs via JSON.

  • Details about the memory usage of different components of the process.

  • The current set of configuration flags.

  • Currently running threads and their resource consumption.

  • A JSON endpoint exposing metrics about the server.

  • The version number of the daemon deployed on the cluster.

These interfaces are linked from the landing page of each daemon’s web UI.

Kudu Metrics

Kudu daemons expose a large number of metrics. Some metrics are associated with an entire server process, whereas others are associated with a particular tablet replica.

Listing available metrics

The full set of available metrics for a Kudu server can be dumped using a special command line flag:

$ kudu-tserver --dump_metrics_json
$ kudu-master --dump_metrics_json

This will output a large JSON document. Each metric indicates its name, label, description, units, and type. Because the output is JSON-formatted, this information can easily be parsed and fed into other tooling which collects metrics from Kudu servers.

For the complete list of metrics collected by Cloudera Manager for a Kudu service, look for the Kudu metrics listed under Cloudera Manager Metrics .

Collecting metrics via HTTP

Metrics can be collected from a server process via its HTTP interface by visiting /metrics. The output of this page is JSON for easy parsing by monitoring services. This endpoint accepts several GET parameters in its query string:

  • /metrics?metrics=<substring1>,<substring2>,…​ - Limits the returned metrics to those which contain at least one of the provided substrings. The substrings also match entity names, so this may be used to collect metrics for a specific tablet.

  • /metrics?include_schema=1 - Includes metrics schema information such as unit, description, and label in the JSON output. This information is typically omitted to save space.

  • /metrics?compact=1 - Eliminates unnecessary whitespace from the resulting JSON, which can decrease bandwidth when fetching this page from a remote host.

  • /metrics?include_raw_histograms=1 - Include the raw buckets and values for histogram metrics, enabling accurate aggregation of percentile metrics over time and across hosts.

For example:

$ curl -s 'http://example-ts:8050/metrics?include_schema=1&metrics=connections_accepted'
[
    {
        "type": "server",
        "id": "kudu.tabletserver",
        "attributes": {},
        "metrics": [
            {
                "name": "rpc_connections_accepted",
                "label": "RPC Connections Accepted",
                "type": "counter",
                "unit": "connections",
                "description": "Number of incoming TCP connections made to the RPC server",
                "value": 92
            }
        ]
    }
]
$ curl -s 'http://example-ts:8050/metrics?metrics=log_append_latency'
[
    {
        "type": "tablet",
        "id": "c0ebf9fef1b847e2a83c7bd35c2056b1",
        "attributes": {
            "table_name": "lineitem",
            "partition": "hash buckets: (55), range: [(<start>), (<end>))",
            "table_id": ""
        },
        "metrics": [
            {
                "name": "log_append_latency",
                "total_count": 7498,
                "min": 4,
                "mean": 69.3649,
                "percentile_75": 29,
                "percentile_95": 38,
                "percentile_99": 45,
                "percentile_99_9": 95,
                "percentile_99_99": 167,
                "max": 367244,
                "total_sum": 520098
            }
        ]
    }
]

Collecting metrics to a log

Kudu can be configured to periodically dump all of its metrics to a local log file using the --metrics_log_interval_ms flag. Set this flag to the interval at which metrics should be written to a log file.

The metrics log will be written to the same directory as the other Kudu log files, and with the same naming format. After any metrics log file reaches 64MB uncompressed, the log will be rolled and the previous file will be gzip-compressed.

The log file generated has three space-separated fields:
  • The first field is the word metrics.
  • The second field is the current timestamp in microseconds since the Unix epoch.
  • The third is the current value of all metrics on the server, using a compact JSON encoding. The encoding is the same as the metrics fetched via HTTP described above.

Common Kudu workflows

Migrating to Multiple Kudu Masters

For high availability and to avoid a single point of failure, Kudu clusters should be created with multiple masters. Many Kudu clusters were created with just a single master, either for simplicity or because Kudu multi-master support was still experimental at the time. This workflow demonstrates how to migrate to a multi-master configuration.

Prepare for the migration

  1. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster will be unavailable.
  2. Decide how many masters to use. The number of masters should be odd. Three or five node master configurations are recommended; they can tolerate one or two failures respectively.
  3. Perform the following preparatory steps for the existing master:
    • Identify and record the directory where the master’s data lives. If you are using Kudu system packages, the default value is /var/lib/kudu/master, but it may be customized using the fs_wal_dir and fs_data_dirs configuration parameters. If you’ve set fs_data_dirs to some directories other than the value of fs_wal_dir, it should be explicitly included in every command (in the following procedure) where fs_wal_dir is also included.

    • Identify and record the port the master is using for RPCs. The default port value is 7051, but it may have been customized using the rpc_bind_addresses configuration parameter.

    • Identify the master’s UUID. It can be fetched using the following command:

      $ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
      master_data_dir

      The location of the existing master’s previously recorded data directory.

      For example:
      $ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
      4aab798a69e94fab8d77069edff28ce0
    • (Optional) Configure a DNS alias for the master. The alias could be a DNS cname (if the machine already has an A record in DNS), an A record (if the machine is only known by its IP address), or an alias in /etc/hosts. The alias should be an abstract representation of the master (e.g. master-1).

  4. Perform the following preparatory steps for each new master:
    • Choose an unused machine in the cluster. The master generates very little load so it can be collocated with other data services or load-generating processes, though not with another Kudu master from the same configuration.

    • Ensure Kudu is installed on the machine, either using system packages (in which case the kudu and kudu-master packages should be installed), or some other means.

    • Choose and record the directory where the master’s data will live.

    • Choose and record the port the master should use for RPCs.

    • (Optional) Configure a DNS alias for the master (e.g. master-2, master-3, etc).

Perform the migration

  1. Stop all the Kudu processes in the entire cluster.
  2. Format the data directory on each new master machine, and record the generated UUID. Use the following commands:
    $ kudu fs format --fs_wal_dir=<master_data_dir>
    $ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
    master_data_dir

    The new master’s previously recorded data directory.

    For example:
    $ kudu fs format --fs_wal_dir=/var/lib/kudu/master
    $ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
    f5624e05f40649b79a757629a69d061e
  3. If you are using Cloudera Manager, add the new Kudu master roles now, but do not start them.
    • If using DNS aliases, override the empty value of the Master Address parameter for each role (including the existing master role) with that master’s alias.

    • Add the port number (separated by a colon) if using a non-default RPC port value.

  4. Rewrite the master’s Raft configuration with the following command, executed on the existing master:
    $ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_data_dir> <tablet_id> <all_masters>
    master_data_dir

    The existing master’s previously recorded data directory

    tablet_id

    This must be set to the string, 00000000000000000000000000000000.

    all_masters

    A space-separated list of masters, both new and existing. Each entry in the list must be a string of the form <uuid>:<hostname>:<port>.

    uuid

    The master’s previously recorded UUID.

    hostname

    The master’s previously recorded hostname or alias.

    port

    The master’s previously recorded RPC port number.

    For example:
    $ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
  5. Modify the value of the master_addresses configuration parameter for both existing master and new masters. The new value must be a comma-separated list of all of the masters. Each entry is a string of the form, <hostname>:<port>.
    hostname

    The master's previously recorded hostname or alias.

    port

    The master's previously recorded RPC port number.

  6. Start the existing master.
  7. Copy the master data to each new master with the following command, executed on each new master machine:
    $ kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <existing_master>
    master_data_dir

    The new master's previously recorded data directory.

    tablet_id

    Must be set to the string, 00000000000000000000000000000000.

    existing_master

    RPC address of the existing master. It must be a string of the form <hostname>:<port>.

    hostname

    The existing master's previously recorded hostname or alias.

    port

    The existing master's previously recorded RPC port number.

    Example
    $ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 master-1:7051
  8. Start all the new masters.
  9. Modify the value of the tserver_master_addrs configuration parameter for each tablet server. The new value must be a comma-separated list of masters where each entry is a string of the form <hostname>:<port>
    hostname

    The master's previously recorded hostname or alias

    port

    The master's previously recorded RPC port number

  10. Start all the tablet servers.
To verify that all masters are working properly, consider performing the following sanity checks:
  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool. For more details, see Monitoring Cluster Health with ksck.

Recovering from a dead Kudu Master in a Multi-Master Deployment

Kudu multi-master deployments function normally in the event of a master loss. However, it is important to replace the dead master; otherwise a second failure may lead to a loss of availability, depending on the number of available masters. This workflow describes how to replace the dead master.

Due to KUDU-1620, it is not possible to perform this workflow without also restarting the live masters. As such, the workflow requires a maintenance window, albeit a potentially brief one if the cluster was set up with DNS aliases.

Prepare for the recovery

  1. If the cluster was configured without DNS aliases perform the following steps. Otherwise move on to step 2:
    1. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster will be unavailable.
    2. Shut down all Kudu tablet server processes in the cluster.
  2. Ensure that the dead master is well and truly dead. Take whatever steps needed to prevent it from accidentally restarting; this can be quite dangerous for the cluster post-recovery.
  3. Choose one of the remaining live masters to serve as a basis for recovery. The rest of this workflow will refer to this master as the "reference" master.
  4. Choose an unused machine in the cluster where the new master will live. The master generates very little load so it can be colocated with other data services or load-generating processes, though not with another Kudu master from the same configuration. The rest of this workflow will refer to this master as the "replacement" master.
  5. Perform the following preparatory steps for the replacement master:
    • Ensure Kudu is installed on the machine, either via system packages (in which case the kudu and kudu-master packages should be installed), or via some other means.

    • Choose and record the directory where the master’s data will live.

  6. Perform the following preparatory steps for each live master:
    • Identify and record the directory where the master’s data lives. If using Kudu system packages, the default value is /var/lib/kudu/master, but it may be customized via the fs_wal_dir and fs_data_dirs configuration parameter. Please note if you’ve set fs_data_dirs to some directories other than the value of fs_wal_dir, it should be explicitly included in every command below where fs_wal_dir is also included.

    • Identify and record the master’s UUID. It can be fetched using the following command:

      $ kudu fs dump uuid --fs_wal_dir=<master_data_dir> 2>/dev/null
      master_data_dir

      live master’s previously recorded data directory

      Example
      $ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
      80a82c4b8a9f4c819bab744927ad765c
  7. Perform the following preparatory steps for the reference master:
    • Identify and record the directory where the master’s data lives. If using Kudu system packages, the default value is /var/lib/kudu/master, but it may be customized using the fs_wal_dir and fs_data_dirs configuration parameter. If you have set fs_data_dirs to some directories other than the value of fs_wal_dir, it should be explicitly included in every command below where fs_wal_dir is also included.

    • Identify and record the UUIDs of every master in the cluster, using the following command:

      $ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_data_dir> <tablet_id> 2>/dev/null
      master_data_dir

      The reference master’s previously recorded data directory.

      tablet_id

      Must be set to the string, 00000000000000000000000000000000.

      Example
      $ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 2>/dev/null
      80a82c4b8a9f4c819bab744927ad765c 2a73eeee5d47413981d9a1c637cce170 1c3f3094256347528d02ec107466aef3
  8. Using the two previously-recorded lists of UUIDs (one for all live masters and one for all masters), determine and record (by process of elimination) the UUID of the dead master.

Perform the recovery

  1. Format the data directory on the replacement master machine using the previously recorded UUID of the dead master. Use the following command sequence:
    $ kudu fs format --fs_wal_dir=<master_data_dir> --uuid=<uuid>
    master_data_dir

    The replacement master’s previously recorded data directory.

    uuid

    The dead master’s previously recorded UUID.

    For example:
    $ kudu fs format --fs_wal_dir=/var/lib/kudu/master --uuid=80a82c4b8a9f4c819bab744927ad765c
  2. Copy the master data to the replacement master with the following command:
    $ kudu local_replica copy_from_remote --fs_wal_dir=<master_data_dir> <tablet_id> <reference_master>
    master_data_dir

    The replacement master’s previously recorded data directory.

    tablet_id

    Must be set to the string, 00000000000000000000000000000000.

    reference_master

    The RPC address of the reference master. It must be a string of the form <hostname>:<port>.

    hostname

    The reference master’s previously recorded hostname or alias.

    port

    The reference master’s previously recorded RPC port number.

    For example:
    $ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 master-2:7051
  3. If you are using Cloudera Manager, add the replacement Kudu master role now, but do not start it.
    • Override the empty value of the Master Address parameter for the new role with the replacement master’s alias.

    • If you are using a non-default RPC port, add the port number (separated by a colon) as well.

    • If the cluster was set up with DNS aliases, reconfigure the DNS alias for the dead master to point at the replacement master.
    • If the cluster was set up without DNS aliases, perform the following steps:
      1. Stop the remaining live masters.
      2. Rewrite the Raft configurations on these masters to include the replacement master. See Step 4 of Perform the Migration for more details.
  4. Start the replacement master.
  5. Restart the remaining masters in the new multi-master deployment. While the masters are shut down, there will be an availability outage, but it should last only as long as it takes for the masters to come back up.
To verify that all masters are working properly, consider performing the following sanity checks:
  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool. For more details, see Monitoring Cluster Health with ksck.

Removing Kudu Masters from a Multi-Master Deployment

In the event that a multi-master deployment has been overallocated nodes, the following steps should be taken to remove the unwanted masters.

Prepare for removal

  1. Establish a maintenance window (one hour should be sufficient). During this time the Kudu cluster will be unavailable.
  2. Identify the UUID and RPC address current leader of the multi-master deployment by visiting the /masters page of any master’s web UI. This master must not be removed during this process; its removal may result in severe data loss.
  3. Stop all the Kudu processes in the entire cluster.
  4. If you are using Cloudera Manager, remove the unwanted Kudu master from your cluster's Kudu service.

Perform the removal

  1. Rewrite the Raft configuration on the remaining masters to include only the remaining masters. See Step 4 of Perform the Migration for more details.
  2. Remove the data directories and WAL directory on the unwanted masters. This is a precaution to ensure that they cannot start up again and interfere with the new multi-master deployment.
  3. Modify the value of the master_addresses configuration parameter for the masters of the new multi-master deployment. If migrating to a single-master deployment, the master_addresses flag should be omitted entirely.
  4. Start all of the masters that were not removed.
  5. Modify the value of the tserver_master_addrs configuration parameter for the tablet servers to remove any unwanted masters.
  6. Start all of the tablet servers.
To verify that all masters are working properly, consider performing the following sanity checks:
  • Using a browser, visit each master’s web UI and navigate to the /masters page. All the masters should now be listed there with one master in the LEADER role and the others in the FOLLOWER role. The contents of /masters on each master should be the same.

  • Run a Kudu system check (ksck) on the cluster using the kudu command line tool. For more details, see Monitoring Cluster Health with ksck.

Monitoring Cluster Health with ksck

The kudu CLI includes a tool called ksck which can be used for monitoring cluster health and data integrity. ksck will identify issues such as under-replicated tablets, unreachable tablet servers, or tablets without a leader.

ksck should be run from the command line, and requires you to specify the complete list of Kudu master addresses:

kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com

To see the full list of the options available with ksck, either use the --help flag or visit the Kudu command line reference documentation.

If the cluster is healthy, ksck will print a success message, and return a zero (success) exit status.

Connected to the Master
Fetched info from all 1 Tablet Servers
Table IntegrationTestBigLinkedList is HEALTHY (1 tablet(s) checked)

The metadata for 1 table(s) is HEALTHY
OK

If the cluster is unhealthy, for instance if a tablet server process has stopped, ksck will report the issue(s) and return a non-zero exit status:

Connected to the Master
WARNING: Unable to connect to Tablet Server 8a0b66a756014def82760a09946d1fce
(tserver-01.example.com:7050): Network error: could not send Ping RPC to server: Client connection negotiation failed: client connection to 192.168.0.2:7050: connect: Connection refused (error 61)
WARNING: Fetched info from 0 Tablet Servers, 1 weren't reachable
Tablet ce3c2d27010d4253949a989b9d9bf43c of table 'IntegrationTestBigLinkedList'
is unavailable: 1 replica(s) not RUNNING
  8a0b66a756014def82760a09946d1fce (tserver-01.example.com:7050): TS unavailable [LEADER]
  Table IntegrationTestBigLinkedList has 1 unavailable tablet(s)

  : 1 out of 1 table(s) are not in a healthy state
  ==================
  Errors:
  ==================
  error fetching info from tablet servers: Network error: Not all Tablet Servers are reachable
  table consistency check error: Corruption: 1 table(s) are bad

  FAILED
  Runtime error: ksck discovered errors

To verify data integrity, the optional --checksum-scan flag can be set, which will ensure that the cluster has consistent data by scanning each tablet replica and comparing results. The --tables and --tablets flags can be used to limit the scope of the checksum scan to specific tables or tablets, respectively.

For example, use the following command to check the integrity of data in the IntegrationTestBigLinkedList table:

kudu cluster ksck --checksum-scan --tables IntegrationTestBigLinkedList master-01.example.com,master-02.example.com,master-03.example.com

Changing Directory Configuration

Kudu does not allow for the addition or removal of directories on existing master or tablet servers. In order to start a server with a different directory configuration from what it was created with, the server needs to be rebuilt.

The first step to starting up a server with a new directory configuration is emptying all of the server’s existing directories. For example, if a tablet server is configured with --fs_wal_dir=/data/0/kudu-tserver-wal and --fs_data_dirs=/data/1/kudu-tserver,/data/2/kudu-tserver, the following commands will remove the write-ahead-log (WAL) directory’s and data directories' contents:

rm -rf /data/0/kudu-tserver-wal/* /data/1/kudu-tserver/* /data/2/kudu-tserver/*

After the WAL and data directories are emptied, and any new directories are created with the appropriate permissions, the server process can be started with the new directory configuration. When Kudu is installed using system packages, service is typically used as follows:

sudo service kudu-tserver start

Recovering from Disk Failure

Kudu tablet servers are not resilient to disk failure. When a disk containing a data directory or WAL fails, the server will crash, and the entire server must be rebuilt. Kudu will automatically re-replicate tablets on other servers after a tablet server fails, but manual intervention is needed in order to restore the failed tablet server to a running state.

To rebuild the tablet server after a disk failure, the failed disk needs to be replaced or removed from the data-directory and/or WAL configuration. Once this is complete the server must be rebuilt with this new configuration. See the section on Changing Directory Configuration for more details.

Scaling Storage on Kudu Master and Tablet Servers in the Cloud

If you find that the size of your Kudu cloud deployment has exceeded previous expectations, or you simply wish to allocate more storage to Kudu, use the following set of high-level steps as a guide to increasing storage on your Kudu master or tablet server hosts. You must work with your cluster's Hadoop administrators and the system administrators to complete this process. Replace the file paths in the following steps to those relevant to your setup.

  1. Run a consistency check on the cluster hosts. For instructions, see Monitoring Cluster Health with ksck.
  2. On all Kudu hosts, create a new file system with the storage capacity you require. For example, /new/data/dir.
  3. Shutdown cluster services. For a cluster managed by Cloudera Manager cluster, see Starting and Stopping a Cluster.
  4. Copy the contents of your existing data directory, /current/data/dir, to the new filesystem at /new/data/dir.
  5. Move your existing data directory, /current/data/dir, to a separate temporary location such as /tmp/data/dir.
  6. Create a new /current/data/dir directory.
    mkdir /current/data/dir
  7. Mount /new/data/dir as /current/data/dir. Make changes to fstab as needed.
  8. Perform steps 4-7 on all Kudu hosts.
  9. Startup cluster services. For a cluster managed by Cloudera Manager cluster, see Starting and Stopping a Cluster.
  10. Run a consistency check on the cluster hosts. For instructions, see Monitoring Cluster Health with ksck.
  11. After 10 days, if everything is in working order on all the hosts, get approval from the Hadoop administrators to remove the /backup/data/dir directory.