Apache Kudu Known Issues

Kudu Masters unable to join back after a restart

In a multi master Kudu environment, if a master is restarted or goes offline for a few minutes, it can occasionally have trouble joining the cluster on startup. For example, if this happens in case of three kudu masters, and one of the other two masters is stopped or dies during this time, then the overall Kudu cluster is down because the majority of the masters are not running.

This issue is resolved by the KUDU-2748 upstream JIRA.

Products affected: Apache Kudu

Affected version:
  • CDH 5.14.0, 5.14.2, 5.14.4
  • CDH 5.15.0, 5.15.1, 5.15.2
  • CDH 5.16.1, 5.16.2
  • CDH 6.0.0, 6.0.1
  • CDH 6.1.0, 6.1.1
  • CDH 6.2.0, 6.2.1
Fixed version:
  • CDH 6.3.0

For the latest update on this issue see the corresponding Knowledge article:TSB 2020-442: Kudu Masters unable to join back after a restart

Inconsistent rows returned from queries in Kudu

Due to KUDU-2463, upon restarting Kudu, inconsistent rows may be returned from tables that have not recently been written to, resulting in any of the following:

  • multiple rows for the same key being returned
  • deleted data being returned
  • inconsistent results consistently being returned for the same query

If this happens, you have two options to resolve the conflicts: write to the affected Kudu partitions by:

  • re-deleting the known and deleted data
  • upserting the most up-to-date version of affected rows.

Products affected: Apache Kudu

Releases affected:
  • CDH 5.12.2, 5.13.3, 5.14.4, 5.15.1, 5.16.1
  • CDH 6.0.1, 6.1.0, 6.1.1

For the latest update on this issue see the corresponding Knowledge article:TSB 2019-353: Inconsistent rows returned from queries in Kudu

CFile Checksum Failure Causes Query to Fail

When a CFile checksum fails, for example, due to a underlying disk corruption, queries against the replica will fail with an error message, such as this:
Unable to advance iterator: Corruption: checksum error on CFile block

Workaround: Remove the corrupted replica from the tablet's Raft configuration. See Kudu Troubleshooting Guide for the detailed steps.

Affected Versions: CDH 6.0.x and lower

Apache Issue: KUDU-2469

The Kudu CLI tool crashes before commencing cluster rebalancing

The Kudu CLI crashes when running the 'kudu cluster rebalance' sub-command on the following platforms:
  • RHEL/CentOS 7
  • Ubuntu14.04 LTS (Trusty)
  • SLES12

Affected Versions: CDH 5.16.0, CDH 5.16.1

Workaround: Specify an additional run-time flag --move_single_replicas=disabled when running the kudu cluster rebalance command. Specifying that flag explicitly disables balancing of non-replicated tables.

NOTE: As a good practice, non-replicated tables should not be used in production Kudu clusters.

Addressed in releases: CDH 5.16.2

Crash due to CHECK failure during compactions: pv_delete_redo != nullptr

Due to KUDU-2233, tablet servers might crash on restart with a fatal log message like the following:

F1201 14:55:37.052140 10508 compaction.cc:756] Check failed: pv_delete_redo != nullptr
 *
 **
 *** Check failure stack trace: ***
 Wrote minidump to /var/log/kudu/minidumps/kudu-tserver/215cde39-7795-0885-0b51038d-771d875e.dmp
 *** Aborted at 1512161737 (unix time) try "date -d @1512161737" if you are using GNU date ***
 PC: @ 0x3ec3632625 (unknown)
 *** SIGABRT (@0x3b98eec0000028e3) received by PID 10467 (TID 0x7f8b02c58700) from PID 10467; stack trace: ***
@ 0x3ec3a0f7e0 (unknown)
@ 0x3ec3632625 (unknown)
@ 0x3ec3633e05 (unknown)
0x1b53f59 (unknown)
@ 0x8b9f6d google::LogMessage::Fail()
@ 0x8bbe2d google::LogMessage::SendToLog()
@ 0x8b9aa9 google::LogMessage::Flush()
@ 0x8bc8cf google::LogMessageFatal::~LogMessageFatal()
@ 0x9db0fe kudu::tablet::FlushCompactionInput()
@ 0x9a056a kudu::tablet::Tablet::DoMergeCompactionOrFlush()
@ 0x9a372d kudu::tablet::Tablet::Compact()
@ 0x9bd8d1 kudu::tablet::CompactRowSetsOp::Perform()
@ 0x1b4145f kudu::MaintenanceManager::LaunchOp()
@ 0x1b8da06 kudu::ThreadPool::DispatchThread()
@ 0x1b888ea kudu::Thread::SuperviseThread()
@ 0x3ec3a07aa1 (unknown)
@ 0x3ec36e893d (unknown)
@ 0x0 (unknown)}}

After this crash occurs, the affected tablet server requires a patch in order to start again.

This crash is much more likely to occur on workloads involving deleting and re-inserting the same rows repeatedly.

Although data loss has not been observed as a result of this issue, data loss or corruption is possible. The integrity of a Kudu cluster’s data can be verified using a ksck checksum scan:
sudo -u kudu kudu cluster ksck -checksum_scan <master addresses>

Products affected: Kudu

Releases affected:
  • CDH 5.12.0, 5.12.1, 5.12.2
  • CDH 5.13.0, CDH 5.13.1, CDH 5.13.2
  • CDH 5.14.0, 5.14.1

Users affected: All users running Kudu, but the risk is much greater for workloads that involve deleting and re-inserting rows repeatedly.

Severity: High

Impact: The affected tablet server will not be able to start until a patch is applied. Data loss or corruption is possible.

Immediate action required: Upgrade to a version containing a fix.

If this is not possible, raising the minimum number of log segments retained to 2 will make the problem very unlikely to occur. To do this, go to the configuration page for Kudu and add --log_min_segments_to_retain=2 to the Kudu Service Advanced Configuration Snippet (Safety Valve) for gflagfile option.

If the issue still occurs, it’s necessary to patch the affected tablet server or upgrade to a version containing a fix.

Once upgraded to a version containing a fix, the --log_min_segments_to_retain setting should be removed.

Addressed in release: CDH5.13.3, CDH5.14.2

Partitions may be pruned incorrectly when range-partitioned on a primary key prefix

Without the fix from KUDU-2173, the Kudu C++ client’s partition pruner may incorrectly prune a partition when evaluating certain queries containing predicates on both the range partition columns and other columns that are part of the primary key. This may cause queries to fail to return some matching rows.

Specifically, the pruner mistakenly treats a range partition which is a proper prefix of a primary key as an exclusive bound, when in fact it is an inclusive bound if the remaining primary key column constraints are greater than the minimum value.

For example, given a simple schema:
CREATE TABLE t ( 
a INT NOT NULL,
b INT NOT NULL,
PRIMARY KEY (a,b)
) PARTITION BY RANGE(a) PARTITIONS (
...
)
With the following range partition:
RANGE (a) PARTITION VALUES >= 10
And the following query:
SELECT * FROM t WHERE a < 11 AND b < 11

The partition pruner would exclude the above partition, even though it contains rows like (10, 1). Instead, the partition pruner treats >= 10 as > 10. This wouldn't be a problem if the range partition was on column (b) instead, since that is not a prefix of the primary key. The same applies for (a, b), since that is the whole primary key and not a prefix.

Products affected: Kudu

Releases affected:

  • CDH 5.10.0, 5.10.1, 5.10.2
  • CDH 5.11.0, 5.11.1, 5.11.2
  • CDH 5.12.0, 5.12.1
  • CDH 5.13.0

Users affected: Direct Kudu users via its C++ or Python client. Indirect users via Impala.

Severity (Low/Medium/High): High

Impact: Certain queries won’t return all the available matching rows and no error message will be displayed.

Immediate action required: Assess if any tables are currently affected by this bug.
  1. Download the tool at https://www.cloudera.com/content/dam/www/Support/Downloads/tsb-257.tar.gz.
  2. Extract the tool:
    tar xzvf tsb-257.tar.gz
    Run it:
    java -jar tsb-257-check-0.1.0.jar <master-addresses> [table-name]
  3. Capture the output. Depending on the output:
    • If no tables are affected, ensure no new table is created with range partitioning on a proper prefix of the primary key until a new version is available for upgrade.
    • If at least one table is affected, contact Cloudera Support to understand the next steps.
Addressed in release/refresh/patch:
  • CDH 5.12.2
  • CDH 5.13.1
  • CDH 5.14.0
  • CDH 5.15.0

C++ Client Fails to Re-acquire Authentication Token in Multi-master Clusters

A security-related issue can cause Impala queries to start failing on busy clusters in the following scenario:

  • The cluster runs with the --rpc_authentication set as optional or required. The default is optional. Secure clusters use required.
  • The cluster is using multiple masters.
  • Impala queries happen frequently enough that the leader master connection to some impalad isn't idle-closed (more than 1 query per 65 seconds).
  • The connection stays alive for longer than the authentication token timeout (1 week by default).
  • A master leadership change occurs after the authentication token expiration.
Impala queries will start failing with errors in the impalad logs like:
I0904 13:53:08.748968 95857 client-internal.cc:283] Unable to determine the new leader Master: Not authorized: Client connection negotiation failed: client connection to 10.164.44.13:7051: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: authentication token expired
I0904 13:53:10.389009 95861 status.cc:125] Unable to open Kudu table: Timed out: GetTableSchema timed out after deadline expired
 @ 0x95b1e9 impala::Status::Status()
 @ 0xff22d4 impala::KuduScanNodeBase::Open()
 @ 0xff101e impala::KuduScanNode::Open()
 @ 0xb73ced impala::FragmentInstanceState::Open()
 @ 0xb7532b impala::FragmentInstanceState::Exec()
 @ 0xb64ae8 impala::QueryState::ExecFInstance()
 @ 0xd15193 impala::Thread::SuperviseThread()
 @ 0xd158d4 boost::detail::thread_data<>::run()
 @ 0x129188a (unknown)
 @ 0x7f717ceade25 start_thread
 @ 0x7f717cbdb34d __clone
Impala shell queries will fail with a message like:
Unable to open Kudu table: Timed out: GetTableSchema timed out after deadline expired
Workaround:
  • Restart the affected Impala Daemons. Restarting a daemon ensures the problem will not reoccur for at least the authentication token lifetime, which defaults to one week.
  • Increase the authentication token lifetime (--authn_token_validity_seconds). Beware that raising this lifetime increases the window of vulnerability of the cluster if a client is compromised. It is recommended that you keep the token lifetime at one month maximum for a secure cluster. For unsecured clusters, a longer token lifetime is acceptable, and a 3 month lifetime is recommended.

Affected Versions: From CDH 5.11 through CDH 6.0

Fixed Versions: CDH 5.15.2, CDH 5.16.1

Apache Issue: KUDU-2580

Timeout Possible with Log Force Synchronization Option

If the Kudu master is configured with the -log_force_fsync_all option, tablet servers and clients will experience frequent timeouts, and the cluster may become unusable.

Longer Startup Times with a Large Number of Tablets

If a tablet server has a very large number of tablets, it may take several minutes to start up. It is recommended to limit the number of tablets per server to 1000 or fewer. The maximum allowed number of tablets is 2000 per server. Consider this limitation when pre-splitting your tables. If you notice slow start-up times, you can monitor the number of tablets per server in the web UI.

Kerberos Authentication Issues

  • Kerberos authentication does not function correctly on hosts which contain capital letters in their hostname.

  • Kerberos authentication does not function correctly if rdns = false is configured in krb5.conf.

Memory Issue with Fault Tolerant Scans

Unlike regular scans, fault tolerant scans will allocate all required memory when the scan begins rather than as it progresses. This can be significant for big tablets. Moreover, this memory usage isn't counted towards the tablet server's overall memory limit, raising the likelihood of the tablet server being out-of-memory killed by the kernel.

Affected Versions: CDH 6.2 / Kudu 1.9 and lower

Apache Issue: KUDU-2466

Confusing Descriptions for Kudu TLS/SSL Settings in Cloudera Manager

Descriptions in the Cloudera Manager Admin Console for TLS/SSL settings are confusing, and will be replaced in a future release. The correct information about the settings is in the Usage Notes column:

Field Usage Notes
Kerberos Principal Set to the default principal, kudu.
Enable Secure Authentication And Encryption Select this checkbox to enable authentication and RPC encryption between all Kudu clients and servers, as well as between individual servers. Only enable this property after you have configured Kerberos.
Master TLS/SSL Server Private Key File (PEM Format) Set to the path containing the Kudu master host's private key (PEM-format). This is used to enable TLS/SSL encryption (over HTTPS) for browser-based connections to the Kudu master web UI.
Tablet Server TLS/SSL Server Private Key File (PEM Format) Set to the path containing the Kudu tablet server host's private key (PEM-format). This is used to enable TLS/SSL encryption (over HTTPS) for browser-based connections to Kudu tablet server web UIs.
Master TLS/SSL Server Certificate File (PEM Format) Set to the path containing the signed certificate (PEM-format) for the Kudu master host's private key (set in Master TLS/SSL Server Private Key File). The certificate file can be created by concatenating all the appropriate root and intermediate certificates required to verify trust.
Tablet Server TLS/SSL Server Certificate File (PEM Format) Set to the path containing the signed certificate (PEM-format) for the Kudu tablet server host's private key (set in Tablet Server TLS/SSL Server Private Key File). The certificate file can be created by concatenating all the appropriate root and intermediate certificates required to verify trust.
Master TLS/SSL Server CA Certificate (PEM Format) Disregard this field.
Tablet Server TLS/SSL Server CA Certificate (PEM Format) Disregard this field.
Enable TLS/SSL for Master Server Enables HTTPS encryption on the Kudu master web UI.
Enable TLS/SSL for Tablet Server Enables HTTPS encryption on the Kudu tablet server web UIs.