Apache Hadoop Known Issues

This page includes known issues and related topics, including:

Deprecated Properties

Several Hadoop and HDFS properties have been deprecated as of Hadoop 2.0.0 (Hadoop 0.23.1, CDH 4 Beta) and later. For details, see https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/DeprecatedProperties.html.

HDFS

Potential Block Corruption and Data Loss During Pipeline Recovery

A bug in the HDFS block pipeline recovery code can cause blocks to be unrecoverable due to miscalculation of the block checksum. On a busy cluster where data is written and flushed frequently, when a write pipeline recovery occurs, a node newly added to the write pipeline may calculate the checksum incorrectly. This miscalculation is very rare, but when it does occur, the replica becomes corrupted and data can be lost if all replicas are simultaneously affected.

Bug: HDFS-4660 , HDFS-9220

Detecting this known issue requires correlating multiple log messages. Below is an example DataNode log error message captured at the time of block creation:
java.io.IOException: Terminating due to a checksum error.java.io.IOException:
Unexpected checksum mismatch while writing
BP-1800173197-10.x.y.z-1444425156296:blk_1170125248_96458336 from /10.x.y.z
This issue affects these versions of CDH:
  • CDH 5.0.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6
  • CDH 5.1.0, 5.1.2, 5.1.3, 5.1.4, 5.1.5
  • CDH 5.2.0, 5.2.1, 5.2.3, 5.2.4, 5.2.5, 5.2.6
  • CDH 5.3.0, 5.3.1, 5.3.2, 5.3.3, 5.3.4, 5.3.5, 5.3.7, 5.3.8, 5.3.9, 5.3.10
  • CDH 5.4.0, 5.4.1, 5.4.2, 5.4.3, 5.4.4, 5.4.5, 5.4.7, 5.4.8, 5.4.9, 5.4.10
  • CDH 5.5.0, 5.5.1
Workaround: None. Upgrade to a CDH version that includes fixes for HDFS-4660 and HDFS-9220.

Users affected: All users running the affected CDH versions and using the HDFS file system.

Severity (Low/Medium/High): High

Impact: Potential loss of block data.

Immediate action required: Upgrade to a CDH version that includes the fix, specifically:
  • CDH 5.4.11, CDH 5.5.2, CDH 5.6.0 and higher

DiskBalancer Occasionally Emits False Error Messages

Diskbalancer occasionally emits false error messages. For example:

2016-08-03 11:01:41,788 ERROR org.apache.hadoop.hdfs.server.datanode.DiskBalancer:
Disk Balancer is not enabled.

You can safely ignore this error message if you are not using DiskBalancer.

Workaround: Use the following command against all DataNodes to suppress DiskBalancer logs:

hadoop daemonlog -setlevel <host:port> org.apache.hadoop.hdfs.server.datanode.DiskBalancer FATAL 

Another workaround is to supress the warning by setting the log level of DiskBalancer to FATAL. Add the following to log4j.properties (DataNode Logging Advanced Configuration Snippet (Safety Valve)) and restart your DataNodes:

log4j.logger.org.apache.hadoop.hdfs.server.datanode.DiskBalancer = FATAL      

Upgrade Requires an HDFS Upgrade

Upgrading from any release earlier than CDH 5.2.0 to CDH 5.2.0 or later requires an HDFS Upgrade.

See Upgrading Unmanaged CDH Using the Command Line for further information.

Optimizing HDFS Encryption at Rest Requires Newer openssl Library on Some Systems

CDH 5.3 implements the Advanced Encryption Standard New Instructions (AES-NI), which provide substantial performance improvements. To get these improvements, you need a recent version of libcrypto.so on HDFS and MapReduce client hosts that is, any host from which you originate HDFS or MapReduce requests. Many OS versions have an older version of the library that does not support AES-NI.

See HDFS Transparent Encryption in the Encryption section of the Cloudera Security guide for instructions for obtaining the right version.

Other HDFS Encryption Known Issues

Potentially Incorrect Initialization Vector Calculation in HDFS Encryption

A mathematical error in the calculation of the Initialization Vector (IV) for encryption and decryption in HDFS could cause data to appear corrupted when read. The IV is a 16-byte value input to encryption and decryption ciphers. The calculation of the IV implemented in HDFS was found to be subtly different from that used by Java and OpenSSL cryptographic routines. The result is that data could possibly appear to be corrupted when it is read from a file inside an Encryption Zone.

Fortunately, the probability of this occurring is extremely small. For example, the maximum size of a file in HDFS is 64 TB. This enormous file would have a 1-in-4- million chance of hitting this condition. A more typically sized file of 1 GB would have a roughly 1-in-274-billion chance of hitting the condition.

Workaround: If you are using the experimental HDFS encryption feature in CDH 5.2, upgrade to CDH 5.3 and verify the integrity of all files inside an Encryption Zone.

DistCp between unencrypted and encrypted locations fails

By default, DistCp compares checksums provided by the filesystem to verify that data was successfully copied to the destination. However, when copying between unencrypted and encrypted locations, the filesystem checksums will not match since the underlying block data is different.

Workaround: Specify the -skipcrccheck and -update distcp flags to avoid verifying checksums.

Cannot move encrypted files to trash

With HDFS encryption enabled, you cannot move encrypted files or directories to the trash directory.

Bug: HDFS-6767

Workaround: To remove encrypted files/directories, use the following command with the -skipTrash flag specified to bypass trash.
rm -r -skipTrash /testdir

HDFS NFS gateway and CDH installation (using packages) limitation

HDFS NFS gateway works as shipped ("out of the box") only on RHEL-compatible systems, but not on SLES, Ubuntu, or Debian. Because of a bug in native versions of portmap/rpcbind, the HDFS NFS gateway does not work out of the box on SLES, Ubuntu, or Debian systems when CDH has been installed from the command-line, using packages. It does work on supported versions of RHEL-compatible systems on which rpcbind-0.2.0-10.el6 or later is installed, and it does work if you use Cloudera Manager to install CDH, or if you start the gateway as root. For more information, see supported versions

Bug: 731542 (Red Hat), 823364 (SLES), 594880 (Debian)

Workarounds and caveats:
  • On Red Hat and similar systems, make sure rpcbind-0.2.0-10.el6 or later is installed.
  • On SLES, Debian, and Ubuntu systems, do one of the following:
    • Install CDH using Cloudera Manager; or
    • As of CDH 5.1, start the NFS gateway as root; or
    • Start the NFS gateway without using packages; or
    • You can use the gateway by running rpcbind in insecure mode, using the -i option, but keep in mind that this allows anyone from a remote host to bind to the portmap.

HDFS does not currently provide ACL support for the HDFS gateway

Bug: HDFS-6949

No error when changing permission to 777 on .snapshot directory

Snapshots are read-only; running chmod 777 on the .snapshots directory does not change this, but does not produce an error (though other illegal operations do).

Bug: HDFS-4981

Workaround: None

Snapshot operations are not supported by ViewFileSystem

Bug: None

Workaround: None

Snapshots do not retain directories' quotas settings

Bug: HDFS-4897

Workaround: None

Permissions for dfs.namenode.name.dir incorrectly set.

Hadoop daemons should set permissions for the dfs.namenode.name.dir (or dfs.name.dir) directories to drwx------ (700), but in fact these permissions are set to the file-system default, usually drwxr-xr-x (755).

Bug: HDFS-2470

Workaround: Use chmod to set permissions to 700. See Configuring Local Storage Directories for Use by HDFS for more information and instructions.

hadoop fsck -move does not work in a cluster with host-based Kerberos

Bug: None

Workaround: Use hadoop fsck -delete

HttpFS cannot get delegation token without prior authenticated request

A request to obtain a delegation token cannot initiate an SPNEGO authentication sequence; it must be accompanied by an authentication cookie from a prior SPNEGO authentication sequence.

Bug: HDFS-3988

Workaround: Make another WebHDFS request (such as GETHOMEDIR) to initiate an SPNEGO authentication sequence and then make the delegation token request.

DistCp does not work between a secure cluster and an insecure cluster in some cases

See the upstream bug reports for details.

Bug: HDFS-7037, HADOOP-10016, HADOOP-8828

Workaround: None

Port configuration required for DistCp to Hftp from secure cluster (SPNEGO)

To copy files using DistCp to Hftp from a secure cluster using SPNEGO, you must configure the dfs.https.port property on the client to use the HTTP port (50070 by default).

Bug: HDFS-3983

Workaround: Configure dfs.https.port to use the HTTP port on the client

Non-HA DFS Clients do not attempt reconnects

This problem means that streams cannot survive a NameNode restart or network interruption that lasts longer than the time it takes to write a block.

Bug: HDFS-4389

DataNodes may become unresponsive to block creation requests

DataNodes may become unresponsive to block creation requests from clients when the directory scanner is running.

Bug: HDFS-7489

Workaround: Disable the directory scanner by setting dfs.datanode.directoryscan.interval to -1.

The active NameNode will not accept an fsimage sent from the standby during rolling upgrade

The result is that the NameNodes fail to checkpoint until the upgrade is finalized.

Bug: HDFS-7185

Workaround: None.

Block report can exceed maximum RPC buffer size on some DataNodes

On a DataNode with a large number of blocks, the block report may exceed the maximum RPC buffer size.

Bug: None

Workaround: Increase the value ipc.maximum.data.length in hdfs-site.xml:
<property>
  <name>ipc.maximum.data.length</name>
  <value>268435456</value>
</property>

DistCp to S3a fails due to integer overflow in retry timer

Writing to Amazon S3 under high load can cause com.amazonaws.AmazonClientException: Unable to complete transfer: timeout value is negative.

Bug: HADOOP-12267

Workaround: Reduce the load to S3 by reducing the number of reducers or mappers.

MapReduce, YARN

NodeManager mount point mismatch (YARN)

NodeManager may select a cgroups (Linux control groups) mount point that is not accessible to user yarn, resulting in failure to start up. The mismatch occurs because YARN uses cgroups in mount point /run/lxcfs/controllers, while Cloudera Manager typically configures cgroups at /sys/fs/cgroups. This issue has occurred on Ubuntu 16.04 systems.

Bug: CDH-52263

Workaround: Look through your YARN logs to identify the failing mount point, and unmount it:
$ umount errant_mount_point
You can also:
  1. apt-get remove lxcfs
  2. Reboot the node

Hadoop Pipes should not be used in secure clusters

Hadoop Pipes should not be used in secure clusters. A shared password used by the framework for parent-child communications in the clear. A malicious user could intercept that password and potentially use it to access private data in a running application.

JobHistory URL mismatch after server relocation

After moving the JobHistory Server to a new host, the URLs listed for the JobHistory Server on the ResourceManager web UI still point to the old JobHistory Server. This affects existing jobs only. New jobs started after the move are not affected.

Workaround: For any existing jobs that have the incorrect JobHistory Server URL, there is no option other than to allow the jobs to roll off the history over time. For new jobs, make sure that all clients have the updated mapred-site.xml that references the correct JobHistory Server.

Starting an unmanaged ApplicationMaster may fail

Starting a custom Unmanaged ApplicationMaster may fail due to a race in getting the necessary tokens.

Bug: YARN-1577

Workaround: Try to get the tokens again; the custom unmanaged ApplicationMaster should be able to fetch the necessary tokens and start successfully.

Moving jobs between queues not persistent after restart

CDH 5 adds the capability to move a submitted application to a different scheduler queue. This queue placement is not persisted across ResourceManager restart or failover, which resumes the application in the original queue.

Bug: YARN-1558

Workaround: After ResourceManager restart, re-issue previously issued move requests.

No JobTracker becomes active if both JobTrackers are migrated to other hosts

If JobTrackers in an High Availability configuration are shut down, migrated to new hosts, then restarted, no JobTracker becomes active. The logs show a Mismatched address exception.

Bug: None

Workaround: After shutting down the JobTrackers on the original hosts, and before starting them on the new hosts, delete the ZooKeeper state using the following command:
$ zkCli.sh rmr /hadoop-ha/<logical name>

Hadoop Pipes may not be usable in an MRv1 Hadoop installation done through tarballs

Under MRv1, MapReduce's C++ interface, Hadoop Pipes, may not be usable with a Hadoop installation done through tarballs unless you build the C++ code on the operating system you are using.

Bug: None

Workaround: Build the C++ code on the operating system you are using. The C++ code is present under src/c++ in the tarball.

Display may misrepresent task completion

Task-completed percentage may report slightly under 100% in the web UI despite all job-related tasks completing successfully.

Bug: None

Workaround: None

Oozie workflows not recovered after JobTracker failover on a secure cluster

Delegation tokens created by clients (via JobClient#getDelegationToken()) do not persist when the JobTracker fails over. This limitation means that Oozie workflows will not be recovered successfully in the event of a failover on a secure cluster.

Bug: None

Workaround: Re-submit the workflow.

Encrypted shuffle may fail (MRv2, Kerberos, TLS)

In MRv2, if the LinuxContainerExecutor is used (usually as part of Kerberos security), and hadoop.ssl.enabled is set to true (see Configuring Encrypted Shuffle, Encrypted Web UIs, and Encrypted HDFS Transport), then the encrypted shuffle does not work and the submitted job fails.

Bug: MAPREDUCE-4669

Workaround: Use encrypted shuffle with Kerberos security without encrypted web UIs, or use encrypted shuffle with encrypted web UIs without Kerberos security.

ResourceManager-to-Application Master HTTPS link fails

In MRv2 (YARN), if hadoop.ssl.enabled is set to true (use HTTPS for web UIs), then the link from the ResourceManager to the running MapReduce Application Master fails with an HTTP Error 500 because of a PKIX exception.

A job can still be run successfully, and, when it finishes, the link to the job history does work.

Bug: YARN-113

Workaround: Do not use encrypted web UIs.

Hadoop client JARs do not include all classes needed for client-code compile

The compile does succeed, but you may see warnings as in the following example:
 $ javac -cp '/usr/lib/hadoop/client/*' -d wordcount_classes WordCount.java
org/apache/hadoop/fs/Path.class(org/apache/hadoop/fs:Path.class): warning: Cannot find annotation method 'value()'
in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
1 warning 

Bug:None

Workaround: None

Misapplied user-limits setting possible

The ulimits setting in /etc/security/limits.conf is applied to the wrong user when security is enabled.

Bug: https://issues.apache.org/jira/browse/DAEMON-192

Anticipated Resolution: None

Workaround: To increase the ulimits applied to DataNodes, you must change the ulimit settings for the root user, not the hdfs user.

Routable IP address required by ResourceManager

ResourceManager requires routable host:port addresses for yarn.resourcemanager.scheduler.address, and does not support using the wildcard 0.0.0.0 address.

Bug: None

Workaround: Set the address, in the form host:port, either in the client-side configuration, or on the command line when you submit the job.

Amazon S3 copy may time out

The Amazon S3 filesystem does not support renaming files, and performs a copy operation instead. If the file to be moved is very large, the operation can time out because S3 does not report progress to the TaskTracker during the operation.

Bug: MAPREDUCE-972

Workaround: Use -Dmapred.task.timeout=15000000 to increase the MR task timeout.

Task controller for MapReduce changes in CDH5

In CDH 5, the MapReduce task controller is changed from DefaultTaskController to LinuxTaskController. The new task controller has different directory ownership requirements which can cause jobs to fail. You can switch back to DefaultTaskController by adding the following to the MapReduce Advanced Configuration Snippet if you use Cloudera Manager, or directly to mapred-default.xml otherwise.
<property>
  <name>mapreduce.tasktracker.taskcontroller</name>
  <value>org.apache.hadoop.mapred.DefaultTaskController</value>
</property>

Out-of-memory errors may occur with Oracle JDK 1.8

The total JVM memory footprint for JDK8 can be larger than that of JDK7 in some cases. This may result in out-of-memory errors.

Bug: None

Workaround: Increase max default heap size (-Xmx). In the case of MapReduce, for example, increase Reduce Task Maximum Heap Size in Cloudera Manager (mapred.reduce.child.java.opts, or mapreduce.reduce.java.opts for YARN) to avoid out-of-memory errors during the shuffle phase.

MapReduce JAR file renamed (CDH 5.4.0)

As of CDH 5.4.0, hadoop-test.jar has been renamed to hadoop-test-mr1.jar. This JAR file contains the mrbench, TestDFSIO, and nnbench tests.

Bug: None

Workaround: None.

Jobs in pool with DRF policy will not run if root pool is FAIR

If a child pool using DRF policy has a parent pool using Fairshare policy, jobs submitted to the child pool do not run.

Bug: YARN-4212

Workaround:Change parent pool to use DRF.

Large TeraValidate data sets can fail with MapReduce

In a cluster using MR1 as the MapReduce service (JobTracker/TaskTracker), TeraValidate fails when run over large TeraGen/TeraSort data sets (1TB and larger) with an IndexOutOfBoundsException. Smaller data sets do not show this issue.

Bug: MAPREDUCE-6481

Workaround:Use YARN as the MapReduce service.

FairScheduler might not Assign Containers

Under certain circumstances, turning on Fair Scheduler Assign Multiple Tasks (yarn.scheduler.fair.assignmultiple) causes the scheduler to stop assigning containers to applications. Possible symptoms are that running applications show no progress, and new applications do not start, staying in an Assigned state, despite the availability of free resources on the cluster.

Bug: YARN-4477

Workaround: Turn off Fair Scheduler Assign Multiple Tasks (yarn.scheduler.fair.assignmultiple) and restart the ResourceManager.

Jobs with encrypted spills do not recover if the AM goes down

The fix to CVE-2015-1776 leads to not having enough information to recover a job should the Application Master fail. Releases with this security fix cannot tolerate Application Master failures.

Bug: MAPREDUCE-6638

Workaround: None. Fix to come in a later release.

FairScheduler: AMs can consume all vCores leading to a livelock

When using FAIR policy with the FairScheduler, Application Masters can consume all vCores which may lead to a livelock.

Bug: YARN-4866

Workaround: Use Dominant Resource Fairness (DRF) instead of FAIR; or make sure that the cluster has enough vCores in proportion to the memory.

MapReduce job failure and rolling upgrade (CDH 5.6.0)

MapReduce jobs might fail during a rolling upgrade to or from CDH 5.6.0. Cloudera recommends that you avoid doing rolling upgrades to CDH 5.6.0.

Bug: None.

Workaround: Restart failed jobs.

Unsupported Features

The following features are not currently supported:
  • FileSystemRMStateStore: Cloudera recommends you use ZKRMStateStore (ZooKeeper-based implementation) to store the ResourceManager's internal state for recovery on restart or failover. Cloudera does not support the use of FileSystemRMStateStore in production.
  • ApplicationTimelineSever (also known as Application History Server): Cloudera does not support ApplicationTimelineServer v1. ApplicationTimelineServer v2 is under development and Cloudera does not currently support it.
  • Scheduler Reservations: Scheduler reservations are currently at an experimental stage, and Cloudera does not support their use in production.
  • Scheduler node-labels: Node-labels are currently experimental with CapacityScheduler. Cloudera does not support their use in production.