This is the documentation for Cloudera Manager 4.8.5.
Documentation for other versions is available at Cloudera Documentation.

Resource Management

The 4.5 release of Cloudera Manager reinforces existing resource management techniques and introduces several new ones. These are primarily intended to isolate compute frameworks from one another. For example, MapReduce and Impala often work with the same data set and run side-by-side on the same physical hardware. Without explicitly managing the cluster's resources, Impala queries may affect MapReduce job SLAs, and vice versa.

Resource Management via Control Groups (Cgroups)

Cloudera Manager 4.5 introduces support for the Control Groups (cgroups) Linux kernel feature. With cgroups, administrators can impose per-resource restrictions and limits on CDH processes.

Enabling Resource Management

  Important:

If you've upgraded from an older version of Cloudera Manager to Cloudera Manager 4.5, you must restart every Cloudera Manager supervisor process before using cgroups-based resource management. The easiest and safest way to do this is:

  1. Stop all services, including the Cloudera Management Services.
  2. On each cluster node, run as root:
    $ service cloudera-scm-agent hard_restart
  3. Start all services.

Cgroups-based resource management can be enabled for all hosts, or on a per-host basis.

To enable cgroups for all hosts:

  1. Click the Hosts tab.
  2. Click on the Configuration tab, then select View and Edit.
  3. Select the Resource Management category.
  4. Check the box next to Enable Cgroup-based Resource Management.

To enable cgroups for individual hosts:

  1. Click the Hosts tab.
  2. Click the link for the host where you want to enable cgroups.
  3. Click on the Configuration tab, then select View and Edit.
  4. Select the Resource Management category.
  5. Check the box next to Enable Cgroup-based Resource Management.

When cgroups-based resource management is enabled for a particular host, all roles on that host must be restarted for the changes to take effect.

Using Resource management

After enabling cgroups, one can restrict and limit the resource consumption of roles (or role config groups) on a per-resource basis. All of these parameters can be found in the Cloudera Manager configuration UI, under the Resource Management category.

CPU Shares

The more CPU shares given to a role, the larger its share of the CPU when under contention. Until processes on the host (including both roles managed by Cloudera Manager and other system processes) are contending for all of the CPUs, this will have no effect. When there is contention, those processes with higher CPU shares will be given more CPU time. The effect is linear: a process with 4 CPU shares will be given roughly twice as much CPU time as a process with 2 CPU shares.

Updates to this parameter will be dynamically reflected in the running role.

I/O Weight

Much like CPU shares, the more the I/O weight, the higher priority will be given to I/O requests made by the role when I/O is under contention (either by roles managed by Cloudera Manager or by other system processes). Note that this only affects read requests; write requests remain unprioritized.

Updates to this parameter will be dynamically reflected in the running role.

Memory Hard Limit

When a role's resident set size (RSS) exceeds the value of this parameter, the kernel will swap out some of the role's memory. If it's unable to do so, it will kill the process. Note that the kernel measures memory consumption in a manner that doesn't necessarily match what the top or ps report for RSS, so expect that this limit is a rough approximation.

After updating this parameter, the role must be restarted before changes take effect.

Known issues

  1. The role config group and role override cgroup-based resource management parameters must be saved one at a time. Otherwise some of the changes that should be reflected dynamically will be ignored.
  2. The role config group abstraction is an imperfect fit for resource management parameters, where the goal is often to take a numeric value for a host resource and distribute it amongst running roles. The role config group represents a "horizontal" slice: the same role across a set of hosts. However, the cluster is often viewed in terms of "vertical" slices, each being a combination of slave roles (such as TaskTracker, DataNode, Region Server, Impala daemon, etc.). Nothing in Cloudera Manager guarantees that these disparate horizontal slices are "aligned" (meaning, that the role assignment is identical across hosts). If they unaligned, some of the role config group values will be incorrect on unaligned hosts. For example, a host whose role config groups have been configured with memory limits but that's missing a role will probably have unassigned memory.

Linux distribution support

Cgroups are a feature of the Linux kernel, and as such, support depends on the underlying host's Linux distribution and version.

Distribution

CPU Shares

I/O Weight

Memory Hard Limit

Red Hat Enterprise Linux (or CentOS) 5

 

 

 

Red Hat Enterprise Linux (or CentOS) 6

images/image43.png

images/image43.png

images/image43.png

SUSE Linux Enterprise Server 11

images/image43.png

images/image43.png

images/image43.png

Ubuntu 10.04 LTS

images/image43.png

 

images/image43.png

Ubuntu 12.04 LTS

images/image43.png

images/image43.png

images/image43.png

Debian 6.0

images/image43.png

 

 

If the distribution lacks support for a given parameter, changes to it will have no effect.

The exact level of support can be found in the Cloudera Manager Agent's log file, shortly after the agent has started. See Viewing the Cloudera Manager Server and Agent Logs to find the agent log. In the log file, look for an entry like this:

Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_blkio': True}

The 'has_memory' and similar entries correspond directly to support for the CPU, I/O, and Memory parameters.

Existing resource management controls

Cloudera Manager 4.5 reorganizes existing resource management parameters under the Resource Management configuration UI category. Affected parameters include:

  • Java maximum heap sizes for all Java-based roles.
  • Impala Daemon Memory Limit.
  • MapReduce Child Java Maximum Heap Size (Gateway and Client Override).
  • MapReduce Map Task Maximum Heap Size (Gateway and Client Override).
  • MapReduce Reduce Task Maximum Heap Size (Gateway and Client Override).
  • MapReduce Maximum Virtual Memory (Gateway and Client Override).
  • MapReduce Map Task Maximum Virtual Memory (Gateway and Client Override).
  • MapReduce Reduce Task Maximum Virtual Memory (Gateway and Client Override).
  • YARN Container Memory.
  • YARN Virtual to Physical Memory Ratio.
  • YARN Map Task Maximum Heap Size.
  • YARN Reducer Task Maximum Heap Size.
  • YARN Map Task Memory.
  • YARN Reduce Task Memory.
  • YARN Application Master Java Maximum Heap Size.
  • YARN Application Master Memory.

Examples

Protecting production MapReduce jobs from Impala queries

Suppose you have MapReduce deployed in production and want to roll out Impala without clobbering your production MapReduce jobs.

For simplicity, we will make the following assumptions:

  1. The cluster is using homogenous hardware.
  2. Each slave node has two cores.
  3. Each slave node has 8 GB of RAM.
  4. Each slave node is running a Datanode, Tasktracker, and an Impala daemon.
  5. Each role type is in a single role config group.
  6. Cgroups-based resource management has been enabled on all hosts.

CPU:

  1. Leave Datanode and Tasktracker role config group CPU shares at 1024.
  2. Set Impala daemon role config group's CPU shares to 256.
  3. The Tasktracker role config group should be configured with a Maximum Number of Simultaneous Map Tasks of 2 and a Maximum Number of Simultaneous Reduce Tasks of 1. This yields an upper bound of three MapReduce tasks at any given time; this is an important detail for memory sizing.

Memory:

  1. Set Impala daemon role config group memory limit to 1024 MB.
  2. Leave Datanode maximum Java heap size at 1 GB.
  3. Leave Tasktracker maximum Java heap size at 1 GB.
  4. Leave MapReduce Child Java Maximum Heap Size for Gateway at 1 GB.
  5. Leave cgroups hard memory limits alone. We'll rely on "cooperative" memory limits exclusively, as they yield a nicer user experience than the cgroups-based hard memory limits.

I/O:

  1. Leave Datanode and Tasktracker role config group I/O weight at 500.
  2. Impala daemon role config group I/O weight is set to 125.

When you're done with configuration, restart all services for these changes to take effect.

The results are:

  1. When MapReduce jobs are running, all Impala queries together will consume up to a fifth of the cluster's CPU resources.
  2. Individual Impala daemons won't consume more than 1 GB of RAM. If this figure is exceeded, new queries will be cancelled.
  3. Datanodes and TaskTrackers can consume up to 1 GB of RAM each.
  4. We expect up to 3 MapReduce tasks at a given time, each with a maximum heap size of 1 GB of RAM. That's up to 3 GB for MR tasks.
  5. The remainder of each host's available RAM (6 GB) is reserved for other host processes.
  6. When MapReduce jobs are running, read requests issued by Impala queries will receive a fifth of the priority of either HDFS read requests or MapReduce read requests.