Your browser is out of date!

Update your browser to view this website correctly. Update my browser now

×

Easily Manage Hadoop in Production

Cloudera Manager makes it easy to manage Hadoop deployments of any scale in production. Quickly deploy, configure, and monitor your cluster through an intuitive UI - complete with rolling upgrades, backup and disaster recovery, and customizable alerting.

 

Cloudera Manager is available as an integrated and supported part of Cloudera Enterprise.

Cloudera Manager 5.3.1



The recommended tool for installing Cloudera Enterprise

This download installs Cloudera Enterprise or Cloudera Express.

 

Cloudera Enterprise requires a license; however, when installing Cloudera Express you will have the option to unlock Cloudera Enterprise features for a free 60-day trial.

 

Once the trial has concluded, the Cloudera Enterprise features will be disabled until you obtain and upload a license.

Thank you for choosing Cloudera Manager, your download instructions are below:

Instructions for Installing Previous Cloudera Manager Versions

 

To install any previous version of Cloudera Manager 5, you must create a custom repository for the RPM or DEB Files.

See the instructions in Installing Older Versions of Cloudera Manager 5.

The download locations of the 5.0.x RPM or DEB Files are listed in Cloudera Manager Version and Download Information.

Sign in or complete our product interest form to continue.

Please Read and Accept our Terms

Supported Operating Systems

 

Cloudera Manager supports the following operating systems:

  • RHEL-compatible
    • Red Hat Enterprise Linux and CentOS
      • 5.7, 64-bit
      • 6.4, 64-bit
      • 6.4 in SE Linux mode
      • 6.5, 64-bit
    • Oracle Enterprise Linux with default kernel and Unbreakable Enterprise Kernel, 64-bit
      • 5.6 (UEK R2)
      • 6.4 (UEK R2)
      • 6.5 (UEK R2, UEK R3)
  • SLES - SUSE Linux Enterprise Server 11, 64-bit. Service Pack 2 or later is required for CDH 5, and Service Pack 1 or later is required for CDH 4. To use the embedded PostgreSQL database that is installed when you follow Installation Path A - Automated Installation by Cloudera Manager, the Updates repository must be active. The SUSE Linux Enterprise Software Development Kit 11 SP1 is required on hosts running the Cloudera Manager Agents.
  • Debian - Wheezy (7.0 and 7.1), Squeeze (6.0) (deprecated), 64-bit
  • Ubuntu - Trusty (14.04), Precise (12.04), Lucid (10.04) (deprecated), 64-bit

Note:

  • Debian Squeeze and Ubuntu Lucid are supported only for CDH 4.
  • Using the same version of the same operating system on all cluster hosts is strongly recommended.

 

 

 

Selected tab: SupportedOperatingSystems

Supported JDK Versions

 

Cloudera Manager supports Oracle JDK 1.7.0_67 and 1.8.0_11 when it's managing CDH 5.x and Oracle JDK 1.6.0_31 and 1.7.0_67 when it's managing CDH 4.x. Cloudera Manager supports Oracle JDK 1.7.0_67 and 1.8.0_11 when it's managing both CDH 4.x and CDH 5.x clusters. Oracle JDK 1.6.0_31 and 1.7.0_67 can be installed during the installation and upgrade. For further information, see Java Development Kit Installation.

 

Selected tab: SupportedJDKVersions

Supported Browsers

The Cloudera Manager Admin Console, which you use to install, configure, manage, and monitor services, supports the following browsers:

  • Mozilla Firefox 24 and 31
  • Google Chrome
  • Internet Explorer 9 and higher. Internet Explorer 11 Native Mode.
  • Safari 5 and higher

Selected tab: SupportedBrowsers

Supported Databases

 

Cloudera Manager requires several databases. The Cloudera Manager Server stores information about configured services, role assignments, configuration history, commands, users, and running processes in a database of its own. You must also specify a database for the Activity Monitor and Reports Manager management services.

 

Important: When processes restart, the configuration for each of the services is redeployed using information that is saved in the Cloudera Manager database. If this information is not available, your cluster will not start or function correctly. You must therefore schedule and maintain regular backups of the Cloudera Manager database in order to recover the cluster in the event of the loss of this database.

See Backing Up Databases.

 

The database you use must be configured to support UTF8 character set encoding. The embedded PostgreSQL database that is installed when you follow Installation Path A - Automated Installation by Cloudera Manager automatically provides UTF8 encoding. If you install a custom database, you may need to enable UTF8 encoding. The commands for enabling UTF8 encoding are described in each database topic under Cloudera Manager and Managed Service Data Stores.

After installing a database, upgrade to the latest patch version and apply any other appropriate updates. Available updates may be specific to the operating system on which it is installed.

Cloudera Manager and its supporting services can use the following databases:

  • MySQL - 5.5 and 5.6
  • Oracle 11gR2
  • PostgreSQL - 8.4, 9.2, and 9.3

Cloudera supports the shipped version of MySQL and PostgreSQL for each supported Linux distribution. Each database is supported for all components in Cloudera Manager and CDH subject to the notes in CDH 4 Supported Databases and CDH 5 Supported Databases.

 

 

Selected tab: SupportedDatabases

Supported CDH and Managed Service Versions

 

The following versions of CDH and managed services are supported:

Warning: Cloudera Manager 5 does not support CDH 3 and you cannot upgrade Cloudera Manager 4 to Cloudera Manager 5 if you have a cluster running CDH 3.Therefore, to upgrade CDH 3 clusters to CDH 4 using Cloudera Manager you must use Cloudera Manager 4.

  • CDH 4 and CDH 5. The latest released versions of CDH 4 and CDH 5 are strongly recommended. For information on CDH 4 requirements, see CDH 4 Requirements and Supported Versions. For information on CDH 5 requirements, see CDH 5 Requirements and Supported Versions.
  • Cloudera Impala - Cloudera Impala is included with CDH 5. Cloudera Impala 1.2.1 with CDH 4.1.0 or later. For more information on Cloudera Impala requirements with CDH 4, see Cloudera Impala Requirements.
  • Cloudera Search - Cloudera Search is included with CDH 5. Cloudera Search 1.2.0 with CDH 4.6.0. For more information on Cloudera Search requirements with CDH 4, see Cloudera Search Requirements.
  • Apache Spark - 0.90 or later with CDH 4.4.0 or later.
  • Apache Accumulo - 1.4.3 with CDH 4.3.0, 1.4.4 with CDH 4.5.0, and 1.6.0 with CDH 4.6.0.

For more information, see the Product Compatibility Matrix.

 

 

Selected tab: SupportedCDHandManagedServiceVersions

Resource Requirements

Cloudera Manager requires the following resources:

  • Disk Space
    • Cloudera Manager Server
      • 5 GB on the partition hosting /var.
      • 500 MB on the partition hosting /usr.
      • For parcels, the space required depends on the number of parcels you download to the Cloudera Manager Server and distribute to Agent hosts. You can download multiple parcels of the same product, of different versions and builds. If you are managing multiple clusters, only one parcel of a product/version/build/distribution is downloaded on the Cloudera Manager Server—not one per cluster. In the local parcel repository on the Cloudera Manager Server, the approximate sizes of the various parcels are as follows:
        • CDH 4.6 - 700 MB per parcel; CDH 5 (which includes Impala and Search) - 1.5 GB per parcel (packed), 2 GB per parcel (unpacked)
        • Cloudera Impala - 200 MB per parcel
        • Cloudera Search - 400 MB per parcel
    • Cloudera Management Service -The Host Monitor and Service Monitor databases are stored on the partition hosting /var. Ensure that you have at least 20 GB available on this partition.For more information, see Data Storage for Monitoring Data.
    • Agents - On Agent hosts each unpacked parcel requires about three times the space of the downloaded parcel on the Cloudera Manager Server. By default unpacked parcels are located in /opt/cloudera/parcels.
  • RAM - 4 GB is recommended for most cases and is required when using Oracle databases. 2 GB may be sufficient for non-Oracle deployments with fewer than 100 hosts. However, to run the Cloudera Manager Server on a machine with 2 GB of RAM, you must tune down its maximum heap size (by modifying -Xmx in /etc/default/cloudera-scm-server). Otherwise the kernel may kill the Server for consuming too much RAM.
  • Python - Cloudera Manager and CDH 4 require Python 2.4 or later, but Hue in CDH 5 and package installs of CDH 5 require Python 2.6 or 2.7. All supported operating systems include Python version 2.4 or later.
  • Perl - Cloudera Manager requires perl.

Selected tab: ResourceRequirements

Networking and Security Requirements

 

The hosts in a Cloudera Manager deployment must satisfy the following networking and security requirements:

  • Cluster hosts must have a working network name resolution system and correctly formatted /etc/hosts file. All cluster hosts must have properly configured forward and reverse host resolution through DNS. The /etc/hosts files must
    • Contain consistent information about hostnames and IP addresses across all hosts
    • Not contain uppercase hostnames
    • Not contain duplicate IP addresses

    Also, do not use aliases, either in /etc/hosts or in configuring DNS. A properly formatted /etc/hosts file should be similar to the following example:

    127.0.0.1 localhost.localdomain localhost
    192.168.1.1 cluster-01.example.com cluster-01
    192.168.1.2 cluster-02.example.com cluster-02
    192.168.1.3 cluster-03.example.com cluster-03

  • In most cases, the Cloudera Manager Server must have SSH access to the cluster hosts when you run the installation or upgrade wizard. You must log in using a root account or an account that has password-less sudo permission. For authentication during the installation and upgrade procedures, you must either enter the password or upload a public and private key pair for the root or sudo user account. If you want to use a public and private key pair, the public key must be installed on the cluster hosts before you use Cloudera Manager.

    Cloudera Manager uses SSH only during the initial install or upgrade. Once the cluster is set up, you can disable root SSH access or change the root password. Cloudera Manager does not save SSH credentials, and all credential information is discarded when the installation is complete. For more information, see Permission Requirements for Package-based Installations and Upgrades of CDH.

  • If single user mode is not enabled, the Cloudera Manager Agent runs as root so that it can make sure the required directories are created and that processes and files are owned by the appropriate user (for example, the hdfs and mapred users).
  • No blocking is done by Security-Enhanced Linux (SELinux).
  • IPv6 must be disabled.
  • No blocking by iptables or firewalls; port 7180 must be open because it is used to access Cloudera Manager after installation. Cloudera Manager communicates using specific ports, which must be open.
  • For RedHat and CentOS, the /etc/sysconfig/network file on each host must contain the hostname you have just set (or verified) for that host.
  • Cloudera Manager and CDH use several user accounts and groups to complete their tasks. The set of user accounts and groups varies according to the components you choose to install. Do not delete these accounts or groups and do not modify their permissions and rights. Ensure that no existing systems prevent these accounts and groups from functioning. For example, if you have scripts that delete user accounts not in a whitelist, add these accounts to the list of permitted accounts. Cloudera Manager, CDH, and managed services create and use the following accounts and groups:

Table 1. Users and Groups

Component (Version)

Unix User ID Groups Notes
Cloudera Manager (all versions) cloudera-scm cloudera-scm Cloudera Manager processes such as the Cloudera Manager Server and the monitoring roles run as this user.

The Cloudera Manager keytab file must be named cmf.keytab since that name is hard-coded in Cloudera Manager.

Note: Applicable to clusters managed by Cloudera Manager only.

 

Apache Accumulo (Accumulo 1.4.3 and higher) accumulo accumulo Accumulo processes run as this user.
Apache Avro   No special users.
Apache Flume (CDH 4, CDH 5) flume flume The sink that writes to HDFS as this user must have write privileges.
Apache HBase (CDH 4, CDH 5) hbase hbase The Master and the RegionServer processes run as this user.
HDFS (CDH 4, CDH 5) hdfs hdfs, hadoop The NameNode and DataNodes run as this user, and the HDFS root directory as well as the directories used for edit logs should be owned by it.
Apache Hive (CDH 4, CDH 5) hive hive

The HiveServer2 process and the Hive Metastore processes run as this user.

A user must be defined for Hive access to its Metastore DB (e.g. MySQL or Postgres) but it can be any identifier and does not correspond to a Unix uid. This isjavax.jdo.option.ConnectionUserNamein hive-site.xml.

Apache HCatalog (CDH 4.2 and higher, CDH 5) hive hive

The WebHCat service (for REST access to Hive functionality) runs as the hive user.

HttpFS (CDH 4, CDH 5) httpfs httpfs

The HttpFS service runs as this user. SeeHttpFS Security Configuration for instructions on how to generate the mergedhttpfs-http.keytab file.

Hue (CDH 4, CDH 5) hue hue

Hue services run as this user.

Cloudera Impala (CDH 4.1 and higher, CDH 5) impala impala, hadoop, hdfs, hive Impala services run as this user.
Apache Kafka (Cloudera Distribution of Kafka 1.2.0) kafka kafka Kafka services run as this user.
KMS (File) (CDH 5.2.1 and higher) kms kms The KMS (File) service runs as this user.
KMS (Navigator Key Trustee) (CDH 5.3 and higher) kms kms The KMS (Navigator Key Trustee) service runs as this user.
Llama (CDH 5) llama llama Llama runs as this user.
Apache Mahout   No special users.
MapReduce (CDH 4, CDH 5) mapred mapred, hadoop Without Kerberos, the JobTracker and tasks run as this user. The LinuxTaskController binary is owned by this user for Kerberos.
Apache Oozie (CDH 4, CDH 5) oozie oozie The Oozie service runs as this user.
Parquet   No special users.
Apache Pig   No special users.
Cloudera Search (CDH 4.3 and higher, CDH 5) solr solr The Solr processes run as this user.
Apache Spark (CDH 5) spark spark The Spark History Server process runs as this user.
Apache Sentry (incubating) (CDH 5.1 and higher) sentry sentry The Sentry service runs as this user.
Apache Sqoop (CDH 4, CDH 5) sqoop sqoop This user is only for the Sqoop1 Metastore, a configuration option that is not recommended.
Apache Sqoop2 (CDH 4.2 and higher, CDH 5) sqoop2 sqoop, sqoop2 The Sqoop2 service runs as this user.
Apache Whirr   No special users.
YARN (CDH 4, CDH 5) yarn yarn, hadoop Without Kerberos, all YARN services and applications run as this user. The LinuxContainerExecutor binary is owned by this user for Kerberos.
Apache ZooKeeper (CDH 4, CDH 5) zookeeper zookeeper The ZooKeeper processes run as this user. It is not configurable.

 

Selected tab: NetworkingandSecurityRequirements
Selected tab: SystemRequirements

Issues Fixed in Cloudera Manager 5.3.1

Deploy client configuration no longer fails after 60 seconds

When configuring a gateway role on a host that already contains a role of the same type, for example, an HDFS gateway on a DataNode, the deploy client configuration command no longer fails after 60 seconds.

 

service cloudera-scm-server force_start now works

 

After deleting services, the Cloudera Manager Server log no longer contains foreign key constraint failure exceptions

 

When using Isilon, Cloudera Manager now sets mapred_submit_replication correctly

When EMC Isilon storage is used there is no DataNode so it's not possible to set mapred_submit_replication to a number smaller or equal to the number of DataNodes in the network. Cloudera Manager now does the following when setting mapred_submit_replication:

  • If using HDFS, set to a minimum of 1 and issue a warning when > # of DataNodes
  • If using Isilon, set to 1 and do not check against # of DataNodes

 

The Cloudera Manager Agent now sets the file descriptor ulimit correctly on Ubuntu

 

During upgrade, bootstrapping the standby NameNode step no longer fails with standby NameNode getting connection refused when connecting to active NameNode

 

Deploy krb5.conf now also deploys it on hosts with Cloudera Management Service roles

 

Cloudera Manager allows upgrades to unknown CDH maintenance releases

Cloudera Manager 5.3.0 should support any CDH release less that or equal to 5.3, even if such a release didn't exist at the time that Cloudera Manager 5.3.0 was released. For packages, there's currently no way to use the upgrade wizard to perform an upgrade to such a release. This release adds a custom CDH field for the package case, where you can type in a version that didn't exist at the time of the Cloudera Manager release.

 

Impalad memory limit units error in EnableLlamaRMCommand

The EnableLlamaRMCommand sets the value of the Impalad memory limit to equal the NM container memory value. But the latter is in MB, while the former is in bytes. Previously, the command wasn't doing the conversion, which has now been fixed.

 

Running MapReduce v2 jobs are now visible using the Application Master view

In the Application view, selecting Application Master for a MRv2 job previously resulted in no action.

 

Deleting services no longer results in foreign key constraint exceptions

The Cloudera Manager Server log previously showed several foreign key constraint exceptions that were associated with deleted services. This has been fixed.

 

HiveServer2 keystore and LDAP group mapping passwords are no longer exposed in client configuration files

The HiveServer2 keystore password and LDAP group mapping passwords were being emitted into the client configuration files. This exposed the passwords in plain text in a world-readable file. This has now been fixed.

 

Fixed a cross-site scripting vulnerability in Cloudera Management Service web UIs

 

The high availability wizard now sets the HDFS dependency on ZooKeeper

Workaround: Before enabling high availability, do the following:

  1. Create and start a ZooKeeper service if one doesn't exist.
  2. Go to the HDFS service.
  3. Click the Configuration tab.
  4. In the Service-Wide category, set the ZooKeeper Service property to the ZooKeeper service.
  5. Click Save Changes to commit the changes.

 

BDR no longer assumes superuser is common if clusters have the same realm

If source and destination clusters are in same Kerberos realm, Cloudera Manager assumed that superuser of destination is also the superuser on source cluster. But that's not necessarily the case because HDFS can be configured so that that's not the case.

 

 

Selected tab: WhatsNew

Cloudera Enterprise Extensions

ODBC and JDBC Drivers

The Cloudera ODBC and JDBC Drivers for Hive and Impala enable your enterprise users to access Hadoop data through Business Intelligence (BI) applications with ODBC/JDBC support.

Data Transfer Connectors

Sqoop Connectors are used to transfer data between Apache Hadoop systems and external databases or Enterprise Data Warehouses. These connectors allow Hadoop and platforms like CDH to complement existing architecture with seamless data transfer.

Want to Get Involved or Learn More?

Check out our other resources

Cloudera Community

Collaborate with your peers, industry experts, and Clouderans to make the most of your investment in Hadoop.

Cloudera University

Receive expert Hadoop training through Cloudera University, the industry's only truly dynamic Hadoop training curriculum that’s updated regularly to reflect the state of the art in big data.