Upgrade Hortonworks Data Platform (HDP) to Cloudera Data Platform (CDP) Private Cloud Base
A step by Step Guide
CDP Private Cloud Base is an on-premises version of Cloudera Data Platform (CDP). This new product combines the best of Cloudera Enterprise Data Hub and Hortonworks Data Platform Enterprise along with new features and enhancements across the stack. This unified distribution is a scalable and customizable platform where you can securely run many types of workloads. CDP is an easy, fast, and secure enterprise analytics and management platform with the following capabilities:
- Enables ingesting, managing, and delivering of any analytics workload from Edge to AI
- Provides enterprise grade security and governance
- Provides self-service access to integrated, multi-function analytics on centrally managed and secured business data
- Provides a consistent experience on Public Cloud, Multi-Cloud, and Private Cloud deployments
One of our previous blogs discussed the four paths to get from legacy platforms to CDP Private Cloud Base. In this blog and accompanying video, we deep dive into the mechanics of running an in-place upgrade from HDP3 to CDP Private Cloud Base. The overall upgrade follows a three staged process illustrated below.
In the video below, we walk through a complete end-to-end upgrade of HDP3 to CDP Private Cloud Base.
How to upgrade from HDP to CDP
In-Place Upgrade Overview
HDP3 to CDP Private Cloud Base transition essentially involves two high-level processes after preparing the cluster for upgrade (See Pre-Upgrade Stage) and is represented via the architectural diagram below.
- Upgrade HDP 3.1.5 to Cloudera Runtime 7.1.x using Ambari.
- Transition the management platform from Ambari to Cloudera Manager.
Stage 1: Pre-Upgrade Steps
Before proceeding with the upgrade, review the CDP Private Cloud Base prerequisites as specified in the documentation. As a starting point to the upgrade, we’d recommend performing a full cluster health check (which our Professional Services team can also help with). Having a good understanding of the current status and health of the cluster will be critical to a successful upgrade. It would also be worth assessing the cluster readiness for the upgrade. Your Cloudera Account team can help you with this assessment.
The goal of the pre-upgrade steps is to prepare the HDP cluster for upgrade and ensure that the cluster meets minimum version requirements to facilitate the work. This would also be a good place to review the version compatibility for other components like OS, JDK, and backend databases. Please note that you should plan for the downtime required for an in-place upgrade.
It is also worth checking any behavioral changes of the HDP components and application compatibility against the new versions of components in CDP Private Cloud Base. At the very least one should expect to review any API changes and recompile any applications. In some cases, applications may require changes if they depend on components that are removed and unsupported.
Finally we also recommend that you take a full backup of your cluster configurations, metadata, other supporting details, and backend databases. Full details are available for HDP2 and HDP3.
Stage 2: Upgrade Steps
The upgrade activity can be broken down into four tasks:
A- Review and Perform Upgrade Checklist Steps
- Before upgrading, it is recommended that you review the upgrade checklist to confirm that cluster operation is healthy along with any prerequisites for large clusters
- Download the cluster blueprints from Ambari
- Review compatibility for Management packs (MPacks)
- It is also recommend that you take a full backup of your cluster, including:
- RDBMS
- Zookeeper data
- HDFS Master Node data directories
- Ambari Config directory data
B- Upgrade Ambari
Upgrading Ambari is independent of upgrading the HDP cluster. The high level process of upgrading Ambari is shown below.
After Ambari has been upgraded, download the cluster blueprints with hosts. Since Ambari has been upgraded to Ambari7, one must follow steps to upgrade Ambari Infra, Ambari Logsearch and Ambari Metrics.
After upgrading Ambari, ensure that the cluster is operating normally and service checks are passed prior to attempting an HDP upgrade. If you upgrade an unhealthy cluster, you may experience failures during the process that require rolling back the cluster.
C- Upgrade HDP3 to HDP 7 intermediary bits.
The high-level process for performing an HDP intermediate bits upgrade is as follows:
Essentially the steps include:
- Reviewing the prerequisites
- Taking backups
- Performing pre-upgrade steps
- Registering the repositories
- Installing the packages
- Performing the upgrade
- Executing post HDP-upgrade tasks
D- Transition to Cloudera Manager
Once the upgrade to HDP7 is complete, proceed to transition the Ambari managed cluster to Cloudera Manager (CM). This is achieved using the AM2CM tool. Before using the tool, you must follow these preparatory steps.
- Perform pre-transition steps for services and components deployed in the cluster.
- Cloudera Manager Installation and Setup – This includes installing Cloudera Manager Agents on all hosts and installing the Cloudera Manager Server & Database.
Once the pre-transition steps complete and CM is installed and running, the next step is to transition the Ambari managed cluster to CM via AM2CM. The purpose of this tool is to convert the Ambari blueprint to Cloudera Manager Deployment template. The figure below depicts the use of the AM2CM tool.
As shown in the diagram, the following high level steps occur with AM2CM
- Supply the tool with already downloaded Ambari blueprints
- AM2CM converts the blueprint to a CM deployment template
- Import the converted template to Cloudera Manager
- Start the services through the Cloudera Manager UI, and validate the cluster
The AM2CM tool transitions the service configurations. However, you must configure and perform additional steps to start the services in CDP Private Cloud Base. Post-transition to CM, perform the following steps to ensure correctness of deployment:
- Review configuration warning for all the services
- Review JVM parameters, log4j, and other configurations for all services as some of the JVM parameters and configurations are not transitioned
- Generate Kerberos credentials for services if required
- For each services complete the post-transition steps before starting the cluster
Once all the post-transition steps have been completed, review all the warnings and configurations, and start the services in the cluster.
Stage 3: Post-Upgrade Steps
Post-upgrade steps include application upgrade testing, validations, configuration and tuning. These are the tasks that you should have identified and run before the upgrade allowing you to compare pre-upgrade versus post-upgrade test results. These tests should also include any parts of the application that required code changes due to the changes in the platform. You must verify the functionality and performance of various applications and services, and adjust tuning parameters of services accordingly. New features and product behaviors may change the performance characteristics of your workloads and require further adjustments. This would also be an appropriate time to add any newer services, like Hue, to the cluster.
As a part of the post-upgrade step, if you configured LDAP on your cluster, you would want to set up the external authentication and authorization in CM.
Completion and Finalization
Once the upgrade is complete all services should be up and running. At this point you should perform another health check and ensure that all services are working correctly with Cloudera Manager. Additionally ensure to stop and uninstall Ambari & HDP packages.
Summary
The end-to-end process is relatively straightforward and well documented. Care should be taken to ensure that applications and workloads are tested in Development and QA environments and that any incompatibilities are ironed out before upgrading production.
Review the video above of an actual cluster upgrade and contact your account team or Cloudera support if you would like to discuss the next steps in your CDP journey.
For additional information on the upgrade process, please see