CDP Private Cloud Base is an on-premises version of Cloudera Data Platform (CDP). This new product combines the best of Cloudera Enterprise Data Hub and Hortonworks Data Platform Enterprise along with new features and enhancements across the stack. This unified distribution is a scalable and customizable platform where you can securely run many types of workloads. CDP is an easy, fast, and secure enterprise analytics and management platform with the following capabilities:
One of our previous blogs discussed the four paths to get from legacy platforms to CDP Private Cloud Base. In this blog and accompanying video, we deep dive into the mechanics of running an in-place upgrade from HDP3 to CDP Private Cloud Base. The overall upgrade follows a three staged process illustrated below.
In the video below, we walk through a complete end-to-end upgrade of HDP3 to CDP Private Cloud Base.
How to upgrade from HDP to CDP
HDP3 to CDP Private Cloud Base transition essentially involves two high-level processes after preparing the cluster for upgrade (See Pre-Upgrade Stage) and is represented via the architectural diagram below.
Before proceeding with the upgrade, review the CDP Private Cloud Base prerequisites as specified in the documentation. As a starting point to the upgrade, we’d recommend performing a full cluster health check (which our Professional Services team can also help with). Having a good understanding of the current status and health of the cluster will be critical to a successful upgrade. It would also be worth assessing the cluster readiness for the upgrade. Your Cloudera Account team can help you with this assessment.
The goal of the pre-upgrade steps is to prepare the HDP cluster for upgrade and ensure that the cluster meets minimum version requirements to facilitate the work. This would also be a good place to review the version compatibility for other components like OS, JDK, and backend databases. Please note that you should plan for the downtime required for an in-place upgrade.
It is also worth checking any behavioral changes of the HDP components and application compatibility against the new versions of components in CDP Private Cloud Base. At the very least one should expect to review any API changes and recompile any applications. In some cases, applications may require changes if they depend on components that are removed and unsupported.
Finally we also recommend that you take a full backup of your cluster configurations, metadata, other supporting details, and backend databases. Full details are available for HDP2 and HDP3.
The upgrade activity can be broken down into four tasks:
A- Review and Perform Upgrade Checklist Steps
B- Upgrade Ambari
Upgrading Ambari is independent of upgrading the HDP cluster. The high level process of upgrading Ambari is shown below.
After Ambari has been upgraded, download the cluster blueprints with hosts. Since Ambari has been upgraded to Ambari7, one must follow steps to upgrade Ambari Infra, Ambari Logsearch and Ambari Metrics.
After upgrading Ambari, ensure that the cluster is operating normally and service checks are passed prior to attempting an HDP upgrade. If you upgrade an unhealthy cluster, you may experience failures during the process that require rolling back the cluster.
The high-level process for performing an HDP intermediate bits upgrade is as follows:
Essentially the steps include:
Once the upgrade to HDP7 is complete, proceed to transition the Ambari managed cluster to Cloudera Manager (CM). This is achieved using the AM2CM tool. Before using the tool, you must follow these preparatory steps.
Once the pre-transition steps complete and CM is installed and running, the next step is to transition the Ambari managed cluster to CM via AM2CM. The purpose of this tool is to convert the Ambari blueprint to Cloudera Manager Deployment template. The figure below depicts the use of the AM2CM tool.
As shown in the diagram, the following high level steps occur with AM2CM
The AM2CM tool transitions the service configurations. However, you must configure and perform additional steps to start the services in CDP Private Cloud Base. Post-transition to CM, perform the following steps to ensure correctness of deployment:
Once all the post-transition steps have been completed, review all the warnings and configurations, and start the services in the cluster.
Post-upgrade steps include application upgrade testing, validations, configuration and tuning. These are the tasks that you should have identified and run before the upgrade allowing you to compare pre-upgrade versus post-upgrade test results. These tests should also include any parts of the application that required code changes due to the changes in the platform. You must verify the functionality and performance of various applications and services, and adjust tuning parameters of services accordingly. New features and product behaviors may change the performance characteristics of your workloads and require further adjustments. This would also be an appropriate time to add any newer services, like Hue, to the cluster.
As a part of the post-upgrade step, if you configured LDAP on your cluster, you would want to set up the external authentication and authorization in CM.
Once the upgrade is complete all services should be up and running. At this point you should perform another health check and ensure that all services are working correctly with Cloudera Manager. Additionally ensure to stop and uninstall Ambari & HDP packages.
The end-to-end process is relatively straightforward and well documented. Care should be taken to ensure that applications and workloads are tested in Development and QA environments and that any incompatibilities are ironed out before upgrading production.
Review the video above of an actual cluster upgrade and contact your account team or Cloudera support if you would like to discuss the next steps in your CDP journey.
For additional information on the upgrade process, please see
This may have been caused by one of the following: