Cloudera Manager now supports parcels as an alternate form of distribution for CDH and other system packages. Among other benefits, parcels provide a mechanism for performing upgrades to the packages installed on a cluster with minimal disruption.
To enable upgrades with minimal disruption, packages are handles in three steps: Download, Distribution, and Activation.
Downloading a parcel downloads the appropriate software to a local parcel repository on the Cloudera Manager server, where it is available for distribution to the nodes in the cluster. You can have multiple parcels downloaded to your cluster.
Distributing a parcel copies the parcel to the member hosts of the cluster. Note that distributing a parcel does not actually upgrade the components; the current services continue to run unchanged. You can have multiple parcels distributed on your cluster.
The Distribute process does not require Internet access; rather the parcels are downloaded by the Cloudera Manager agents on each cluster member from the local parcel repository on the Cloudera Manager server.
Activating a parcel causes the Cloudera Manager to link to the new components, ready to run the new version upon the next restart. Activating does not stop the current services or perform a restart — instead the system administrator can determine the appropriate time to perform those operations.
Cloudera Manager detects when new parcels are available. The parcel indicator in the Admin console navigation bar ( ) indicates when parcels are available for downloading or distribution. If no parcels are available, or if all parcels have been activated, then this indicator will be zero. Note that you can configure Cloudera Manager to download and distribute parcels automatically, if desired.
Downloading a parcel
- Click the parcel indicator in the top navigation bar. This takes you to the Hosts page, Parcels tab. By default, any parcels available for download or distribution are shown, as well as any activated parcels. Parcels available for download will display a Download button.
- Click Download to initiate the download of the parcel to the local repository. You can configure the location of the remote repository, the location of the local parcel repository, and other settings on the Administration page, Properties tab under the Parcels section. You can also access these properties by clicking the link "Click here to configure these settings" near the top of the Parcels page. When the parcel has been downloaded, the button changes to say Distribute.
The parcel download is done at the Cloudera Manager server, so with multiple clusters, the downloaded parcels are shown as available to all clusters managed by the Cloudera Manager server. However, distribution (to a specific cluster's member nodes) must be selected on a cluster-by-cluster basis.
Distributing a Parcel
Parcels that have been downloaded can be distributed to the nodes in your cluster, available for activation.
- From the Parcels tab, click the Distribute button for the parcel you want to distribute. This will begin the process of distributing the parcel to the nodes in the cluster.
If you have a large number of nodes to which the parcels should be distributed, you can control how many concurrent uploads Cloudera Manager will perform. You can configure this setting on the Adminstration page, Properties tab under the Parcels section.
You can delete a parcel that is ready to be distributed; click the triangle at the right end of the Distribute button to access the Delete command. This will delete the downloaded parcel from the local parcel repository.
Note that distributing parcels to the nodes in the cluster does not affect the current running services.
Activating a parcel
Parcels that have been distributed to the nodes in a cluster are ready to be activated.
- From the Parcels tab, click the Activate button for the parcel you want to activate. This will update Cloudera Manager to point to the new software, ready to be run the next time a service is restarted.
- To start using a new version of a component, go to the Services tab and restart your services. Until you restart services, the current software will continue to run. This allows you to restart your services at a time that is convenient based on your maintenance schedules or other considerations.
Deactivating a parcel
You can deactivate an active parcel; this will update Cloudera Manager to point to the previous software version, ready to be run the next time a service is restarted.
To use the previous version of the software, go to the Services tab and restart your services.
If you did your original installation from parcels, and there is only one version of your software installed (i.e. no packages, and no additional parcels have been activated and started) then when you attempt to restart after deactivating the current version, your roles will be stopped but will not be able to restart.
Parcel Configuration Settings
You can configure settings for parcels, on the Administration page, Properties tab under the Parcels section. You can also access these properties by clicking the link Click here to configure these settings near the top of the Parcels page.
The Local Parcel Repository Path defines the path on the Cloudera Manager server host where downloaded parcels are stored.
The Remote Parcel Repository URLs is a list of remote repositories the Cloudera Manager should check for parcels. Initially this points to the default repository at archive.cloudera.com (http://archive.cloudera.com/cdh4/parcels/latest/) but you can added your own repository locations to the list.
You can also:
- Set the frequency with which Cloudera Manager will check for new parcels
- Configure a proxy to access to the remote repositories
- Configure whether downloads and distribution of parcels should occur automatically whenever new ones are detected.
- Control which products can be downloaded if automatic downloading is enabled.
If automatic downloading/distribution are not enabled (the default), you must go to the Parcels page to initiate these actions.
You can configure the bandwidth limits and the number of concurrent uploads, to tune the load that parcel distribution puts on your network. The defaults are up to 50 concurrent parcel uploads and 50 MiB/s aggregate bandwidth.
- The concurrent upload count (Maximum Parcel Uploads) doesn't matter, theoretically, if all nodes have the same speed ethernet. In general, 50 concurrent uploads is an acceptable setting in most cases. However, in a scenario where the server has more bandwidth (say 10Gbe while the normal nodes are using 1Gbe), then the count is important to maximize bandwidth, and would need to be at least the difference in speeds (10x in this case).
- The bandwidth limit (Parcel Distribution Rate Limit) should be your ethernet speed (in MiB/seconds) divided by approximately 16. You can use a higher limit if you have QoS set up to prevent starving other services, or if you are willing accept a higher risk of higher bandwidth load.
For individual hosts (or all hosts) you can "blacklist" selected parcels; this will prevent those parcels from being distributed to or activated upon those hosts.
To blacklist a parcel:
- Go to the Configuration tab for a host (or for All Hosts) and click Edit.
- Under the parcels category, enter the parcel(s) you want to blacklist. Enter the name as it appears on the Parcels page — for example, 4.1.2-1.cdh4.1.2.p0.30 — and click Save.
If a parcel you blacklist has already been distributed to the host, it will be removed from that host. If it is already running on the host, it will continue to run until the next restart, when it will not be restarted.
Parcel File Locations
The default location for the local parcel repository is /opt/cloudera/parcel-repo on the Cloudera Manager server host. This is configurable as described above.
The default path for the distributed parcels on the member nodes is /opt/cloudera/parcels. This is configurable in the config.ini of the Cloudera Manager agent on each node.