Important Notes About Cloudera Director and Azure
Azure Limits, Quotas, and Constraints
Azure limits the number of CPU cores that can be allocated in each region. For details, see Azure subscription and service limits, quotas, and constraints in the Azure documentation. If you need to increase the limit, contact Microsoft Azure support before deploying the cluster with Cloudera Director.
Not all Azure VM types are available in all Azure regions. See Products available by region on the Microsoft Azure web site to confirm that a VM type is available in a particular region. See Cloudera Reference Architecture for Microsoft Azure Deployments for the latest supported VM types.
Azure Resources Managed by Cloudera Director
- Managed disks and storage accounts:
- For VMs using unmanaged disks in storage accounts: one storage account for each VM.
- For VMs using managed disks: one managed disk for each data disk, and one managed disk for the OS disk.
- A NIC for each VM.
- A public IP address for each VM, if public IP addresses are enabled.
Deploying Production Clusters
Although the Cloudera Director web UI can be used for proof-of-concept deployments on Azure, you must use the published sample configuration files for production deployments (see Useful Links below). You can modify the sample configuration file to fit your specific deployment environment, remove services you do not need, and customize the sample bootstrap script. Configurations related to logging and data storage for individual services must not be changed. Deploying a cluster using the Cloudera Director command-line interface and configuration file based on the examples ensures a repeatable deployment with the proper settings for Azure.
See the Cloudera Reference Architecture for Microsoft Azure Deployments document for more details.
Updating the Azure Plug-in Timeout Value
- Download the latest supported azure-plugin.conf file from the Cloudera Director scripts repository.
- Find the parameter azure-backend-operation-polling-timeout-second in the provider section.
- Change the value to the required duration in seconds.
- On the Cloudera Director server, copy the modified azure-plugin.conf to /var/lib/cloudera-director-plugins/azure-provider-x.x.x/etc/azure-plugin.conf (replacing x.x.x with the latest version), and then restart Cloudera Director with sudo service cloudera-director-server restart
- The storage used for the VM OS disk and cluster data disks.
- The NIC created by the plug-in is attached to the VM. Only one NIC is used per VM. Do not manually attach NICs to the VM created by the plug-in.
- If the VM was set up to have a public IP, Cloudera Director will delete it. If a public IP was attached manually after the VM was created, Cloudera Director will not delete it.
- Cloudera Enterprise Reference Architecture for Azure Deployments.
- Configuration files for running Cloudera Director on Microsoft Azure:
- azure.simple.conf: A simple Cloudera Director configuration that creates a Cloudera Manager node and a four-node cluster (one master and three workers).
- azure.reference.conf: A reference Cloudera Director configuration that creates an eight-node cluster (three masters and five workers) with high availability (HA) enabled.
- azure.kerberos.conf: The same Cloudera Director configuration as azure.reference.conf, but with Kerberos enabled.