Environment Setup for Azure

Altus requires access to your Azure subscription to perform tasks on your behalf. To allow Altus to create clusters and run jobs in your Azure subscription, you must give consent to Altus to access the resources in your subscription

Setting up the Altus Environment for Azure

When you create an Altus environment, you specify the Azure subscription and the resources in the subscription that you want to use for Altus clusters and jobs. You must create the Azure resources for the Altus clusters and jobs before you create the Altus environment. The first time that you create an Altus environment, you must complete the access delegation process before you can create the environment.

Altus provides an environment wizard to assist you in creating an Altus environment. When you create the first Altus environment for an Azure subscription, the wizard initiates the access delegation process. After you complete the access delegation process, you can proceed with creating the Altus environment. The access delegation process is only required the first time you create an environment in an Azure subscription.

To create an Altus environment, complete the following steps:
1. Create resources in your Azure subscription for Altus clusters and jobs.
Log in to the Azure portal with your Azure credentials.

Altus provides an ARM template to assist you in creating the resources required by Altus. You can use the template to create the resources quickly or you can create the resources manually.

2. Start the Environment Wizard.
Run the environment wizard on the Altus console.
3. Complete the access delegation process.
You perform the access delegation process on the Azure portal.
4. Create the Altus environment with the Azure resources for the Altus clusters and jobs.
Complete the creation of the Altus environment on the Altus console.

If users must access different resources in the same subscription, you can clone an existing environment and point to different resources in the same subscription. When you clone an Altus environment, you do not need to go through the access delegation process.

If you create an Altus environment for resources in a different subscription, you need to complete the access delegation process for the first environment.

Step 1. Create the Azure Resources with the Altus ARM Template

Altus provides an Azure Resource Manager (ARM) template for creating the resources in your Azure subscription required for Altus clusters and jobs. Before you create the environment in Azure, create the resources that you plan to use for Altus clusters. Then specify the resource names and resource groups in the Altus environment wizard.

The ARM template provided by Altus creates the following resources:

Using the ARM template provided by Altus to create resources for Altus is optional. You can manually create the resources in your subscription. For more information about the resources that you need to create in your Azure subscription, see Azure Resources and Services.

To create the Azure resources using the Altus ARM template:
  1. On the Azure portal, go to the Cloudera Altus Custom Deployment template.

    You must be logged in as a subscription administrator to perform this task.

  2. In the BASICS section, specify the following:
    Property Description
    Subscription ID of the subscription where the resources will be created.

    You must be logged in as an administrator of this subscription.

    Resource Group Name of the resource group that includes the virtual network where the Cloudera Altus VM image will be deployed.

    You can use an existing resource group or create a resource group specifically for resources used by Altus clusters. To create a resource group, select Create new and specify the resource group name. Altus creates a resource group for Altus resources with the name that you provide.

    Location Region where Altus creates the Altus clusters.
    Cloudera Altus supports the following Azure regions:
    • Australia Southeast (australiasoutheast)
    • Central US (centralus)
    • East US 2 (eastus2)
    • North Europe (northeurope)
    • Southeast Asia (southeastasia)
    • West Europe (westeurope)
  3. In the SETTINGS section, specify the following:
    Property Description
    Admin Username

    User name for the administrator for the DNS server to resolve domain names for the Altus VMs. You can use the default administrator name provided by Altus: azureuser

    Admin Password Password for the DNS server administrator.
    Instance Type Instance type of the DNS server for the VM.

    Altus uses the Standard_A2_v2 instance type as the default instance type for the DNS server.

    For more information about VM instance types, see Sizes for Cloud Services

    Resource Name Prefix Name prefix for the virtual network and network security group.

    Default prefix: altus-quickstart

    Based on the prefix, Altus uses the following default names for the resources:

    Virtual network: altus-quickstart-vnet

    Network security group: altus-quickstart-nsg

    DNS Host Name Host name for the DNS server.

    Default: altus-quickstart-dns-server

    DNS Private Zone Name Name for the private DNS zone.

    Default: altus.quickstart

    DNS VM IP Address IP address for the DNS server VM.

    Default: 10.3.0.4

    Network Address Space Address space for the virtual network.

    Default: 10.3.0.0/24

    To complete the template easily, use the default values that Altus provides.

  4. Verify that your entries are correct and agree with the terms and conditions.
  5. Click Purchase.

Altus creates the resources in your Azure subscription. It can take some time before all resources are created in your subscription. Verify that the resources are available in your Azure subscription before you create the Altus environment.

To verify the resources created by the Altus custom deployment template, go to the list of services and click Virtual Network. On the Virtual networks page, click the name of the resource group for the Altus deployment. The Overview page displays the list of resources included in the resource group.

Step 2. Start the Environment Wizard

Use the Altus Environment wizard to create an Altus environment and configure the environment properties with the resources in your Azure subscription that you want to use for Altus clusters and jobs. When you create an environment in the subscription for the first time, the environment wizard initiates the Azure access delegation process before you can create an environment.

To start the Environment Wizard:
  1. Sign in to the Cloudera Altus console:

    https://console.altus.cloudera.com/

  2. On the side navigation panel, click Environments.

    The Environments page displays the list of all the Altus environments that are available for your Azure subscription. It also shows the Azure region where the resources are located.

  3. Click Wizard.

    Altus displays the cloud service providers that are supported for Altus clusters and jobs, including Microsoft Azure.

  4. Enter the subscription ID for the Azure subscription where you want to create Altus clusters and click Create.

The Environment Wizard takes you to the Microsoft Azure portal where you can perform the access delegation steps.

Step 3. Complete the Access Delegation Process

The access delegation process includes the following tasks:
  1. Consenting to Cloudera Altus accessing resources in the Azure subscription
  2. Setting up role-based access for Cloudera Altus
  3. Enabling programmatic deployment of the Cloudera Altus Service VM image

Setting up Role-Based Access for Cloudera Altus

As part of the access delegation process, assign a role to the Cloudera Altus application to enable Altus to create and manage clusters in your subscription. With role-based access control (RBAC), you determine how much access you want to give Altus in your Azure subscription:
  • To provide Altus full access to your subscription, assign the Contributor role to the Cloudera Altus application at the subscription level.

    If you assign the Contributor role to Cloudera Altus for the subscription, Altus can access all resources in your subscription. You can control access by creating an Azure subscription specifically for use by Altus clusters and jobs.

  • To limit Altus access within your subscription, assign the Contributor role to the Cloudera Altus application at the resource level.

    If you do not want to provide Altus with full access to you subscription, you can limit Altus access by assigning the Contributor role to Cloudera Altus for specific resources within the subscription.

    Assign the Contributor role to the following resources:

    • Resource group. The resource group for Altus clusters. This resource group includes the virtual network where the Cloudera Altus VM image will be deployed.
    • Virtual network (VNet). The virtual network in the resource group for Altus clusters.
    • Network security group (NSG). The network security group for Altus clusters.
  • To limit the permissions that you grant to the Cloudera Altus application, you can create a custom role with only the specific permissions required by the Altus application and assign the role to the Altus application at the subscription level.

For more information, see Use Role-Based Access Control to manage access to your Azure subscription resources

Assigning the Contributor Role at Subscription Level

  1. If required, log in to the Microsoft Azure portal with the subscription administrator account.
  2. On the Azure portal, go to Subscriptions.

    Depending on how your Azure portal is set up, you might need to navigate to the Subscriptions panel through More Services.

  3. On the Subscriptions panel, select the subscription you want to use for Altus clusters.
  4. Click Access control (IAM).
  5. On the Access control (IAM) panel, click + Add to add a role.
  6. Select the following application and role:
    • Application: Cloudera Altus
    • Role: Contributor

    The following image shows the Add permissions panel where you assign the Contributor role to the Cloudera Altus application:


  7. Save the role assignment.

Assigning the Contributor Role at Resource Level

  1. If required, log in to the Microsoft Azure portal with the subscription administrator account.
  2. To assign the Contributor role to the resource group for Altus clusters, go to Resource Groups.
  3. On the Resource Groups panel, select the resource group you want to use for Altus clusters.
  4. Click Access control (IAM).
  5. On the Access control (IAM) panel, click + Add to add a role.
  6. Select the following application and role:
    • Application: Cloudera Altus
    • Role: Contributor

    The following image shows the Add permissions panel where you assign the Contributor role to the Cloudera Altus application:


  7. Save the role assignment.

    If the virtual network and network security group you want to use are in the selected resource group, they will inherit the Contributor role of the resource group. View the overview page of the resource group and verify that the virtual network and network security group are assigned the Contributor role.

    If the virtual network and network security group are not in the selected resource group, your can assign the Contributor role to those resources.

  8. To assign the Contributor role, select the virtual network and perform step 4 to step 7.

    Then select the network security group and repeat the same steps.

Creating and Assigning a Custom Role

If you do not want to provide Altus with the level of access that a Contributor role provides, you can create a custom role that includes only the permissions required by the Altus application. Then assign the role to the Cloudera Altus application at the subscription level.

To create a custom role and assign it to the Cloudera Altus application, complete the following steps:
  1. Create a JSON file with the permissions required by Altus and set the scope at subscription level.
    The JSON file must have the following fields and permissions:
    {
    "Name": "RoleName",
    "Description": "CustomRoleDescription",
    "Actions": [
                "Microsoft.Network/virtualNetworks/read",
                "Microsoft.ResourceHealth/availabilityStatuses/read",
                "Microsoft.Resources/subscriptions/resourceGroups/read",
                "Microsoft.Network/networkSecurityGroups/read",
                "Microsoft.Network/networkSecurityGroups/join/action",
                "Microsoft.Network/publicIPAddresses/join/action",
                "Microsoft.Network/virtualNetworks/subnets/join/action",
                "Microsoft.Compute/disks/delete",
                "Microsoft.Compute/disks/write",
                "Microsoft.Compute/virtualMachines/extensions/read",
                "Microsoft.Compute/virtualMachines/extensions/write",
                "Microsoft.Compute/disks/read",
                "Microsoft.Network/networkInterfaces/read",
                "Microsoft.Network/networkInterfaces/join/action",
                "Microsoft.Network/networkInterfaces/write",
                "Microsoft.Network/networkInterfaces/delete",
                "Microsoft.Network/publicIPAddresses/read",
                "Microsoft.Compute/virtualMachines/delete",
                "Microsoft.Compute/virtualMachines/write",
                "Microsoft.Compute/virtualMachines/read",
                "Microsoft.Network/publicIPAddresses/delete",
                "Microsoft.Network/publicIPAddresses/write",
                "Microsoft.ManagedIdentity/userAssignedIdentities/read",
                "Microsoft.ManagedIdentity/userAssignedIdentities/assign/action",
                "Microsoft.MarketplaceOrdering/offertypes/publishers/offers/plans/agreements/read"
               ],
    "AssignableScopes": ["/subscriptions/YourSubscriptionID/"]
    }
  2. Use the JSON file to create a custom role in Azure.
    You must use the Azure CLI to create the custom role. Run the following command:
    az role definition create --role-definition @~/JSONFileName.json

    The command creates a role with the name you specified in the JSON file.

  3. Assign the role to the Cloudera Altus application at subscription level.

    You can use the Azure portal to assign the custom role to the Cloudera Altus application.

    To assign the role at subscription level, follow the instructions in Assigning the Contributor Role at Subscription Level, selecting the role that you created for the Cloudera Altus application instead of the Contributor role.

Enabling Programmatic Deployment of the Altus VM Image

Microsoft requires that you explicitly agree to the terms of use of any VM image in the Azure Marketplace that will be deployed in your Azure subscription. After you agree to the terms of use for the VM image, you must also allow Altus to programmatically deploy the VM image in the clusters Altus creates in your subscription.

To enable programmatic deployment of the Cloudera Altus VM image, follow the instructions in this Microsoft Azure blog: Working with Marketplace Images on Azure Resource Manager.

To allow programmatic deployment of the Cloudera Altus VM image:
  1. Go to the Cloudera Altus VM image Cloudera Altus Service Image in the Azure Marketplace.
  2. Select an Cloudera Altus VM image.
  3. At the bottom of the page, click Want to deploy programmatically? Get Started to allow programmatic deployment of the VM image to your subscription.
  4. Click Enable to agree to the terms of use.
  5. Click Save.

The following image shows the Cloudera Altus Service VM image on the Azure Marketplace:


Step 4. Create the Altus Environment

The resources to be used for Altus clusters and jobs must be available in the Azure subscription before you create the Altus environment. You must be an Altus administrator to perform this task.

After you complete the access delegation process on the Azure portal, go back to the Altus console to complete the process to create the Altus environment using the Environment Wizard.

To create the Altus environment using the Environment Wizard:
  1. Go back to the Wizard page of the Altus console.
  2. On the General Settings page, enter the following information:
    Property Description
    Environment Name Name of the Altus environment to create.
    Region Region where Altus creates the Altus clusters.

    If you used the ARM template provided by Altus to create resources, the region must match the Location property in the BASICS section of the ARM template. See Step 1. Create the Azure Resources with the Altus ARM Template.

    Workload Analytics Enables Altus to send logs, metrics, and configuration details to Workload Analytics for jobs that run on clusters created in the Altus environment. Workload Analytics uses the information to perform health checks and create baselines for jobs.

    For more information about the Workload Analytics tool, see Workload Analytics.

    Secure Clusters Enables authentication and wire encryption for the clusters created using this environment.

    For more information about the enabling secure clusters, see Enable Secure Clusters.

  3. Click Next.

    On the subsequent pages, specify the network and instance settings of the resources in your Azure subscription that you want to use for all clusters created with this Altus environment. The resources must be available in your Azure subscription.

  4. On the Network Settings page, enter the following information in the Virtual Network section:
    Property Description
    Name Name of the virtual network in Azure that you want to use for Altus clusters created with this environment.

    If you used the ARM template provided by Altus to create resources, the default name for the virtual network is altus-quickstart-vnet.

    Resource Group Name of the resource group to which the virtual network belongs.

    If you used the ARM template provided by Altus to create resources, the name of the resource group for the virtual network is the name you specified for the Resource Group property in the BASICS section of the ARM template. See Step 1. Create the Azure Resources with the Altus ARM Template.

    Subnet Name Name of the subnet in the virtual network for use by the Altus clusters.
  5. On the Network Security Group section, enter the following information:
    Property Description
    Name Name of the network security group in Azure that you want to use Altus clusters created with this environment.

    If you used the ARM template provided by Altus to create resources, the default name for the network security group is altus-quickstart-nsg.

    Resource Group Name of the resource group to which the network security group belongs.

    If you used the ARM template provided by Altus to create resources, the name of the resource group for the network security group is the name you specified for the Resource Group property in the BASICS section of the ARM template. See Step 1. Create the Azure Resources with the Altus ARM Template.

  6. The Security Group Inbound Rules section displays the Cloudera Altus IP addresses:
    Property Description
    Cloudera Altus IPs To manage the clusters in your Azure subscription, Cloudera Altus requires access through the following IP address:

    52.88.35.116/32

    52.37.120.7/32

    50.112.20.144/32

    34.211.1.60/32

    34.210.228.237/32

    52.26.206.120/32

    Set up inbound rules in the network security group in your Azure subscription to allow access through the Altus IP address.

  7. Click Next.
  8. On the Instance Settings page, enter the following information:
    Property Description
    Cluster Node Resource Group Name of the resource group that you want to use for the Altus clusters.

    If you used the ARM template provided by Altus to create resources, the name of the cluster node resource group is the name you specified for the Resource Group property in the BASICS section of the ARM template. See Step 1. Create the Azure Resources with the Altus ARM Template.

    When Altus creates a cluster using this environment, Altus adds the cluster to the resource group you specify. When the cluster is terminated, Altus removes the cluster from the resource group.

    User Assigned Managed Service Identity Resource Group Name of the resource group to which the user assigned MSI you create for the Altus clusters belongs.

    Altus uses the user assigned MSI name and resource group name to uniquely identify the user assigned MSI that it assigns to the Altus VM.

    If you used the ARM template provided by Altus to create resources, the name of the resource group for the user-assigned MSI is the name you specified for the Resource Group property in the BASICS section of the ARM template. See Step 1. Create the Azure Resources with the Altus ARM Template.

    User Assigned Managed Service Identity Name Name of the user assigned Managed Service Identity (MSI) with permissions to the Azure Data Lake Store used for the jobs that run on the clusters created with this environment.

    If you used the ARM template provided by Altus to create resources, the default name for the user-assigned MSI is altus-quickstart-ua-msi.

    When Altus creates a cluster using this environment, Altus uses the user assigned MSI name and resource group name to uniquely identify the user assigned MSI that it assigns to the Altus VM.

    For information about setting up a user assigned MSI, see the Microsoft documentation: Create, list or delete a user assigned identity using the Azure CLI.

    You must configure access control lists (ACLs) to allow the user assigned MSI to read from and write to files and folders in Azure Data Lake Store. Cloudera recommends that you configure a default ACL on the folders in Azure Data Lake Store which grants read and write permission to the user assigned MSI. For more information about setting up an ACL for files and folders in Azure Data Lake Store, see the Microsoft documentation on assigning users or security groups as ACLs.

    You can also add the user assigned MSI to an AAD group to control access to files and directories in the Azure Data Lake Store account that you use for Altus clusters.

    Log Archive Enables Altus to archive job logs in an Azure Data Lake Store. Altus writes job logs to Azure Data Lake Store after a job completes.

    If you enable the Log Archive option, you must specify the folder in the Azure Data Lake Store for archiving the job logs. In the Data Lake Store Log Archive Path field, enter the name the Azure Data Lake Store folder designated for the log archive.

    The log archive path requires the absolute path to the log directory in the following format: adl://YourADLSAccountName.azuredatalakestore.net/Your/Path/To/Directory

  9. Click Next.

    Altus displays a summary of the resources you specified for the Altus environment, including the Azure Active Directory tenant ID for the Cloudera Altus application. Review and verify that the Azure resource information is correct.

  10. Click Create.

    Altus creates the environment with the resources that you specified.