Environment Setup for AWS

Altus requires access to your AWS account to perform tasks on an Altus user's behalf. To allow Altus to create clusters and run jobs in your AWS account, you must set up a cross-account access role for Altus in your AWS account.

You create an Altus environment to store information about the cross-account access role and resources in your AWS account that are available to an Altus user.

You can use the following tools to set up an Altus environment for AWS:
Environment Quickstart
The Environment Quickstart simplifies the process of creating an environment. When you use Quickstart, Cloudera Altus creates a cross-account access role, VPC, security group, and other resources in your AWS account using a CloudFormation template.

The Environment Quickstart allows you to specify the names for resources, such as the names for the delegation role and instance profile role. It creates the resources in your AWS account with the names you specify. You must provide your AWS credentials to allow Altus to access your AWS account and create the resources.

Environment Wizard
The Environment Wizard guides you through creating the cross-account access role and other resources in the AWS management console. As you create the subnet, security group, and other resources in AWS, you create the Altus environment by providing the resource information in the Environment Wizard.

If users must access different resources, create an Altus environment for each set of resources to better manage how users access AWS resources.

Creating the Altus Environment with the Quickstart

The Environment Quickstart uses a CloudFormation template to create a stack of resources in your AWS account. You provide the CloudFormation stack name and resource names in the Environment Quickstart and Altus creates the stack and resources in AWS with the names you specify.

The Environment Quickstart accesses your AWS account to create the resources for Altus users. You must provide your AWS credentials to allow the Environment Quickstart access to your AWS account.

If you do not want to provide your AWS credentials, you can create the required resources directly in AWS and use the Environment Wizard to create the environment. For more information about using the wizard, see Creating the Altus Environment with the Wizard.

To create the Altus environment using the Environment Quickstart:
  1. Sign in to the Cloudera Altus console:

    https://console.altus.cloudera.com/

  2. On the side navigation panel, click Environments.

    The Environments page displays the list of all the Altus environments that are available for your AWS account. It also shows the AWS region where the resources are located.

  3. Click Quickstart.
  4. On the Environment Quickstart page, click Create.
  5. Set the name and region for the Altus environment:
    Property Description
    Altus Environment Name The name of the Altus environment to create. The environment name must be unique within your AWS account.

    Altus adds the environment name as a prefix to the names of the resources in the environment to make the resource names unique.

    The name must be an alphanumeric string of up to 40 characters. It can include underscores (_). It cannot include a space.

    AWS Region Region in which to create the AWS resource stack.
  6. Select the options for the clusters that use the resources in this environment:
    Property Description
    Workload XM Select this option to enable Workload Analytics for jobs that run on clusters created in the Altus environment.

    For more information about enabling Workload Analytics, see Enable Workload XM.

    Log Archive Select this option to enable Altus to archive cluster and job logs in an S3 bucket. Altus creates an S3 bucket to use for archiving cluster and job logs with the name EnvironmentNamePrefix-log-archive-bucket.

    For more information about enabling the log archive, see Enable Log Archive.

    Secure Clusters Select this option to enable authentication and wire and at-rest data encryption for the clusters created using this environment.

    For more information about the enabling secure clusters, see Enable Secure Clusters.

    Public IPs Select this option to allow Altus to set up public IP addresses for the clusters created using this environment.

    For more information about the enabling public IP addresses, see Enable Public IPs.

    Cloudera Navigator Integration Select this option to enable Altus to send cluster and workload metadata to an S3 bucket configured as a metadata resource in Cloudera Navigator. Cloudera Navigator can extract metadata from the S3 bucket to generate analytics and data lineage for Altus clusters and workloads. Altus creates an S3 bucket to use for storing Altus cluster and workload metadata with the name EnvironmentNamePrefix-navigator-data-bucket.

    For more information about enabling Cloudera Navigator integration, see Enable Cloudera Navigator Integration.

    S3Guard Consistency Select this option to enable S3Guard in the S3 buckets used for data processed in clusters that use this environment. Altus creates a DynamoDB table for storing the S3Guard metadata with the name EnvironmentNamePrefixs3GuardMetadata.

    For more information about enabling S3Guard, see Enable S3Guard Consistency.

    If you do not want to use the default names for the resources that Altus creates, you can specify the names for the resources. You might choose to customize the resource names to avoid name conflicts in your AWS account or to follow a naming convention used in your organization.

  7. To specify the names for the resources that Altus creates in your AWS account, click Customize Resource Names.
    The console displays the resources that Altus creates in your AWS account with default names and no prefix. The list includes the following resources, in addition to the AWS region and options displayed on the main page:
    Property Description
    Environment Environment that you are creating.
    AWS Stack AWS resource stack that will be created using a CloudFormation template.
    AWS Delegated Role Role that provides delegated access to the Cloudera Altus account.
    AWS Instance Profile Role Instance profile role for the EC2 instances in Altus clusters.

    You can download a copy of the CloudFormation template as a JSON file to view the list of resources that Altus will create in your AWS account.

  8. To download a copy of the CloudFormation template, click Download CloudFormation Template.

    Altus downloads a file named altus-cf-template.json to your local machine.

  9. Click Create to allow Altus to create the CloudFormation resource stack in your AWS account.
  10. On the Provide AWS Credentials page, enter the credentials that Cloudera can use to log in to your AWS account and create the resources.

    You must provide an access key that is registered with AWS.

  11. Click Proceed.

    The Environment Quickstart creates the Altus environment with the environment and resource names that you specified.

Creating the Altus Environment with the Wizard

The Environment Wizard guides you through the steps of setting up the resources you need for Altus clusters in your AWS account. The Wizard provides instructions for setting up a cross-account access role, configuring the network, and creating the EC2 instance policies for the clusters. If you are not familiar with how to create the resources and configure the network in AWS, you can turn on the Step-by-step Guide on the Environment Wizard to view the instructions.

If you already have a cross-account access role and resources in your AWS account that you want to use for Altus, you can enter the resource information in the Environment Wizard without creating new resources.

To create the Altus environment using the Environment Wizard:
  1. Sign in to the Cloudera Altus console:

    https://console.altus.cloudera.com/

  2. On the side navigation panel, click Environments.

    The Environments page displays the list of all the Altus environments that are available for your AWS account. It also shows the AWS region where the resources are located.

  3. Click Wizard.
  4. On the Environment Wizard page, click Create.
  5. On the General Settings page, set the name for the Altus environment.
  6. Select the options for the clusters that use the resources in this environment:
    Property Description
    Workload XM Select this option to enable Workload Analytics for jobs that run on clusters created in the Altus environment.

    For more information about enabling Workload XM, see Enable Workload XM.

    Secure Clusters Select this option to enable authentication and wire and at-rest data encryption for the clusters created using this environment.

    If you select the Secure Clusters option, you can provide the encryption key that Altus uses to encrypt the EBS volumes. In the EBS KMS Encryption Key field, enter the ARN of the KMS key you want to use for encryption. If you do not provide a key, Altus uses the default AWS managed key for encryption.

    For more information about enabling secure clusters, see Enable Secure Clusters.

  7. Click Next.

    On the AWS Delegated Access page, you create the IAM policy and cross-account role to allow Altus to access your AWS account to create clusters on your behalf.

    The Wizard provides the JSON that you can use to create an AWS IAM policy. With this IAM policy, you create a cross-account role to provide Altus access to your AWS account.

  8. Follow the instructions on the Step-by-step Guide to create the IAM policy in your account with the JSON provided.
  9. After you create the IAM policy, follow the instructions on the Step-by-step Guide to create a cross-account access role for Altus with the IAM policy.

    The cross-account access role requires the IAM policy you created in the previous step and the Altus account ID and external ID. Copy the values from the Altus Account ID and External ID fields displayed on the console

    AWS assigns an AWS resource name (ARN) to the cross-account access role that you create.

  10. In the Cross-account Role ARN field, enter the ARN for the cross-account role.
  11. Click Next.

    Altus creates clusters in a VPC in your AWS account. If you do not yet have a VPC identified for Altus clusters, you can follow the instructions on the Step-by-step Guide to create a VPC and configure the VPC with a subnet and a security group. For more information about the VPC requirements for Altus, see AWS Account Requirements.

  12. On the Network Settings page, specify the region where the VPC is located.

    The region where the VPC is located is the same region where the subnet and security group are located.

  13. Specify the subnet ID of the subnet that you configured in the VPC.

    You can specify multiple subnets.

  14. Specify the security group ID of the security group that you configured in the VPC.

    You can specify multiple security groups.

  15. Set the IP address option for the clusters that use the resources in this environment:
    Property Description
    Public IPs Select this option to allow Altus to assign public IP addresses to the clusters created using this environment and make the clusters accessible from external networks.

    For more information about the enabling public IP addresses, see Enable Public IPs.

  16. Click Next.
  17. On the Instance Profile Role page, select additional options for the clusters that use the resources in this environment:
    Property Description
    Log Archive Select this option to enable Altus to archive cluster and job logs in an S3 bucket.

    If you enable the Log Archive option, you must specify the name of the S3 bucket to use for archiving cluster and job logs. In the S3 Log Bucket Name field, enter the name the S3 bucket designated for the log archive.

    If you do not have an S3 bucket set up for the Altus logs, follow the instructions on the Step-by-step Guide to create an S3 bucket for the logs in the same region as the VPC. For more information about enabling workload logs, see Enable Log Archive.

    To ensure that the Altus logs in the S3 bucket are secure, Cloudera recommends that you encrypt the logs using the default encryption for S3 buckets provided by AWS. For more information about the S3 default encryption, see Amazon S3 Default Encryption for S3 Buckets.

    S3Guard Consistency Select this option to enable S3Guard in the S3 buckets used for data processed in clusters that use this environment.

    If you enable this option, you must specify the name of the DynamoDB table in which to store the S3Guard metadata. In the S3Guard Metadata Table Namefield, enter the name of the table. If the table does not exist, Altus creates the table. If the table exists, Altus reuses the table.

    For more information about enabling S3Guard, see Enable S3Guard Consistency.

    Cloudera Navigator Integration Select this option to enable Altus to send cluster and workload metadata to an S3 bucket configured as a metadata resource in Cloudera Navigator. Cloudera Navigator can extract metadata from the S3 bucket to generate analytics and data lineage for Altus clusters and workloads. Specify the name of the S3 bucket to use for storing Altus cluster and workload metadata.

    For more information about enabling Cloudera Navigator integration, see Enable Cloudera Navigator Integration.

    If you enable any of the options, you must create an IAM policy to manage how Altus clusters access S3 buckets that are used by these options. Altus provides the JSON that you could use to create the policy.

  18. If you enable an option, click Generate Policy to generate the JSON for the IAM policy.

    You need to create an instance profile for the cluster EC2 instances that Altus creates in your AWS account. To create an instance profile, you must first create an EC2 instance policy and then an EC2 instance role with that policy.

  19. Follow the instructions on the Step-by-step Guide to create an EC2 instance policy.

    The policy defines the access rules for the S3 buckets used by the options.

    After you create the policy, create an EC2 instance role and assign the policy to the role.

  20. Follow the instructions on the Step-by-step Guide to create an EC2 instance role with the policy you created in the previous step.

    When you create an EC2 instance role, AWS creates an EC2 instance profile with the same name as the EC2 instance role. For more information about EC2 instance profile and roles, see the AWS documentation on using EC2 roles and profiles.

  21. In the AWS instance profile name field, enter the name of the instance profile name for the Altus clusters.
  22. Click Next.
  23. Review the configuration for the Altus environment to verify that it includes the resources that you want to use for Altus clusters.
  24. Click Create.

    The Environment Wizard creates the Altus environment with the resources and name that you specified.