Using Custom DNS in AWS
- Custom private DNS names for your EC2 instances. Use of custom private DNS names requires also configuring a custom DNS server (or Amazon Route 53).
- A custom DNS server, instead of the Amazon-provided DNS server on the VPC.
How Altus Director and Cloudera Manager Communicate with Cluster Instances
When working with the AWS DNS settings for EC2 instances created and managed by Altus Director, it is useful to understand how Altus Director and Cloudera Manager access the instances in a cluster on AWS.
How Cloudera Manager Communicates with Cluster Instances
Whether or not you are using auto-TLS affects the way Altus Director configures Cloudera Manager to communicate among the instances.
- When auto-TLS is not used, Cloudera Manager accesses cluster instances through their private IP address.
- When auto-TLS is used, Cloudera Manager uses private hostnames, as well as IP addresses, to access the instances. Specifically, Cloudera Manager uses private hostnames to configure the Cloudera Manager agents on the EC2 instances. This is because TLS certificates will be generated using the hostname, and then used for secure communication between Cloudera Manager and the cluster.
How Altus Director Communicates with Cluster Instances
- Altus Director uses the IP addresses and hostnames provided by the AWS metadata service to connect via SSH into the cluster instances. When custom hostnames have been configured on the instances, the hostname will not match the name provided by the AWS metadata service.
- In a cross-region, cross-VPC, or cross-cloud-provider configuration, Altus Director will use the private IP address if it is available, and will fall back on the private hostname, public IP address or hostname, in that order, if it is not. For more information on these configurations, see Running Altus Director and Cloudera Manager in Different Regions or Clouds
Using Custom DNS Names
When users configure their own custom hostnames, the AWS metadata service does not update its record of cluster hostnames, so the hostname on the instance level is different from the hostname on the cloud provider level.
- Configuring a custom domain name
- Configuring a custom hostname
Configuring a Custom Domain Name in AWS
- Create a new DHCP options set.
- Set the domain name in the DHCP options set for the VPC.
- Set the IP address of the DNS server in the Domain name servers field of the DHCP option set for the VPC.
- Set enableDnsSupport to false so that the Amazon-provided DNS server in the VPC is not enabled.
- Set up your own DNS server (or Amazon Route 53).
- Add a record for your domain name to your custom DNS server (or Amazon Route 53).
Configuring a Custom Hostname in AWS
- Set up the hostname. There are many ways to do this, for example, through your AMI, user data, orchestration framework (such as Chef or Ansible), or bootstrap scripts.
- Add a record for your hostname to your custom DNS (or Route 53).
Using a Custom DNS Server
You can configure a custom DNS server by entering its IP address in the DHCP options set for the VPC. If you use your own DNS server, ensure that the server can resolve hostnames in the Amazon standard format, ip-x-x-x-x, or change the name on the host to a name that the DNS server can resolve. You can either pre-populate your DNS server with the expected names, or use a tool like nsupdate to register the hosts in the DNS server as they come up.
- In the AWS Admin Console, choose VPC.
- Disable the Amazon-provided DNS server in the VPC by setting enableDnsSupport to false.
- In the VPC dashboard, select the VPC you are using.
- In the Actions dropdown, click Edit DNS Resolution.
- Set the value to No.
- In the left hand pane, click DHCP Options sets.
- Click Create DHCP options set.
- Enter the IP address of your domain name server in the Domain Name Servers field.
- Optionally, configure other fields in the DHCP options set and click Create DHCP options set.
- In the VPC settings, specify the new DHCP options set for your VPC.
- Configure your DNS server to accept updates, if possible. If you do not want your DNS server to accept updates (for example, because of your organization's security policies), configure the DNS server with hostnames before launching the cluster. Work with your infrastructure team to ensure that the hostnames you will use are added to your DNS server.
Whether your domain names use the auto-generated format used by AWS or you create a custom hostname in a format unique to your instances, you need to ensure that your custom DNS server can resolve the cluster hostnames. There are a number of ways to do this: by using a script, by adapting the scripts on the Cloudera GitHub site that were created for Microsoft Azure to in AWS, or by manual configuration, or by using an orchestration framework, such as Chef or Ansible.