Nomad Multi-Region Deployment

Description

This repo contains Terraform code for deploying Nomad clusters across multiple cloud regions.

Table of Contents

• Nomad Multi-Region Deployment
  • Description
  • Table of Contents
  • Architecture Diagram
  • Usage
    • Authenticate to AWS using Environment Variables
    • Add ssh-key to AWS Regions for EC2 SSH Access
    • Authenticate to HCP Packer via Environment Variables
    • Build AMI using Packer and copy AMI to regions
    • This iteration associated to the Fingerprint is complete. Change the Packer Build Fingerprint
    • Deploy Infrastructure with Terraform
    • Output Example
    • Connecting to Instances
    • Federate Nomad Clusters
    • Troubleshooting

Architecture Diagram

Usage

Authenticate to AWS using Environment Variables

export AWS_ACCESS_KEY_ID=ASIA3PL2TGENYK26B...
export AWS_SECRET_ACCESS_KEY=nAjFUOdynJ+OH...
export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjE...

Add ssh-key to AWS Regions for EC2 SSH Access

ssh-keygen -y -f ahar-keypair-2023.pem > $HOME/.ssh/id_rsa_MyKeyPair.pub

AWS_REGIONS=("us-east-1" "us-east-2" "us-west-1" "us-west-2")
for each_region in ${AWS_REGIONS} ; do aws ec2 import-key-pair --key-name ahar-keypair-2024 --public-key-material fileb://$HOME/.ssh/id_rsa_MyKeyPair.pub --region $each_region ; done

#Alternate method to populate keys across all regions
#
#AWS_REGIONS="$(aws ec2 describe-regions --query 'Regions[].RegionName' --output text)"
#uncomment next line if using zsh shell
#setopt shwordsplit

Authenticate to HCP Packer via Environment Variables

export HCP_CLIENT_ID=xVRieBpO9KY...
export HCP_CLIENT_SECRET=kVAq8HVMu...

Build AMI using Packer and copy AMI to regions

cd packer
packer init image.pkr.hcl
packer build -var-file=variables-packer.hcl image.pkr.hcl
cd ..

This iteration associated to the Fingerprint is complete. Change the Packer Build Fingerprint

#Note: This is Useful when you want to deploy a new packer image without a new git commit (must modify image.pkr.hcl)
export HCP_PACKER_BUILD_FINGERPRINT=$(openssl sha1 image.pkr.hcl | awk '{print $NF}' )

Deploy Infrastructure with Terraform

#terraform destroy --auto-approve
terraform apply --auto-approve
rm nomad.token
rm nomad2.token
rm nomad1.token
./post-setup.sh

Output Example

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

consul_bootstrap_token_secret = "8f966fde72-d76a-2f4d-5b97-e7c9fafa8d"
region_1_nomad_ui = "http://35.221.181.49:4646"
region_1_server_ip = "35.221.181.49"
region_1_vpc_id = "vpc-01g98f754c309be2d"
region_2_nomad_ui = "http://19.221.215.253:4646"
region_2_server_ip = "19.221.215.253"
region_2_vpc_id = "vpc-0fd824260c5035522"

Connecting to Instances

ssh -i ahar-keypair-2023.pem ubuntu@$(terraform output -raw "region_1_server_ip")

Federate Nomad Clusters

nomad server join $(terraform output -raw "region_2_server_ip")

Troubleshooting

#View Log Files on Nomad Server/Client Nodes
cat /var/log/syslog

#Transit Gateway Troubleshooting
[Troubleshooting Transit Gateway Routing](https://aws.amazon.com/premiumsupport/knowledge-center/transit-gateway-fix-vpc-connection/)

#Add permission to anonymous Consul token
export CONSUL_HTTP_TOKEN=8f966e72-d76a-2f4d-5b97-e7c9928d
consul acl token update -accessor-id=00000000-0000-0000-0000-000000000002 -append-policy-name=nomad-auto-join