Recommended Approach for Immutable Atmos Packer AMI's

[jaguer0]

Following up from the Customer Workshop, we’re seeking the recommended pattern for building Immutable AMIs using Atmos and Packer within the Reference Architecture.

1. Repo Structure: Is it better to use a dedicated Packer repo or include it in the infrastructure monorepo?
2. Account Strategy: Should we build in core-auto and store AMI in core-artifacts?
3. Promotion: What is the best way to share AMIs from the artifacts account to the plat OU? (we build once & promote ) Any additional iam's needed beyond packer ami_ vars in the infra repo?
4. Automation: Erik mentioned you may have existing GitHub Actions for this, are there examples we can reference?

[aknysh]

@jaguer0 thanks for opening the discussion. Here are short answers to your questions:

1. Repo Structure: Is it better to use a dedicated Packer repo or include it in the infrastructure monorepo?
   Both are good approaches, depends on your requirements. For some clients, we implemented Atmos/Packer components/stacks in the same 'infra' repo as the Terraform/OpenTofu components. For other clients, we created dedicated repos for packer components and Atmos stacks to build a particular type of AMIs, e.g. one repo for AWS AL2, another repo for AWS Ubuntu, etc. It mostly depends on these factors:

   • Teams, ownership, security, access control: if the same team (e.g. DevOps) manages the terraform and packer components/stacks and has permissions to provision terraform components and run packer commands, and it's the same GitHub codeowners, etc. - then terraform and packer can be in the same repo. If you want to give your teams different permissions (to AWS resources provisioned with terraform and with packer, and to GitHub repos), then it would be wise to separate the packer components/stacks into different repos
   • If the packer components read the remote state of the terraform components (e.g. !terraform.state), then are better together in the same repo with the same Atmos config and all the stacks
   • Automation: we build AMIs with Atmos+Packer using GitHub actions. If you need to separate the GH workflows (for security, access control, and other reasons), then the packer components/stacks should be in a separate repo

2. Account Strategy: Should we build in core-auto and store AMI in core-artifacts?
   Yes, this is what we usually do. We run a GitHub action workflow that executes the Atmos/Packer commands and build AMIs in core-auto (where the GH workflow executes), then store the AMIs in core-artifacts. Then, in the same GH workflow, we share/promote the AMIs with other accounts (prod, dev, staging).

3. Promotion: What is the best way to share AMIs from the artifacts account to the plat OU? (we build once & promote ) Any additional iam's needed beyond packer ami_ vars in the infra repo?
   We build AMIs using GH actions and GitHub OIDC. The GH workflow assumes an AIM role with the following permissions:

data "aws_iam_policy_document" "packer_cicd_policy" {
count = local.packer_cicd_policy_enabled ? 1 : 0

############################################
# Read-only describe for EC2
############################################
statement {
  sid    = "EC2Describe"
  effect = "Allow"
  actions = [
    "ec2:Describe*"
  ]
  resources = ["*"]
}

########################################################################
# Core EC2/AMI lifecycle for amazon-ebs (incl. tagging, temp SG/key)
########################################################################
statement {
  sid    = "EC2BuildLifecycle"
  effect = "Allow"
  actions = [
    # Instance lifecycle for Packer build
    "ec2:RunInstances",
    "ec2:TerminateInstances",
    "ec2:StopInstances",
    "ec2:StartInstances",

    # AMI lifecycle
    "ec2:CreateImage",
    "ec2:RegisterImage",
    "ec2:DeregisterImage",
    "ec2:ModifyImageAttribute",
    "ec2:CopyImage",

    # Volumes/Snapshots
    "ec2:CreateVolume",
    "ec2:DeleteVolume",
    "ec2:AttachVolume",
    "ec2:DetachVolume",
    "ec2:CreateSnapshot",
    "ec2:CreateSnapshots",
    "ec2:DeleteSnapshot",

    # Temporary KeyPair + Security Group (if Packer creates them)
    "ec2:CreateKeyPair",
    "ec2:DeleteKeyPair",
    "ec2:CreateSecurityGroup",
    "ec2:DeleteSecurityGroup",
    "ec2:AuthorizeSecurityGroupIngress",
    "ec2:AuthorizeSecurityGroupEgress",
    "ec2:RevokeSecurityGroupIngress",
    "ec2:RevokeSecurityGroupEgress",

    # Tagging
    "ec2:CreateTags",
    "ec2:DeleteTags",

    # Toggle IP / NIC attributes
    "ec2:ModifyInstanceAttribute",
    "ec2:ModifyNetworkInterfaceAttribute"
  ]
  resources = ["*"]
}

# Explicit resource-scoped tagging convenience
statement {
  sid    = "TagCommonEc2Resources"
  effect = "Allow"
  actions = [
    "ec2:CreateTags",
    "ec2:DeleteTags"
  ]
  resources = [
    "arn:aws:ec2:*:*:instance/*",
    "arn:aws:ec2:*:*:image/*",
    "arn:aws:ec2:*:*:volume/*",
    "arn:aws:ec2:*:*:snapshot/*",
    "arn:aws:ec2:*:*:security-group/*"
  ]
}

# Snapshot attribute modifications
statement {
  sid    = "ModifySnapshotAttributes"
  effect = "Allow"
  actions = [
    "ec2:ModifySnapshotAttribute"
  ]
  resources = ["arn:aws:ec2:*:*:snapshot/*"]
}

#########################################################
# AMI launch permissions (accounts, orgs, and OUs)
#########################################################
statement {
  sid    = "AllowModifyImageLaunchPermissions"
  effect = "Allow"
  actions = [
    "ec2:ModifyImageAttribute"
  ]
  resources = ["arn:aws:ec2:*:*:image/*"]
}

#########################################################
# Pass the instance profile to EC2 (optional)
#########################################################
dynamic "statement" {
  for_each = lookup(var.packer_cicd_policy_configuration, "instance_profile_arn", "") != "" ? [1] : []
  content {
    sid     = "PassRoleToEC2"
    effect  = "Allow"
    actions = ["iam:PassRole"]
    resources = [
      lookup(var.packer_cicd_policy_configuration, "instance_profile_arn", "")
    ]
    condition {
      test     = "StringEquals"
      variable = "iam:PassedToService"
      values   = ["ec2.amazonaws.com"]
    }
  }
}

############################################
# Optional: KMS access for EBS/AMI
############################################
dynamic "statement" {
  for_each = lookup(var.packer_cicd_policy_configuration, "enable_kms_access", false) ? [1] : []
  content {
    sid    = "KmsForEbsAndAmiEncryption"
    effect = "Allow"
    actions = [
      "kms:DescribeKey",
      "kms:CreateGrant",
      "kms:ListGrants",
      "kms:RevokeGrant",
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*"
    ]
    resources = lookup(var.packer_cicd_policy_configuration, "kms_key_arns", ["*"])
    condition {
      test     = "Bool"
      variable = "kms:GrantIsForAWSResource"
      values   = ["true"]
    }
  }
}

#########################################################
# Auto Scaling + Launch Templates (for ASG launches)
#########################################################
statement {
  sid    = "AutoScalingLifecycle"
  effect = "Allow"
  actions = [
    "autoscaling:CreateAutoScalingGroup",
    "autoscaling:UpdateAutoScalingGroup",
    "autoscaling:DeleteAutoScalingGroup",
    "autoscaling:SetDesiredCapacity",
    "autoscaling:AttachInstances",
    "autoscaling:DetachInstances",
    "autoscaling:TerminateInstanceInAutoScalingGroup",
    "autoscaling:EnableMetricsCollection",
    "autoscaling:DisableMetricsCollection",
    "autoscaling:Describe*",
    "autoscaling:CreateLaunchConfiguration",
    "autoscaling:DeleteLaunchConfiguration"
  ]
  resources = ["*"]
}

statement {
  sid    = "LaunchTemplateLifecycle"
  effect = "Allow"
  actions = [
    "ec2:CreateLaunchTemplate",
    "ec2:DeleteLaunchTemplate",
    "ec2:CreateLaunchTemplateVersion",
    "ec2:DeleteLaunchTemplateVersions",
    "ec2:ModifyLaunchTemplate",
    "ec2:DescribeLaunchTemplates",
    "ec2:DescribeLaunchTemplateVersions"
  ]
  resources = ["*"]
}
}

After assuming the github-oidc role, the GH workflow executes the following script to share the AMI and the snapshot with the other accounts:

# Normalize -> array ACCOUNTS (valid 12-digit, trimmed, unique)
ACCOUNTS=()
while IFS= read -r line; do
  # strip comments
  line="${line%%#*}"
  # trim whitespace
  line="$(printf '%s' "$line" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')"
  [[ -z "$line" ]] && continue
  [[ "$line" =~ ^[0-9]{12}$ ]] || continue
  ACCOUNTS+=("$line")
done < <(printf '%s\n' "$SHARE_ACCOUNT_IDS" | sort -u)

if [[ ${#ACCOUNTS[@]} -eq 0 ]]; then
  echo "Error: built-in account list is empty or invalid." >&2
  exit 1
fi

echo "Describing AMI $AMI_ID ..."
IMG_JSON="$(aws ec2 describe-images --image-ids "$AMI_ID" --no-cli-pager --output json || true)"
if [[ "$(jq -r '.Images | length' <<<"$IMG_JSON")" -eq 0 ]]; then
  echo "AMI not found: $AMI_ID" >&2
  exit 1
fi

echo "Target accounts (${#ACCOUNTS[@]}): ${ACCOUNTS[*]}"

# 1) Share AMI launch permissions
echo "Adding AMI launch permissions..."
LP_ENTRIES=()
for acct in "${ACCOUNTS[@]}"; do
  # extra safety: only allow strict 12-digit accounts
  [[ "$acct" =~ ^[0-9]{12}$ ]] || { echo "Invalid account in list: $acct" >&2; exit 1; }
  LP_ENTRIES+=("{UserId=${acct}}")
done
LP_ARG="Add=[$(IFS=,; echo "${LP_ENTRIES[*]}")]"

aws ec2 modify-image-attribute \
  --image-id "$AMI_ID" \
  --launch-permission "$LP_ARG" \
  --no-cli-pager

# 2) Share all backing EBS snapshots
SNAPSHOT_IDS=()
while IFS= read -r sid; do
  SNAPSHOT_IDS+=("$sid")
done < <(jq -r '.Images[0].BlockDeviceMappings[] | select(.Ebs and .Ebs.SnapshotId) | .Ebs.SnapshotId' <<<"$IMG_JSON" | sort -u)

if [[ ${#SNAPSHOT_IDS[@]} -gt 0 ]]; then
  echo "Found snapshots: ${SNAPSHOT_IDS[*]}"
  for sid in "${SNAPSHOT_IDS[@]}"; do
    aws ec2 modify-snapshot-attribute \
      --attribute createVolumePermission \
      --operation-type add \
      --user-ids "${ACCOUNTS[@]}" \
      --snapshot-id "$sid" \
      --no-cli-pager
  done
else
  echo "No EBS snapshots found (instance-store AMI?). Skipping snapshot sharing."
fi

7. Automation: Erik mentioned you may have existing GitHub Actions for this, are there examples we can reference?
   Yes, we have GitHub actions and the scripts in private repos. There are many things that needs to be configured, and Atmos docs definitely lacks these steps. Let us discuss internally, we'll probably update/improve the Atmos Packer docs to describe the entire solution, not only the Atmos packer stack config and CLi commands, which you can find here:

• https://github.com/cloudposse/atmos/tree/main/tests/fixtures/scenarios/packer
• https://atmos.tools/cli/configuration/components/packer
• https://atmos.tools/components/packer
• https://atmos.tools/stacks/components/packer
• https://atmos.tools/cli/commands/packer/usage

we'll be working on adding this info to the Atmos docs. Meanwhile, feel free to ask any questions here or in SweetOps slack.

[jaguer0]

@aknysh Thank you, I'll review those references a bit closer in the morning, but def look forward to seeing the new docs & EBS logic !

I forgot to note, we're using the runs-on component in our ref arch. Not sure if that changes anything or makes things easier, but wanted to surface just in case