A production-shaped AWS platform — EKS, Aurora, a CloudFront/WAF edge, Datadog monitoring, and
keyless CI — built as reusable OpenTofu modules and Terragrunt stacks, isolated across
management/staging/prod accounts. It is meant to be read as much as run: the non-obvious choices
are written down as ADRs, and everything is sanitized to placeholder values so you can clone it,
fill in a handful of inputs, and plan it. Aimed at platform engineers evaluating a Terragrunt
layout, and at anyone who wants a worked example rather than a tutorial.
Three accounts, one stack file each. The management account holds state and identity and runs no workloads; the workload accounts share a shape but not a size, and the edge tier exists only in prod.
Traffic resolves via Route53 → CloudFront (WAFv2, ACM cert in us-east-1) → EKS workloads. Pods reach Aurora and Redis privately inside the VPC. State lives per-account in S3 with native locking. Full write-up: docs/architecture.md.
| Stack | Provisions | Rough monthly cost¹ |
|---|---|---|
| management | GitHub OIDC provider, IAM users/groups/IRSA baseline | ~$0 |
| staging | VPC (1 NAT), EKS + spot nodes, Aurora (1× t4g.medium), Redis (t4g.small), Route53, Datadog monitors | ~$300 |
| prod | VPC (NAT/AZ), EKS + on-demand nodes, Aurora (2× r6g.large), Redis (r7g.large), CloudFront+WAF, ACM, Route53, monitors | ~$1,500 |
¹ Approximate, us-west-2 on-demand list prices, excluding data transfer and request charges. These
are hand estimates; the plan workflow posts a precise Infracost diff
on every PR once INFRACOST_API_KEY is set.
The state bucket and its KMS key are in none of these stacks. Every account gets one, bootstrapped
once by hand on its own lifecycle from live/<account>/<region>/bootstrap/state-backend — see step
3 of the Quickstart and ADR-0011. Budget a
KMS key plus negligible storage per account for it, on top of the figures above.
Prerequisites: mise (installs the pinned OpenTofu/Terragrunt/tflint/etc.),
git, and AWS credentials for the account you're deploying into.
git clone https://github.com/bezilla/terragrunt-reference-architecture
cd terragrunt-reference-architecture
mise install # pinned toolchain
# 1. Fill in the values you own (see "Required inputs"):
for a in management staging prod; do
cp live/$a/account.hcl.example live/$a/account.hcl
done
$EDITOR live/*/account.hcl # set account_id, deploy_role_arn
# 2. Validate everything offline — no AWS credentials needed:
make validate
# 3. Bootstrap remote state. ONCE PER ACCOUNT, by hand, and it is its own lifecycle.
#
# live/root.hcl derives the backend bucket per account (tfstate-<account-id>-<region>), so
# management, staging and prod each need their own before anything else in that account can
# initialise a backend. The bootstrap unit lives OUTSIDE the environment stacks, at
# live/<account>/us-west-2/bootstrap/state-backend, so no `run --all` can plan, apply or
# destroy it — the bucket everything else stores state in is not something a stack owns.
#
# It keeps LOCAL state permanently and by design (it cannot live in the bucket it creates),
# in that directory rather than in any generated tree. This step is NOT repeatable on a
# machine that lacks that state file: recovery is by import. See ADR-0011 and
# modules/state-backend/README.md before running it a second time anywhere.
aws sso login # or however you get credentials
make bootstrap ACCOUNT=management # bucket + KMS key, on local state
# 4. Apply the management stack, then each workload account: bootstrap it, then apply its stack.
cd live/management/us-west-2/global
terragrunt stack generate
terragrunt run --all apply # into the bucket step 3 created
cd -
make bootstrap ACCOUNT=staging # once per account
cd live/staging/us-west-2/staging
terragrunt stack generate
terragrunt run --all plan # review
terragrunt run --all apply
# Then repeat for prod. Order: management → staging → prod.CI does step 4 for you via the apply workflow once you wire up OIDC (see CI/CD).
Step 3 it deliberately does not do — CI does not create the bucket it stores its own state in.
Everything else has a safe default. These do not:
| Input | Where | How to know it's right |
|---|---|---|
account_id |
live/<account>/account.hcl |
12-digit AWS account ID per environment; aws sts get-caller-identity |
deploy_role_arn |
live/<account>/account.hcl |
ARN of the role Terragrunt assumes; leave empty to use ambient creds |
| GitHub org / repo | catalog/units/iam-github-oidc |
your owner/repo, for the OIDC deploy-role trust sub claim |
| DNS zone | stack values (route53-zone, acm-certificate) |
a domain you control in the account (Route53 must be authoritative) |
| Datadog keys | DD_API_KEY / DD_APP_KEY env vars |
from Datadog; never committed — read by the datadog-monitors unit |
AWS_PLAN_ROLE_ARN / AWS_APPLY_ROLE_ARN |
repo Actions variables | ARNs from the iam-github-oidc module output; enables the CI pipeline |
| Path | What's in it |
|---|---|
modules/ |
16 reusable OpenTofu modules (each with tests, a README, and a lockfile) |
catalog/units/ |
Terragrunt unit definitions that source the modules |
live/ |
root.hcl + per-account account.hcl/region.hcl/env.hcl + environment stacks + the per-account state bootstrap |
policy/ |
OPA/conftest policies (tags, ingress, encryption, IMDSv2) + tests |
docs/ |
architecture, ADRs, and how-to guides |
.github/workflows/ |
validate (PR), plan (PR), apply (merge), drift (weekly) |
Each links to its ADR:
- OpenTofu default, Terraform drop-in — avoid the BSL, stay compatible.
- Multi-account isolation — account boundaries over VPC-only separation.
- Terragrunt Stacks — one stack file per environment, not copied unit trees.
- Modules in-repo — one clone, referenced by repo root.
- S3 native locking — no DynamoDB lock table.
- Managed node groups — one worker strategy, not three.
- Tagging via default_tags — baseline tags once, in the provider.
- Deploy pipeline — apply on merge, gated by a prod environment reviewer.
- Observability & the OTel collector — the collector as the cloud-portability seam; Kafka only above a stated bar.
- Kubernetes version & upgrade policy — pin the oldest minor still in standard support; control plane → add-ons → nodes.
- State bootstrap outside the stacks — a once-per-account lifecycle no
run --allcan reach. - Per-unit AWS provider generation — don't hand an AWS provider to units that don't use AWS.
Monitoring is shown mid-migration on purpose: the incumbent datadog-monitors and the portable OpenTelemetry collector layer run side by side, the export seam letting them coexist rather than forcing a big-bang cutover.
The retirement criteria for the Datadog unit — and what stays vendor-native on purpose — are in ADR-0009.
How telemetry actually flows — the three signal pipelines, the export seam, the recording rules, and the multi-window SLO burn-rate alerting — is written up in OBSERVABILITY.md, which opens with a diagram of the module's scope: where the application boundary falls, which exporters are swappable, and which route is optional.
What this is, stated plainly, because several things it is not look like omissions otherwise.
It is an application-platform reference. VPC, EKS, data stores, edge, observability and the pipeline that plans and applies them, expressed as reusable modules and per-account stacks.
It is not a landing zone. There is no AWS Organizations setup, no Control Tower, no SCPs, no account vending, no centralised logging or GuardDuty/Security Hub baseline. It assumes the accounts already exist and that you own credentials for them. If you are looking for the layer that creates accounts and governs them, that is a different repository and a much larger one — graded as a landing zone, this will always look incomplete, because it is not attempting that job.
It is validated, not apply-tested. Every push runs fmt, offline validate across all stacks,
module unit tests (tofu test) and policy unit tests (conftest verify). Nothing in CI has ever
applied this to a live AWS account: there is no account behind it. The plan and apply workflows are
real and wired, and they are inert until you supply AWS_PLAN_ROLE_ARN / AWS_APPLY_ROLE_ARN. Treat
the resource configurations as reviewed-and-validated, not as battle-tested.
The deploy role is deliberately under-powered. catalog/units/iam-github-oidc attaches
ReadOnlyAccess, which is enough to plan and not enough to apply. Scoping a real per-account
plan/apply role — least-privilege write access to exactly the resources these stacks manage, brokered
per environment — is a genuine piece of work this repository documents rather than performs. Widening
that policy is a decision you should make explicitly, with your own resource scope, rather than
inherit from an example.
Scope is a choice; these omissions are intentional, not unfinished:
- No multi-region. One region per account. Aurora Global Database is available behind a flag, but active-active multi-region is a large operational commitment this reference doesn't pretend to.
- No service mesh. Namespace RBAC and security groups, not Istio/Linkerd. A mesh earns its complexity at a scale a reference repo can't honestly demonstrate.
- One Kubernetes compute model. Managed node groups only — no self-managed ASGs, no Karpenter, no Fargate mix. When each of those is worth it is in ADR-0006.
- One CDN, and monitoring shown mid-migration. CloudFront, not a multi-vendor edge; and a single incumbent monitoring vendor (Datadog) running alongside the portable OpenTelemetry layer, not an abstraction over observability backends (see ADR-0009).
Destroy in reverse dependency order, and never destroy management before the workload accounts
(it holds their state):
for stack in prod/us-west-2/prod staging/us-west-2/staging; do
cd live/$stack && terragrunt run --all destroy && cd -
done
# Aurora has deletion_protection and skip_final_snapshot=false: disable protection and take/accept
# the final snapshot first, or the destroy will refuse. Then, last:
cd live/management/us-west-2/global && terragrunt run --all destroyThe state buckets survive all of that on purpose: the bootstrap units are not part of any stack, so
run --all destroy cannot reach them. Each bucket has versioning and a KMS key — empty and delete
it by hand, per account, once you are sure nothing still needs the state inside it.
Four workflows, all of which skip cleanly (with a message) when their AWS role variables are unset:
- validate (PR) —
tofu fmt,terragrunt hcl fmt, tflint, trivy, terraform-docs drift, gitleaks, offlinerun --all validate, andmake test(module + policy tests). - plan (PR) — detects changed stacks, assumes
AWS_PLAN_ROLE_ARNvia OIDC, plans each, checks the plan against the conftest policies, and comments the plan + Infracost diff. - apply (merge to
main) — applies management → staging → prod, each behind a GitHub Environment. You must create these Environments and put required reviewers onprod. It assumes each account is already bootstrapped; it never creates a state bucket itself. - drift (weekly) — plans every stack and opens an issue if one has drifted.
Only pushes that touch infrastructure enter the environments, so only those ask the prod reviewer to approve — a docs-only push records a no-op deployment for each environment and prompts nobody. That prompt is deliberate; see ADR-0008.
To adopt: bootstrap each account's state backend (make bootstrap ACCOUNT=<account> — once, by
hand, see ADR-0011), run the
iam-github-oidc module, set the two role ARNs as repository Variables, and create the three
Environments. The exact IAM trust policy is documented at the top of
plan.yml. No long-lived AWS keys are used anywhere.
- capsize — a read-only Kubernetes CLI that scores cost waste and blast radius together, for the kind of EKS clusters this architecture provisions.