[POC] Manual plan approval for account customizations pipelines

[v-rosa]

[POC] Manual plan approval for account customizations pipelines

This is a working reference implementation, shared for feedback.

Summary

I run AFT in production and needed what #153 #481 #541 have been asking for for years, a human review between terraform plan and terraform apply in the per-account customizations pipelines. This PoC was built on a fork of AFT 1.21.1 using only native CodePipeline features, and have been testing it running it across our small account fleet.

We know this project does not currently accept community contributions, so this is deliberately not a PR. We're sharing the design as a concrete starting point, hoping it helps the maintainers shape and prioritize an official feature

This also gives the community something specific to give feedback on.

EDIT: After publishing this, I've found Igor's proposal: #542 linking for visibility. As I missed it for 1y.

What the reference implementation does

• Each customizations stage is split in two: Plan (runs terraform plan -out=tfplan,
  publishes the plan as a pipeline artifact) and Approve + Apply (native Manual
  Approval action, then terraform apply tfplan on the reviewed artifact).
• No changes requires no approval: a CodePipeline V2 before_entry rule skips the
  Approve+Apply stage entirely when the plan is empty.
• The approval dialog shows the account name/id and links to the plan's build logs;
  notifications go to an SNS topic. The timeout is configurable (default 5 minutes).
• Backward compatible during rollout: the shared buildspecs branch on a TF_COMMAND
  env var injected by the pipeline (plan / apply / unset = upstream behavior verbatim), so
  pipelines that haven't been re-provisioned with the new stage layout keep working
  unchanged.

How to use it

1. Update your root TF invokation.

module "aft" {
  source                                        = "github.com/v-rosa/terraform-aws-control_tower_account_factory?ref=approval-stage-poc"
  ....
  # these 2 properties are really needed in order to allow the pipeline customization SFN to work as expected.
  aft_framework_repo_url                        = "https://github.com/v-rosa/terraform-aws-control_tower_account_factory.git"
  aft_framework_repo_git_ref                    = "approval-stage-poc"
  aft_customizations_approval_timeout_in_minutes = 6
}

2. For existing pipelines, select a sandbox account, and update the customization pipeline by running the SFN aft-invoke-customizations with the payload:

{
  "include": [
    {
      "type": "accounts",
      "target_value": [
        "<account-id>"
      ]
    }
  ]
}

3. Wait until the execution finished. After this your customization pipeline will automatically trigger and look like:

The approval modal:

4. And if you want you can play with the approval notification topic (aft_approvals_sns_topic_arn) and publish your cool approvals.

Deliberate opinions (where feedback would be most valuable)

This is intentionally opinionated; an official feature might choose differently:

1. Short (5-minute) approval expiry, but configurable.
2. Skip-on-no-changes is automatic and non-overridable. the console also renders the engaged SKIP rule in red ("expected true but got false"), which is cosmetic but confusing at first sight.
3. Reviewers read the plan in the CodeBuild logs (deep link in the approval dialog)

Known gaps

• I've left approval-group management out of scope. (IDK if this is really a gap compared with the current status).
• Not tested with Terraform Cloud/Enterprise workspaces. I run the OSS distribution;
• I've tested only with single pipeline run, no bulk rollout of changes for many accounts. This is not our use case, some edge cases might appear ofc.

Availability & maintenance

The full diff against the 1.21.1 release is available here: v-rosa#1. It is shared as-is: I can't promise to keep the fork up-to-date with future AFT releases, so please treat it as a design artifact and conversation starter.

Have a safe deployment!

[hacker65536]

Strong support, from someone who has wanted this for years

I've been running AFT in production for a few years too (OSS distribution, a few hundred
managed accounts). plan → approve → apply is something I've wanted for as long as you
have. Thank you for putting a concrete implementation on the table to move the discussion
forward. I strongly support this becoming an official feature.

We have been getting by with this outside the pipeline. It ended up being a different
design, so I'm sharing it in case it is useful input.

Background. We already had a Terraform CI/CD on GitHub Actions before adopting AFT:
terraform plan runs on the pull request, the plan is posted as a PR comment, reviewers
approve or ask for changes, and on merge terraform apply runs against the plan file
generated earlier. Everything stayed inside GitHub.

We adopted AFT because account creation as IaC, and the non-Control-Tower baseline
rollout, were an excellent experience. aft-account-request and
aft-global-customizations fit us very well.

We don't manage everything with AFT, though. We keep aft-account-customizations split
one directory — one Terraform root module — per account, with state per account as
well. Piling hundreds of resources into that unit is not a good way to manage them, so
application workloads stay in our other IaC/CI. What we put in aft-account-customizations
is limited to long-lifecycle networking resources, account-wide unique resources, and
anything not tied to a specific workload. Exactly the kind of change that deserves a
review gate.

So we added PR-triggered GitHub Actions workflows to aft-account-request and
aft-account-customizations that simulate the pipeline's CodeBuild buildspec and run
terraform plan:

1. Open a PR on GitHub
2. The workflow fires
3. terraform plan runs and the result is posted as a PR comment
4. A reviewer reads the plan and approves, or asks for changes
5. On merge, a <account_customizations_name>/**.tf push trigger on the CodePipeline
   starts the pipeline and the apply happens

But this is a substitute, not an equivalent. And what it is missing is exactly why I
support your design.

The plan our reviewer approves is not the plan that gets applied. After the merge the
pipeline re-runs terraform init and terraform apply --auto-approve from scratch, so
anything that changed in between is applied unreviewed. Your Plan stage publishing
tfplan as an artifact, and Approve + Apply applying that artifact, restores exactly
the guarantee we had before adopting AFT and lost. To me that is the strongest argument for
doing the approval inside the pipeline.

One more thing: our approach has a hidden dependency. The gate only works because
merging starts the pipeline. But AFT's template declares no trigger and sets
DetectChanges / PollForSourceChanges to false on both source actions. So we set the
trigger on each pipeline individually, after AFT has finished creating it.

Being per-pipeline work, it gets missed. And when it does, the failure mode is
silence: the plan is reviewed, the PR merges, CI is green, and nothing applies. We
ended up writing a read-only checker whose only job is to find the pipelines we forgot.

Putting it into aft-create-pipeline would remove the omissions, but that means forking
AFT itself and carrying the change forward on every AFT upgrade, which we haven't been
willing to do. Needing a detector for a gap we introduced ourselves is not where I wanted
to end up. Approval inside the pipeline avoids this entire class of problem.

Scale notes (input for the "not tested with bulk rollout" gap)

Not requests — just field data for the gap you flagged yourself.

We once had a trigger on aft-global-customizations as well. Every update there
started tens to hundreds of CodePipelines at once, and it broke in two places:

1. The Source stage. Once more than 100 source fetches happen concurrently, you hit
   GitHub's secondary rate limit (no more than 100 concurrent requests)
   and the source fetch itself fails.
2. The CodeBuild concurrent-build quota. Concurrently running builds for Linux/Medium environment — that is the environment AFT's customizations projects run in. The
   starting value seems to vary by account, but either way it has to be raised to match the
   size of the fleet.

We removed the trigger from the global side in the end.

The order matters: Source fails first. The CodeBuild quota can be raised to keep up,
but GitHub's limit of 100 cannot be. And since pipelines past the hundredth never reach
the build, your Plan stage — being downstream of Source — would produce neither a plan nor
an approval in that situation. For a change that fans out across the whole fleet, the
fan-out hits a ceiling before the question of how to handle approvals even arises.

On top of that, with the default 5-minute approval expiry, even if every pipeline did
start, I don't think we could review them before they lapse. (What an expired approval does
to the run, I haven't checked either.)

And because global and account customizations are serial stages of the same pipeline, your
split gives two approval gates per account per run, and a global-only change still runs the
account stage. The number of approvals for one logical change grows with the number of
accounts.

None of this is a flaw in the PoC. I'm writing it down as the data point of what a large
fleet runs into first.

That serial execution is the part that has hurt us longest

And it interacts with your design. Because the two are serial, every time we want to add
something to aft-global-customizations we have to check whether it causes drift in any
account's aft-account-customizations. Today the only way to preview that is a local
scratch workspace per account with a terraform plan in each — which at a few hundred
accounts is genuinely painful.

Your Plan stage would produce exactly that per-account preview natively. For us that
is a significant improvement on its own, before the approval gate even comes into it.

Again, I support this becoming an official feature.

Aside

Off-topic, but running AFT at this scale pushed me into writing a few tools of my own
(aft-ops) — checking the state of a few hundred
customizations pipelines at once, pulling the terraform error out of a failed build,
triggering Release change, and the trigger checker mentioned above. Mostly read-only
operations, MIT licensed. Still early, but I use it daily. Help yourself if any of it looks
useful.

[v-rosa]

Thanks for the detailed feedback @hacker65536 I really hope your feedback can be leveraged to keep improving this project and remove the many friction points the community reported over the years.