Detect terraform drift in deploy / integration envs

[whi-tw]

What problem does this pull request solve?

Trello card: https://trello.com/c/QV2ruMG8/777-automate-checks-for-whether-deploy-and-integration-are-up-to-date-with-main

This PR adds drift detection for deploy and integration deployments on top of the SSM parameter recording from the previous PR.

Currently, this just runs a codebuild project weekly. need to discuss alerting / general logic.

Things to consider when reviewing

Prerequisites

Deploy all integration infrastructure to establish baseline:

gds aws forms-integration-admin -- make integration_apply_all

This will:

• Apply all integration roots in the correct order
• Record deployment metadata (SHA + branch) to SSM for each root
• Create the drift-detection CodeBuild project with hcl2json installed
• Set up EventBridge schedule and IAM roles with SSM read permissions

Test 1: Verify No Drift (Baseline)

With all integration roots freshly applied, trigger drift detection:

gds aws forms-integration-admin -- aws codebuild start-build --project-name drift-check-integration
gds aws forms-integration-admin -- aws logs tail /aws/codebuild/drift-check-integration --follow

Expected: All roots show ✅ UP TO DATE: No changes detected

Test 2: Create Drift and Verify Detection

Push a commit with changes but don't apply it:

# Make a dummy change to trigger drift
echo "# Test drift detection" >> infra/deployments/integration/account/README.md
git add infra/deployments/integration/account/README.md
git commit -m "Test: dummy change for drift detection"
git push

Trigger drift detection:

gds aws forms-integration-admin -- aws codebuild start-build --project-name drift-check-integration
gds aws forms-integration-admin -- aws logs tail /aws/codebuild/drift-check-integration --follow

Expected output:

• integration/account shows: ⚠️ DRIFT DETECTED
• Changed files listed: infra/deployments/integration/account/README.md
• Other integration roots (like integration/review) show: ✅ UP TO DATE
• This confirms drift detection works - only the changed root shows drift

Test 3: Verify Module Dependency Graph

Test that drift is correctly isolated to specific module dependencies:

# Change a module used by only integration/review
echo "# Test" >> infra/modules/cloudfront_waf_protection/README.md
git commit -am "Test: change to cloudfront module"
git push

gds aws forms-integration-admin -- aws codebuild start-build --project-name drift-check-integration
gds aws forms-integration-admin -- aws logs tail /aws/codebuild/drift-check-integration --follow

Expected: Only integration/review should show drift (not integration/account), verifying the dependency tree is built correctly per-root.

Test 4: Verify hcl2json Installation

Check that hcl2json is successfully installed in the install phase:

# Look for hcl2json installation logs
gds aws forms-integration-admin -- aws logs filter-pattern /aws/codebuild/drift-check-integration \
  --filter-pattern "hcl2json" \
  --start-time=$(date -u -v-10M +%s)000

Expected: Should see hcl2json version downloaded and hcl2json --version output.

Clean Up

# Remove test commits
git reset --hard HEAD~2  # Remove both test commits
git push --force

Key Behaviors to Verify

• Drift detection runs without errors in integration account
• hcl2json successfully installed in CodeBuild install phase
• Module dependency tree built for each root (look for "Building module dependency tree..." in logs)
• Only roots using changed modules show drift (not all roots)
• Changed files clearly listed in output under "Changed files in dependent modules"
• SSM parameters correctly parsed for SHA extraction using jq

Reminders

If you've made changes to the deployer role (files in modules/deployer-access):

• Remember to run make <environment> forms/account apply on the relevant environments (dev, staging, user-research, and/or prod)
• Check the #govuk-forms-deployment-notifications Slack channel to ensure the apply-forms-terraform-<environment> pipelines have run successfully

[whi-tw]

NB: the TODO in infra/deployments/integration/account/drift-detection.tf should be removed to revert back to main before merging.

[Copilot]

Pull request overview

This PR implements automated drift detection for Terraform deployments in the deploy and integration environments. The solution uses CodeBuild to periodically check if deployed infrastructure matches the current state of the main branch by comparing git SHAs stored in SSM parameters.

Key Changes:

• Added a new drift-detection Terraform module that creates CodeBuild projects, EventBridge schedules, and IAM roles for automated drift checking
• Implemented a bash script that clones the repository, builds module dependency trees, and compares deployed SHAs against current code
• Configured drift detection in both integration/account and deploy/tools roots with weekly Monday 9 AM UTC schedules

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
infra/modules/drift-detection/variables.tf	Defines input variables for deployment name, schedule, git repository URL, and branch
infra/modules/drift-detection/outputs.tf	Exports CodeBuild project details, EventBridge rule, and CloudWatch log group names
infra/modules/drift-detection/main.tf	Creates CodeBuild project with buildspec that installs hcl2json and runs drift detection
infra/modules/drift-detection/iam.tf	Sets up IAM roles and policies for CodeBuild and EventBridge with appropriate SSM read permissions
infra/modules/drift-detection/eventbridge.tf	Configures EventBridge scheduled rule to trigger CodeBuild drift checks
infra/modules/drift-detection/drift-detection.sh	Bash script that performs drift detection by comparing git SHAs and analyzing module dependencies
infra/deployments/integration/inputs.tf	Adds drift_detection_schedule variable for integration environment
infra/deployments/integration/account/drift-detection.tf	Instantiates drift detection module for integration (currently pointing to test branch)
infra/deployments/deploy/tools/drift-detection.tf	Instantiates drift detection module for deploy environment
infra/deployments/deploy/inputs.tf	Adds drift_detection_schedule variable for deploy environment

Comments suppressed due to low confidence (1)

infra/modules/drift-detection/main.tf:27

• The script downloads the latest version of hcl2json from GitHub without verifying checksums or signatures. This could be a security risk if the GitHub API is compromised or returns unexpected data. Consider pinning to a specific version and verifying checksums, or using a more secure installation method.

          "HCL2JSON_VERSION=\"$(curl -s https://api.github.com/repos/tmccombs/hcl2json/releases/latest | jq -r '.tag_name')\"",
          "curl https://github.com/tmccombs/hcl2json/releases/download/$${HCL2JSON_VERSION}/hcl2json_linux_amd64 -L -o hcl2json",

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[sarahseewhy]

This is really cool and will save us the toil task of doing manual checks. Thank you 👏

Let's chat as a team about alert etc (which you mention in the PR description).

And thanks for the note about changing the branch name back to main before merging. Just so we don't forget to change the branch name, I'm going to mark this as "Request changes" although there's nothing else I think needs changing.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sarahseewhy]

I think this looks fine. Copilot has some granular suggestions (I do agree with the typo one about list vs last, can't believe I missed that). How would you like to proceed?

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.