Skip to content

Terraform Upgrade + terraform upgrade process upgrade#1714

Merged
whi-tw merged 8 commits into
mainfrom
whi-tw/terraform-upgrade
Sep 11, 2025
Merged

Terraform Upgrade + terraform upgrade process upgrade#1714
whi-tw merged 8 commits into
mainfrom
whi-tw/terraform-upgrade

Conversation

@whi-tw
Copy link
Copy Markdown
Contributor

@whi-tw whi-tw commented Sep 9, 2025

What problem does this pull request solve?

Trello card: https://trello.com/c/MCDTH5Ll/644-terraform-upgrade

This pull request modernizes and improves the Terraform upgrade process for the forms-deploy repository by introducing automated tooling and GitHub workflows to handle version updates and provider lock file management.

It also updates terraform and the providers.

Key improvements:

  1. Replaces manual Ruby script with tfupdate - Switches from the custom Ruby upgrade script to the industry-standard tfupdate tool, which provides more reliable and comprehensive Terraform version management.

  2. Automated provider lock file management - Introduces a GitHub workflow that automatically handles .terraform.lock.hcl file updates when infra/shared/versions.tf is modified, with different behavior for Dependabot vs. human-created PRs.

  3. Dependabot integration - Adds Dependabot configuration for Terraform providers and ensures lock files are automatically updated in Dependabot PRs.

  4. HCL format standardization - Converts from versions.tf.json to versions.tf to ensure compatibility with tfupdate and maintain consistency with Terraform best practices.

Things to consider when reviewing

  • Workflow permissions: The new GitHub workflow requires the DEPENDABOT_PAT secret to be configured for Dependabot PRs to work correctly
  • Breaking change: The switch from fuzzy version matching to exact versions is intentional and improves version predictability
  • Error handling: Test scenarios where lock files are out of sync to ensure the workflow properly catches and reports issues

Key files to review:

  • infra/scripts/upgrade_tf_version.sh - New upgrade script using tfupdate
  • .github/workflows/update-provider-locks.yml - Automated lock file management
  • .github/dependabot.yml - Dependabot configuration for Terraform
  • infra/shared/versions.tf - Converted from JSON to HCL format

Reminders

  • Remember to add the DEPENDABOT_PAT repository secret before merging
  • The workflow will start monitoring infra/shared/versions.tf changes after merge
  • Future Terraform provider updates via Dependabot will automatically include lock file

@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch 16 times, most recently from b81a368 to 6231b85 Compare September 9, 2025 12:10
@whi-tw whi-tw requested a review from Copilot September 10, 2025 08:25

This comment was marked as outdated.

@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch from 6231b85 to 7177003 Compare September 10, 2025 09:09
@whi-tw whi-tw requested a review from Copilot September 10, 2025 09:16

This comment was marked as outdated.

@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch 2 times, most recently from ee50873 to 7177003 Compare September 10, 2025 09:20
Comment thread .github/workflows/update-provider-locks.yml Outdated
@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch from 1b3e447 to 7177003 Compare September 10, 2025 10:09
@whi-tw
Copy link
Copy Markdown
Contributor Author

whi-tw commented Sep 10, 2025

TODO: check new terraform version doesn't ruin everything
TODO: check that provider bumps are sensible (terraform apply to dev = no changes)

@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch 4 times, most recently from 30f953d to 60a3177 Compare September 10, 2025 13:34
@whi-tw whi-tw requested a review from Copilot September 10, 2025 14:39
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR modernizes the Terraform upgrade process by replacing a custom Ruby script with the industry-standard tfupdate tool and introduces automated workflows for managing provider lock files. The changes upgrade Terraform from version 1.11.4 to 1.13.1 and convert all version constraints from JSON to HCL format while switching from fuzzy version matching to exact version pinning.

Key changes:

  • Replaces the Ruby upgrade script with a new Bash script using tfupdate for more reliable version management
  • Adds automated GitHub workflow to handle .terraform.lock.hcl file updates when infra/shared/versions.tf changes
  • Introduces Dependabot configuration for Terraform providers with automatic lock file updates

Reviewed Changes

Copilot reviewed 55 out of 81 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
infra/shared/versions.tf.json → infra/shared/versions.tf Converts from JSON to HCL format and upgrades all provider versions
infra/scripts/upgrade_tf_version.rb Removes the legacy Ruby upgrade script
infra/scripts/upgrade_tf_version.sh New Bash script using tfupdate for version management
.github/workflows/update-provider-locks.yml Automated workflow for provider lock file management
.github/dependabot.yml Adds Terraform provider dependency management
Multiple versions.tf symlinks Updates all symlink references from .json to .tf files
Files not reviewed (26)
  • infra/deployments/deploy/account/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/coordination/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/e2e-tests-image-builder/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/ecr/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/engineer-access/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/image-builders/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/tools/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/account/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/account/init-state-bucket/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/auth0/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/dns/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/environment/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-admin/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-api/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-product-page/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-runner/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/health/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/health/monitoring/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/pipelines/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/rds/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/redis/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/ses/.terraform.lock.hcl: Language not supported
  • infra/deployments/integration/account/.terraform.lock.hcl: Language not supported
  • infra/deployments/integration/review/.terraform.lock.hcl: Language not supported
  • infra/deployments/integration/review/vpc/.terraform.lock.hcl: Language not supported
  • infra/modules/forms-api/.terraform.lock.hcl: Language not supported

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread infra/scripts/upgrade_tf_version.sh
Comment thread .github/workflows/update-provider-locks.yml
Comment thread .github/workflows/update-provider-locks.yml
Comment thread .github/workflows/update-provider-locks.yml
Copy link
Copy Markdown
Contributor

@cadmiumcat cadmiumcat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great 👏

I'm happy to approve once you've gone through the TODOs and your own self deprecating comments

Comment thread .github/workflows/update-provider-locks.yml
Comment thread .github/workflows/update-provider-locks.yml
@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch 2 times, most recently from 9dbd8dc to 417bc2f Compare September 10, 2025 15:56
@whi-tw
Copy link
Copy Markdown
Contributor Author

whi-tw commented Sep 10, 2025

@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch from 0946f5e to 17b006b Compare September 10, 2025 16:12
cadmiumcat
cadmiumcat previously approved these changes Sep 11, 2025
Copy link
Copy Markdown
Contributor

@cadmiumcat cadmiumcat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing

@whi-tw
Copy link
Copy Markdown
Contributor Author

whi-tw commented Sep 11, 2025

@cadmiumcat there was a merge conflict after the redis PR went out - can you please re-review. specifically infra/deployments/forms/redis/.terraform.lock.hcl

tfupdate only works on hcl files, so we need to switch back to hcl.
Rather than the manual ruby script, we can use tfupdate to do the heavy
lifting of finding and updating the versions of terraform and the
providers.

Tfupdate also has the ability to update the lock files for us using an
in-memory cache of the plugins, removing the need for us to `init` and
download the plugins multiple times.

This does switch us away from fuzzy version matching to exact versions,
but that is a good thing as it means we know exactly what version we are
using.

We no longer read the terraform version from terraform_version.tf.json,
instead we read it from .terraform-version. This ensures that the
version used in codebuild and locally are the same, as tfenv reads
this file.
- if dependabot opened the PR, commit any updated lock files
- if a human opened the PR, fail if lock files are not updated
Some people prefer `mise` - this lets it fail over to that when run
locally.

Also, bump a gemfile lock for a new MacOS version.
We use the null provider in a few places, so it makes sense for us to
actually pin and lock the version we use.
@whi-tw whi-tw force-pushed the whi-tw/terraform-upgrade branch from 79b4ce7 to 00ee757 Compare September 11, 2025 08:57
@github-actions
Copy link
Copy Markdown

Warning

You are changing the Terraform version.

Before you merge this PR, you must apply the forms/pipelines
Terraform root from this branch in each of the forms environments.

make ENV forms/pipelines apply

@whi-tw whi-tw disabled auto-merge September 11, 2025 09:02
@whi-tw whi-tw requested a review from Copilot September 11, 2025 09:04
@whi-tw whi-tw enabled auto-merge September 11, 2025 09:05
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 57 out of 84 changed files in this pull request and generated 1 comment.

Files not reviewed (26)
  • infra/deployments/deploy/account/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/coordination/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/e2e-tests-image-builder/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/ecr/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/engineer-access/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/image-builders/.terraform.lock.hcl: Language not supported
  • infra/deployments/deploy/tools/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/account/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/account/init-state-bucket/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/auth0/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/dns/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/environment/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-admin/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-api/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-product-page/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/forms-runner/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/health/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/health/monitoring/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/pipelines/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/rds/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/redis/.terraform.lock.hcl: Language not supported
  • infra/deployments/forms/ses/.terraform.lock.hcl: Language not supported
  • infra/deployments/integration/account/.terraform.lock.hcl: Language not supported
  • infra/deployments/integration/review/.terraform.lock.hcl: Language not supported
  • infra/deployments/integration/review/vpc/.terraform.lock.hcl: Language not supported
  • infra/modules/forms-api/.terraform.lock.hcl: Language not supported

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread .github/workflows/update-provider-locks.yml
@whi-tw whi-tw merged commit 5eb4f2e into main Sep 11, 2025
5 checks passed
@whi-tw whi-tw deleted the whi-tw/terraform-upgrade branch September 11, 2025 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants