feat(clean): reap leaked AMP integ-test workspaces - #2233
Open
olowosulu wants to merge 3 commits into
Open
Conversation
Integration tests create one AMP (Amazon Managed Service for Prometheus) workspace per /amp run (alias cwagent-integ-test-<id>) and destroy it via terraform. Workspaces left behind by cancelled or hard-killed jobs, whose terraform state dies with the ephemeral runner, accumulate to the per-region 75-workspace quota. Once at the cap every new run fails at CreateWorkspace with ServiceQuotaExceededException before the Go test binary runs (observed on the al2:selinux_amp_test job). The existing if:cancelled()||failure() terraform destroy step cannot reclaim them because their state is gone. Add tool/clean/clean_amp/clean_amp.go (modeled on clean_eks): list AMP workspaces and delete ACTIVE ones whose alias has the cwagent-integ-test- prefix and were created longer ago than KeepDurationOneDay. Supports -dry-run. Add a clean-amp-workspaces job to clean-aws-resources.yml using the same OIDC role and daily schedule as the other cleaners. Adding service/amp pulled an aligned aws-sdk-go-v2 core bump in the isolated tool/clean module. Verified: go build and go vet are clean for clean_amp; a dry-run against the test account matches an independent audit (targets only cwagent-integ-test-* workspaces and spares non-test workspaces). Note: the clean OIDC role (vars.TERRAFORM_AWS_ASSUME_ROLE) needs amp:ListWorkspaces and amp:DeleteWorkspace permissions for the scheduled job.
musa-asad
reviewed
Aug 6, 2026
musa-asad
previously approved these changes
Aug 6, 2026
- Collect delete errors with errors.Join and return them - Negate KeepDurationOneDay in log message for readable display - Sweep CREATION_FAILED workspaces (dead but hold quota) - Skip empty prefix entries in aliasMatchesWorkspacesToClean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The integration tests create one AMP (Amazon Managed Service for Prometheus) workspace per
/amprun (terraform/ec2/linux/main.tf,module "amp",workspace_alias = "cwagent-integ-test-<testing_id>") and destroy it withterraform destroyat the end of the run.Workspaces left behind by cancelled/superseded or hard-killed jobs, whose Terraform state dies with the ephemeral runner, are never reclaimed: the
if: cancelled() || failure()cleanupterraform destroycannot delete a workspace whose state is gone. These orphans accumulate to the per-region 75-workspace service quota. Once the region is at the cap, every new run fails atCreateWorkspace:terraform applythen exits 1 before the Go test binary runs, soal2:selinux_amp_testfails with a red check unrelated to the code under review. An audit of the test region found dozens of orphanedcwagent-integ-test-*workspaces, the oldest over a year old, none created in the last day (leaked, not in-flight).Fix
Every other leaked resource type already has a cleaner under
tool/clean/clean_*run daily byclean-aws-resources.yml; AMP was the gap. This adds:tool/clean/clean_amp/clean_amp.go(modeled onclean_eks): lists AMP workspaces and deletes ACTIVE ones whose alias has thecwagent-integ-test-prefix and were created longer ago thanclean.KeepDurationOneDay. Supports-dry-run.clean-amp-workspacesjob inclean-aws-resources.yml, using the same OIDC role (vars.TERRAFORM_AWS_ASSUME_ROLE), region, and daily schedule as the sibling cleaners.Verification
go buildandgo vetare clean forclean_amp(the onlygo vet ./...finding is pre-existingunreachable codeinclean_security_group, which this PR does not touch).-dry-runagainst the test account matches an independent audit: it targets onlycwagent-integ-test-*workspaces older than a day and spares non-test workspaces.service/amppulled an alignedaws-sdk-go-v2core bump within the isolatedtool/cleanmodule; all sibling cleaners still vet.Prerequisite
The clean OIDC role (
vars.TERRAFORM_AWS_ASSUME_ROLE) needsamp:ListWorkspacesandamp:DeleteWorkspacefor the scheduled job to run.