-
Notifications
You must be signed in to change notification settings - Fork 5
Improves CI test resource cleanup to prevent orphaned AWS resources from failed/cancelled test runs. #973
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Improves CI test resource cleanup to prevent orphaned AWS resources from failed/cancelled test runs. #973
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
18cf8ef
Add three-layer defense against orphaned test resources
b222487
Fix shellcheck errors in cleanup workflow
95c608c
Fix shellcheck errors in cleanup-test-resources script
b5bb6eb
TICKET-942 Remove scheduled orphaned detection workflow
05f934c
Address PR review feedback
9aad608
Add scheduled workflow to scan for orphaned test resources
c57578a
Add retry logic to teardown functions
d745061
Address PR review comments
0438773
Merge branch 'main' into fix/improve-test-cleanup-process
sean-navapbc c553d1c
Add runCommandWithRetry helper function to template_infra_test.go
68401b4
Temporarily trigger scan workflow on feature branch for testing
203df2f
Rename workflows to template-only- prefix for consistency
d9d84d5
Remove unnecessary chmod commands from workflows
322efb1
Rename cleanup script to template-only- prefix
240f872
Fix shellcheck warning SC2129 - group redirects
44c06b8
Merge branch 'main' into fix/improve-test-cleanup-process
sean-navapbc edc0de4
Update .github/workflows/template-only-cleanup-orphaned-infra-test-re…
sean-navapbc c65abc7
Rename script and update workflow display names
41d64d7
Continue cleanup even when state bucket is missing
882a63c
Suppress broken pipe errors in scan workflow
383f93c
Fix broken pipe errors by redirecting entire pipeline stderr
9a11699
Add ECS task definition cleanup to cleanup script
4529637
Fix cleanup script to handle all orphaned resource types
5a83472
Remove unnecessary private zone filter from Route53 cleanup
964db22
Improve task definition cleanup logic
f3af99c
Fix broken pipe errors in scan workflow
bee83f4
Fix all broken pipe errors in scan workflow
a5d77ae
Fix shellcheck lint warning in cleanup script
eda7577
Clean up task definitions in destroy-app-service script
29ab11d
Remove notify block from scan workflow
743d112
Fix broken pipe errors by disabling pipefail around head commands
1f56642
Merge branch 'main' into fix/improve-test-cleanup-process
sean-navapbc daa0f2e
Fix broken pipe errors and add inactive task definition cleanup
b6dd99c
Fix broken pipe by trapping SIGPIPE signal
aea90f5
Suppress broken pipe error messages with stderr redirect
2dc6051
Fix broken pipe error and show full scan output
765325e
Fix broken pipe errors by using temp file instead of pipes
c318ace
Fix inactive task definition lookup to check tags instead of family p…
005881a
Remove temporary push trigger for testing
b46f459
Remove unused runCommandWithRetry helper function
355e27d
Remove unused AGE_HOURS variable from cleanup script
b9cbff5
Improve task definition cleanup comments and use pushd/popd
5192884
Get project config from terraform instead of pwd/aws config
4c768fe
Remove unused --age-hours argument from scan workflow
e451aa4
Remove unused --age-hours parameter from cleanup workflow
9acd97c
Fix GitHub issue link for task definition deletion
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
45 changes: 45 additions & 0 deletions
45
.github/workflows/template-only-cleanup-orphaned-infra-test-resources.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| name: Template CI Infra Checks Cleanup | ||
|
|
||
| on: | ||
| # Manual trigger only - cleanup is intentionally not automatic | ||
| # to avoid masking underlying test issues that should be fixed | ||
| workflow_dispatch: | ||
| inputs: | ||
| project_name: | ||
| description: 'Specific project to clean up (e.g., plt-tst-act-12345). Leave empty to find all projects.' | ||
| required: false | ||
| type: string | ||
| dry_run: | ||
| description: 'Dry run - list resources without deleting them' | ||
| required: false | ||
| default: true | ||
| type: boolean | ||
|
|
||
| jobs: | ||
| cleanup: | ||
| name: Cleanup Test Resources | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| aws-region: us-east-1 | ||
| aws-access-key-id: ${{ secrets.TESTER_AWS_ACCESS_KEY_ID }} | ||
| aws-secret-access-key: ${{ secrets.TESTER_AWS_SECRET_ACCESS_KEY }} | ||
|
|
||
| - name: Run cleanup script | ||
| run: | | ||
| args=() | ||
| if [ "${{ inputs.dry_run }}" = "true" ]; then | ||
| args+=(--dry-run) | ||
| fi | ||
|
|
||
| if [ -n "${{ inputs.project_name }}" ]; then | ||
| args+=("${{ inputs.project_name }}") | ||
| fi | ||
|
|
||
| ./template-only-bin/cleanup-test-resources "${args[@]}" |
59 changes: 59 additions & 0 deletions
59
.github/workflows/template-only-scan-orphaned-infra-test-resources.yml
sean-navapbc marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| name: Template Scan Orphaned Infra Test Resources | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| # Run every day at 08:00 UTC (4:00am ET, 1:00am PT) | ||
| - cron: "0 8 * * *" | ||
|
|
||
| jobs: | ||
| scan: | ||
| name: Scan for orphaned test resources | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Configure AWS credentials | ||
| uses: aws-actions/configure-aws-credentials@v4 | ||
| with: | ||
| aws-region: us-east-1 | ||
| aws-access-key-id: ${{ secrets.TESTER_AWS_ACCESS_KEY_ID }} | ||
| aws-secret-access-key: ${{ secrets.TESTER_AWS_SECRET_ACCESS_KEY }} | ||
|
|
||
| - name: Scan for orphaned resources | ||
| id: scan | ||
| shell: bash | ||
| run: | | ||
| # Run in dry-run mode to see what would be deleted | ||
| # Write to temp file to avoid broken pipe errors from piping large output | ||
| tmpfile=$(mktemp) | ||
| ./template-only-bin/cleanup-test-resources --dry-run > "$tmpfile" 2>&1 || true | ||
|
|
||
| # Print full output for debugging | ||
| cat "$tmpfile" | ||
|
|
||
| # Check if any resources were found (look for "Found X resources" in output) | ||
| if grep -q "Found [1-9][0-9]* resources" "$tmpfile"; then | ||
| # Extract resource count and project names for notification | ||
| resource_info=$(grep -E "(Found [0-9]+ resources|Cleaning up project:|Would delete)" "$tmpfile" | head -30 || true) | ||
|
|
||
| { | ||
| echo "found=true" | ||
| echo "resource_info<<EOF" | ||
| echo "$resource_info" | ||
| echo "EOF" | ||
| } >> "$GITHUB_OUTPUT" | ||
| rm -f "$tmpfile" | ||
| exit 1 | ||
| else | ||
| echo "found=false" >> "$GITHUB_OUTPUT" | ||
| echo "" | ||
| echo "=== Summary ===" | ||
| echo "No orphaned resources found that need cleanup." | ||
| # Show how many projects were checked | ||
| project_count=$(grep -c "^=== Cleaning up project:" "$tmpfile" || echo "0") | ||
| echo "Checked ${project_count} projects." | ||
| rm -f "$tmpfile" | ||
| fi |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.