-
Notifications
You must be signed in to change notification settings - Fork 1
Add PR environments #51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sean-navapbc
wants to merge
14
commits into
main
Choose a base branch
from
7-add-pr-environments
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 8 commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
cfc8d2c
Add PR environment workflows ported from AWS template
0741f63
Merge branch 'main' into 7-add-pr-environments
sean-navapbc 8c57aca
Address PR #51 review findings
f28e97e
Improve PR environment scripts
fd736ae
Merge branch 'main' into 7-add-pr-environments
sean-navapbc ba6611d
Add resilience improvements for PR environments
bf681a2
Fix workspace existence check for currently selected workspace
3e17118
Merge branch 'main' into 7-add-pr-environments
sean-navapbc 4d60716
Merge branch 'main' into 7-add-pr-environments
sean-navapbc 7323fec
Wait for PR environment health via Azure CLI in a shared script
ef0cc98
Merge branch 'main' into 7-add-pr-environments
sean-navapbc 24cc71c
Address review comments on PR environment scripts
1181fcb
Remove legacy single-app cleanup from destroy-pr-environment
c038f4b
Merge branch 'main' into 7-add-pr-environments
sean-navapbc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
33 changes: 33 additions & 0 deletions
33
.github/workflows/ci-{{app_name}}-pr-environment-checks.yml.jinja
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| name: CI {{ app_name }} PR Environment Checks | ||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| required: true | ||
| type: string | ||
| commit_hash: | ||
| required: true | ||
| type: string | ||
| {% if app_has_dev_env_setup %} | ||
| pull_request: | ||
| {% else %} | ||
| # !! Once you've set up the dev environment and are ready to enable PR | ||
| # environments, run: | ||
| # | ||
| # nava-platform infra update --answers-only --data app_has_dev_env_setup=true . | ||
| # | ||
| # to enable these lines. They are here as comments for context. | ||
| # | ||
| # pull_request: | ||
| {% endif %} | ||
|
|
||
| jobs: | ||
| update: | ||
| name: " " # GitHub UI is noisy when calling reusable workflows, so use whitespace for name to reduce noise | ||
| uses: ./.github/workflows/pr-environment-checks.yml | ||
| if: github.event_name == 'workflow_dispatch' || github.event.pull_request.state == 'open' | ||
| with: | ||
| app_name: "{{ app_name }}" | ||
| environment: "dev" | ||
| pr_number: ${{'{{'}} inputs.pr_number || github.event.number {{'}}'}} | ||
| commit_hash: ${{'{{'}} inputs.commit_hash || github.event.pull_request.head.sha {{'}}'}} |
30 changes: 30 additions & 0 deletions
30
.github/workflows/ci-{{app_name}}-pr-environment-destroy.yml.jinja
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| name: CI {{ app_name }} PR Environment Destroy | ||
| on: | ||
| workflow_dispatch: | ||
| inputs: | ||
| pr_number: | ||
| required: true | ||
| type: string | ||
| {% if app_has_dev_env_setup %} | ||
| pull_request_target: | ||
| types: [closed] | ||
| {% else %} | ||
| # !! Once you've set up the dev environment and are ready to enable PR | ||
| # environments, run: | ||
| # | ||
| # nava-platform infra update --answers-only --data app_has_dev_env_setup=true . | ||
| # | ||
| # to enable these lines. They are here as comments for context. | ||
| # | ||
| # pull_request_target: | ||
| # types: [closed] | ||
| {% endif %} | ||
|
|
||
| jobs: | ||
| destroy: | ||
| name: " " # GitHub UI is noisy when calling reusable workflows, so use whitespace for name to reduce noise | ||
| uses: ./.github/workflows/pr-environment-destroy.yml | ||
| with: | ||
| app_name: "{{ app_name }}" | ||
| environment: "dev" | ||
| pr_number: ${{'{{'}} inputs.pr_number || github.event.number {{'}}'}} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| name: PR Environment Update | ||
| run-name: Update PR Environment ${{ inputs.pr_number }} | ||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| app_name: | ||
| required: true | ||
| type: string | ||
| environment: | ||
| required: true | ||
| type: string | ||
| pr_number: | ||
| required: true | ||
| type: string | ||
| commit_hash: | ||
| required: true | ||
| type: string | ||
|
|
||
| concurrency: pr-environment-${{ inputs.app_name }}-${{ inputs.pr_number }} | ||
|
|
||
| jobs: | ||
| build-and-publish: | ||
| name: " " # GitHub UI is noisy when calling reusable workflows, so use whitespace for name to reduce noise | ||
| uses: ./.github/workflows/build-and-publish.yml | ||
| with: | ||
| app_name: ${{ inputs.app_name }} | ||
| ref: ${{ inputs.commit_hash }} | ||
|
|
||
| update: | ||
| name: Update environment | ||
| needs: [build-and-publish] | ||
| runs-on: ubuntu-latest | ||
|
|
||
| permissions: | ||
| contents: read | ||
| id-token: write | ||
| pull-requests: write # Needed to comment on PR | ||
| repository-projects: read # Workaround for GitHub CLI bug https://github.com/cli/cli/issues/6274 | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Terraform | ||
| uses: ./.github/actions/setup-terraform | ||
|
|
||
| - name: Configure Azure credentials | ||
| uses: ./.github/actions/configure-azure-credentials | ||
| with: | ||
| app_name: ${{ inputs.app_name }} | ||
| environment: ${{ inputs.environment }} | ||
|
|
||
| - name: Update environment | ||
| run: ./bin/update-pr-environment "${{ inputs.app_name }}" "${{ inputs.environment }}" "${{ inputs.pr_number }}" "${{ inputs.commit_hash }}" | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| name: PR Environment Destroy | ||
| run-name: Destroy PR Environment ${{ inputs.pr_number }} | ||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| app_name: | ||
| required: true | ||
| type: string | ||
| environment: | ||
| required: true | ||
| type: string | ||
| pr_number: | ||
| required: true | ||
| type: string | ||
|
|
||
| concurrency: pr-environment-${{ inputs.app_name }}-${{ inputs.pr_number }} | ||
|
|
||
| jobs: | ||
| destroy: | ||
| name: Destroy environment | ||
| runs-on: ubuntu-latest | ||
|
|
||
| permissions: | ||
| contents: read | ||
| id-token: write | ||
| pull-requests: write # Needed to comment on PR | ||
| repository-projects: read # Workaround for GitHub CLI bug https://github.com/cli/cli/issues/6274 | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Terraform | ||
| uses: ./.github/actions/setup-terraform | ||
|
|
||
| - name: Configure Azure credentials | ||
| uses: ./.github/actions/configure-azure-credentials | ||
| with: | ||
| app_name: ${{ inputs.app_name }} | ||
| environment: ${{ inputs.environment }} | ||
|
|
||
| - name: Destroy environment | ||
| run: ./bin/destroy-pr-environment "${{ inputs.app_name }}" "${{ inputs.environment }}" "${{ inputs.pr_number }}" | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # This workflow scans for temporary environments that were not properly cleaned up | ||
| # This can happen if the PR environment destroy workflow failed or didn't run | ||
| # or if the temporary environments created by the infra service tests were not cleaned up | ||
| name: Scan orphaned environments | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| # Run every day at 07:30 UTC (3:30am ET, 12:30am PT) after engineers are likely done with work | ||
| - cron: "30 7 * * *" | ||
|
|
||
| jobs: | ||
| get-app-names: | ||
| name: Get app names | ||
| runs-on: ubuntu-latest | ||
|
|
||
| outputs: | ||
| app_names: ${{ steps.get-app-names.outputs.app_names }} | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Get app names | ||
| id: get-app-names | ||
| run: | | ||
| source bin/util.sh | ||
| app_names="$(get_app_names)" | ||
| # turn app_names into a json list using jq | ||
| app_names="$(echo "${app_names}" | jq -R -s -c 'split("\n")[:-1]')" | ||
| echo "App names retrieved: ${app_names}" | ||
| echo "app_names=${app_names}" >> "$GITHUB_OUTPUT" | ||
| shell: bash | ||
|
|
||
| scan: | ||
| name: Scan | ||
| runs-on: ubuntu-latest | ||
| needs: get-app-names | ||
|
|
||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| app_name: ${{ fromJson(needs.get-app-names.outputs.app_names) }} | ||
| scan_script: [orphaned-pr-environments, stale-test-environments] | ||
|
|
||
| permissions: | ||
| contents: read | ||
| id-token: write | ||
| pull-requests: read | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Terraform | ||
| uses: ./.github/actions/setup-terraform | ||
|
|
||
| - name: Configure Azure credentials | ||
| uses: ./.github/actions/configure-azure-credentials | ||
| with: | ||
| app_name: ${{ matrix.app_name }} | ||
| environment: dev | ||
|
|
||
| - name: List PR workspaces | ||
| run: | | ||
| ./bin/${{ matrix.scan_script }} ${{ matrix.app_name }} | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| TF_IN_AUTOMATION: "true" | ||
|
|
||
| # TODO(#50): Add a notify job that posts to Slack on failure. Until then, | ||
| # orphaned-environment alerts only show up in the Actions UI. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| #!/usr/bin/env bash | ||
| # ----------------------------------------------------------------------------- | ||
| # Destroy the temporary environment that was created for the pull request. | ||
| # | ||
| # Positional parameters: | ||
| # app_name (required) – the name of subdirectory of /infra that holds the | ||
| # application's infrastructure code. | ||
| # environment - the name of the application environment (e.g. dev, staging, prod) | ||
| # pr_number - the pull request number in GitHub | ||
| # ----------------------------------------------------------------------------- | ||
| set -euo pipefail | ||
|
|
||
| app_name="$1" | ||
| environment="$2" | ||
| pr_number="$3" | ||
|
|
||
| # Validate app_name | ||
| if [ ! -d "infra/${app_name}/service" ]; then | ||
| echo "Error: App '${app_name}' not found in infra/" | ||
| exit 1 | ||
| fi | ||
|
|
||
| workspace="p-${pr_number}" | ||
|
|
||
| # Safety check: never allow deleting the default workspace | ||
| if [ "$workspace" = "default" ] || [ -z "$workspace" ]; then | ||
| echo "Error: Refusing to delete default or empty workspace" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "::group::Initialize Terraform with backend for environment: ${environment}" | ||
| terraform -chdir="infra/${app_name}/service" init -backend-config="${environment}.azurerm.tfbackend" | ||
| echo "::endgroup::" | ||
|
|
||
| echo "Check if Terraform workspace exists: ${workspace}" | ||
| # List workspaces and check if our workspace exists | ||
| # Note: workspace list shows current workspace with "* " prefix, others with " " prefix | ||
| if terraform -chdir="infra/${app_name}/service" workspace list | grep -qE "^[* ] +${workspace}$"; then | ||
| echo "Workspace ${workspace} exists, proceeding with destroy" | ||
| else | ||
| echo "Workspace ${workspace} does not exist - nothing to destroy" | ||
| echo "This can happen if the workspace was already cleaned up or never created" | ||
| exit 0 | ||
| fi | ||
|
|
||
| echo "Select Terraform workspace: ${workspace}" | ||
| terraform -chdir="infra/${app_name}/service" workspace select "${workspace}" | ||
|
|
||
| echo "::group::Destroy resources" | ||
| # Retry terraform destroy to handle transient Azure API errors | ||
| max_tf_attempts=3 | ||
| tf_attempt=0 | ||
| tf_success=false | ||
|
|
||
| while [ $tf_attempt -lt $max_tf_attempts ] && [ "$tf_success" = "false" ]; do | ||
| tf_attempt=$((tf_attempt + 1)) | ||
|
|
||
| if [ $tf_attempt -gt 1 ]; then | ||
| echo "Retry attempt ${tf_attempt}/${max_tf_attempts} after transient failure..." | ||
| fi | ||
|
|
||
| if terraform -chdir="infra/${app_name}/service" destroy -var="environment_name=${environment}" -input=false -auto-approve; then | ||
| tf_success=true | ||
| echo "Terraform destroy succeeded" | ||
| else | ||
| if [ $tf_attempt -lt $max_tf_attempts ]; then | ||
| echo "Terraform destroy failed, waiting 30s before retry..." | ||
| sleep 30 | ||
| fi | ||
| fi | ||
| done | ||
|
|
||
| if [ "$tf_success" = "false" ]; then | ||
| echo "Error: Terraform destroy failed after ${max_tf_attempts} attempts" | ||
| exit 1 | ||
| fi | ||
| echo "::endgroup::" | ||
|
|
||
| echo "Select default workspace" | ||
| terraform -chdir="infra/${app_name}/service" workspace select default | ||
|
|
||
| echo "Delete workspace: ${workspace}" | ||
| terraform -chdir="infra/${app_name}/service" workspace delete "${workspace}" | ||
|
|
||
| pr_info=$(cat <<EOF | ||
| <!-- ${app_name} - begin PR environment info --> | ||
| ## Preview environment for ${app_name} | ||
| ♻️ Environment destroyed ♻️ | ||
| <!-- ${app_name} - end PR environment info --> | ||
| EOF | ||
| ) | ||
|
|
||
| echo "Update PR description with PR environment info" | ||
| echo "${pr_info}" | ||
|
|
||
| # Use a retry loop to handle race conditions when multiple workflows update the PR body | ||
| max_attempts=5 | ||
| attempt=0 | ||
| updated=false | ||
|
|
||
| while [ $attempt -lt $max_attempts ] && [ "$updated" = "false" ]; do | ||
| attempt=$((attempt + 1)) | ||
|
|
||
| # Read current PR body | ||
| pr_body="$(gh pr view "${pr_number}" --json body | jq --raw-output .body)" | ||
|
|
||
| # clean up older single-app section if present | ||
| if [[ $pr_body == *"<!-- begin PR environment info -->"*"<!-- end PR environment info -->"* ]]; then | ||
| pr_body="${pr_body//<!-- begin PR environment info -->*<!-- end PR environment info -->}" | ||
| fi | ||
|
|
||
| if [[ $pr_body == *"<!-- ${app_name} - begin PR environment info -->"*"<!-- ${app_name} - end PR environment info -->"* ]]; then | ||
| pr_body="${pr_body//<!-- ${app_name} - begin PR environment info -->*<!-- ${app_name} - end PR environment info -->/$pr_info}" | ||
| else | ||
| pr_body="${pr_body}"$'\n\n'"${pr_info}" | ||
| fi | ||
|
|
||
| # Try to update the PR body | ||
| if gh pr edit "${pr_number}" --body "${pr_body}"; then | ||
| updated=true | ||
| echo "Successfully updated PR description" | ||
| else | ||
| if [ $attempt -lt $max_attempts ]; then | ||
| echo "Failed to update PR description (attempt ${attempt}/${max_attempts}), retrying in 2s..." | ||
| sleep 2 | ||
| fi | ||
| fi | ||
| done | ||
|
|
||
| if [ "$updated" = "false" ]; then | ||
| echo "Warning: Failed to update PR description after ${max_attempts} attempts" | ||
| # Don't fail the workflow just because we couldn't update the PR body | ||
| echo "Continuing despite PR update failure..." | ||
| fi |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Link the full URL of the issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed the TODO comment to include the full URL in commit 59dbd28: 59dbd28
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't see that change in this PR?