Skip to content

Commit 916ac40

Browse files
committed
chore(runway): cherry-pick test: integrate AI test plan into RC build comments (#28951)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until the template has been completely filled out, and PR status checks have passed at least once. --> ## **Description** <!-- Write a short description of the changes included in this pull request, also include relevant motivation and context. Have in mind the following questions: 1. What is the reason for the change? 2. What is the improvement/solution? --> Integrates AI test plan generation directly into RC build comments Changes - New scripts/build-announce/ module (TypeScript): - index.ts - Main orchestrator (replaces post-rc-build-comment.mjs) - test-plan-section.ts - Converts test plan JSON to markdown with collapsible sections - types.ts - TypeScript interfaces - utils.ts - Helper functions (GitHub API, URL validation) - Updated build-rc-auto.yml: - Generates test plan inline during RC build workflow - Removed label requirement (fully automatic) - Uploads JSON to gh-pages for automation - Deleted: - generate-rc-test-plan.yml (no longer separate workflow) - post-rc-build-comment.mjs (replaced by TypeScript) What it does - Stats table (risk score, high/medium risk counts, files changed, teams signed off) - Collapsible sections for executive summary, high/medium risk scenarios - Teams sign-off status section - Excluded features (feature flags) section - JSON link at bottom for automation - Graceful fallback if test plan generation fails ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: null ## **Related issues** Fixes: https://consensyssoftware.atlassian.net/browse/MCRM-69 ## **Manual testing steps** ```gherkin Feature: my feature name Scenario: user [verb for user action] Given [describe expected initial app state] When user [verb for user action] Then [describe expected outcome] ``` ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> #28355 (comment) <img width="608" height="421" alt="Screenshot 2026-04-16 at 5 38 08 PM" src="https://github.com/user-attachments/assets/2d211150-9e2e-4ac7-8f71-f74218989159" /> ### **After** Should match Extension RC PR comment: MetaMask/metamask-extension#41626 (comment) <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** - [ ] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I've included tests if applicable - [ ] I've documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I've applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. #### Performance checks (if applicable) - [ ] I've tested on Android - Ideally on a mid-range device; emulator is acceptable - [ ] I've tested with a power user scenario - Use these [power-user SRPs](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/edit-v2/401401446401?draftShareId=9d77e1e1-4bdc-4be1-9ebb-ccd916988d93) to import wallets with many accounts and tokens - [ ] I've instrumented key operations with Sentry traces for production performance metrics - See [`trace()`](/app/util/trace.ts) for usage and [`addToken`](/app/components/Views/AddAsset/components/AddCustomToken/AddCustomToken.tsx#L274) for an example For performance guidelines and tooling, see the [Performance Guide](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/400085549067/Performance+Guide+for+Engineers). ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes the release CI workflow behavior and permissions (auto-triggering RC builds, generating content via external AI providers, and pushing to `gh-pages`), which could affect release pipeline reliability and repository writes if misconfigured. > > **Overview** > **RC builds are now fully automatic for release branches with an open PR.** The `build-rc-auto.yml` workflow no longer gates builds on an `auto-rc-builds` label; instead it finds the associated PR and proceeds when one exists. > > **RC build announcements now optionally include an AI-generated test plan inline.** The workflow replaces `scripts/post-rc-build-comment.mjs` with a new TypeScript `scripts/build-announce/` runner that posts the build links, generates/parses `release-test-plan.json` via `tests/tools/e2e-ai-analyzer` when AI provider keys are present, formats results into collapsible markdown sections, and minimizes older RC announce comments. > > **Test plan hosting is consolidated into the RC build workflow.** The standalone `generate-rc-test-plan.yml` comment-triggered workflow is removed, and the RC workflow now attempts (best-effort) to commit the generated JSON to the `gh-pages` branch under `test-plans/` for later automation/links. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3f0ea37. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 254a150 commit 916ac40

7 files changed

Lines changed: 842 additions & 606 deletions

File tree

.github/workflows/build-rc-auto.yml

Lines changed: 79 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
11
##############################################################################################
22
#
33
# This Workflow is responsible for triggering release candidate builds (iOS & Android).
4-
# It runs on every commit pushed to a release branch, but only when the release PR
5-
# has the 'auto-rc-builds' label.
4+
# It runs automatically on every commit pushed to a release branch that has an open PR.
65
#
76
# Rolling builds: when a new run starts for the same release branch, any queued or
87
# in-progress run of this workflow for that branch is cancelled (same behavior as
9-
# Bitrise Rolling builds / Abort running builds for one branch + one workflow).
8+
# Bitrise "Rolling builds" / "Abort running builds" for one branch + one workflow).
109
#
1110
# Version bump runs once (update-latest-build-version.yml), then iOS and Android
1211
# builds are triggered in parallel via runway-ota-build-core.yml (skip_version_bump).
1312
#
13+
# The RC build comment includes an AI-generated test plan (inline with collapsible sections).
14+
#
1415
##############################################################################################
1516
name: Auto RC builds
1617

@@ -30,14 +31,14 @@ permissions:
3031
id-token: write
3132

3233
jobs:
33-
validate-and-check-label:
34-
name: Validate branch and check PR label
34+
validate-and-find-pr:
35+
name: Validate branch and find PR
3536
runs-on: ubuntu-latest
3637
outputs:
3738
semver: ${{ steps.extract-version.outputs.semver }}
38-
has-label: ${{ steps.check-label.outputs.has-label }}
39+
has-pr: ${{ steps.find-pr.outputs.has-pr }}
3940
branch-name: ${{ steps.extract-version.outputs.branch-name }}
40-
pr-number: ${{ steps.check-label.outputs.pr-number }}
41+
pr-number: ${{ steps.find-pr.outputs.pr-number }}
4142
permissions:
4243
pull-requests: read
4344
steps:
@@ -64,8 +65,8 @@ jobs:
6465
exit 1
6566
fi
6667
67-
- name: Find PR and check for auto-rc-builds label
68-
id: check-label
68+
- name: Find associated PR
69+
id: find-pr
6970
run: |
7071
BRANCH_NAME="${{ github.ref_name }}"
7172
echo "Looking for PR with head branch: $BRANCH_NAME"
@@ -75,45 +76,36 @@ jobs:
7576
7677
if [[ -z "$PR_NUMBER" ]]; then
7778
echo "No PR found for branch $BRANCH_NAME. Skipping."
78-
echo "has-label=false" >> "$GITHUB_OUTPUT"
79+
echo "has-pr=false" >> "$GITHUB_OUTPUT"
7980
exit 0
8081
fi
8182
82-
echo "Found PR #$PR_NUMBER"
83+
echo "Found PR #$PR_NUMBER - proceeding with RC build"
8384
echo "pr-number=$PR_NUMBER" >> "$GITHUB_OUTPUT"
84-
85-
# Check if PR has the auto-rc-builds label
86-
LABELS=$(gh pr view "$PR_NUMBER" --json labels --jq '.labels[].name' || echo "")
87-
if echo "$LABELS" | grep -qx "auto-rc-builds"; then
88-
echo "PR #$PR_NUMBER has 'auto-rc-builds' label. Proceeding with build."
89-
echo "has-label=true" >> "$GITHUB_OUTPUT"
90-
else
91-
echo "PR #$PR_NUMBER does not have 'auto-rc-builds' label. Skipping build."
92-
echo "has-label=false" >> "$GITHUB_OUTPUT"
93-
fi
85+
echo "has-pr=true" >> "$GITHUB_OUTPUT"
9486
env:
9587
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
9688

9789
update_rc_build_version:
9890
name: Update RC build version
9991
uses: ./.github/workflows/update-latest-build-version.yml
100-
needs: validate-and-check-label
101-
if: needs.validate-and-check-label.outputs.has-label == 'true'
92+
needs: validate-and-find-pr
93+
if: needs.validate-and-find-pr.outputs.has-pr == 'true'
10294
permissions:
10395
contents: write
10496
id-token: write
10597
with:
106-
base-branch: ${{ needs.validate-and-check-label.outputs.branch-name }}
98+
base-branch: ${{ needs.validate-and-find-pr.outputs.branch-name }}
10799
secrets:
108100
PR_TOKEN: ${{ secrets.PR_TOKEN }}
109101

110102
trigger-ios-rc-build:
111103
name: Trigger iOS RC Build
112104
uses: ./.github/workflows/runway-ota-build-core.yml
113105
needs:
114-
- validate-and-check-label
106+
- validate-and-find-pr
115107
- update_rc_build_version
116-
if: needs.validate-and-check-label.outputs.has-label == 'true'
108+
if: needs.validate-and-find-pr.outputs.has-pr == 'true'
117109
with:
118110
platform: ios
119111
source_branch: ${{ needs.update_rc_build_version.outputs.commit-hash }}
@@ -124,57 +116,105 @@ jobs:
124116
name: Trigger Android RC Build
125117
uses: ./.github/workflows/runway-ota-build-core.yml
126118
needs:
127-
- validate-and-check-label
119+
- validate-and-find-pr
128120
- update_rc_build_version
129-
if: needs.validate-and-check-label.outputs.has-label == 'true'
121+
if: needs.validate-and-find-pr.outputs.has-pr == 'true'
130122
with:
131123
platform: android
132124
source_branch: ${{ needs.update_rc_build_version.outputs.commit-hash }}
133125
skip_version_bump: true
134126
secrets: inherit
135127

136128
post-rc-build-comment:
137-
name: Post RC Build Comment
129+
name: Post RC Build Comment with Test Plan
138130
runs-on: ubuntu-latest
139131
needs:
140-
- validate-and-check-label
132+
- validate-and-find-pr
141133
- trigger-ios-rc-build
142134
- trigger-android-rc-build
143-
if: always() && needs.trigger-ios-rc-build.result == 'success' && needs.trigger-android-rc-build.result == 'success' && needs.validate-and-check-label.outputs.pr-number != ''
135+
if: always() && needs.trigger-ios-rc-build.result == 'success' && needs.trigger-android-rc-build.result == 'success' && needs.validate-and-find-pr.outputs.pr-number != ''
136+
environment: release-ci
144137
permissions:
145138
pull-requests: write
139+
contents: write
146140
steps:
147141
- uses: actions/checkout@v4
148142
with:
149-
ref: ${{ needs.validate-and-check-label.outputs.branch-name }}
150-
fetch-depth: 1
143+
ref: ${{ needs.validate-and-find-pr.outputs.branch-name }}
144+
fetch-depth: 50
151145
- uses: actions/setup-node@v4
152146
with:
153147
node-version-file: '.nvmrc'
154148
cache: yarn
155149
- name: Install dependencies
156150
run: yarn install --immutable
157-
- name: Post RC Build Comment
158-
run: node scripts/post-rc-build-comment.mjs
151+
- name: Post RC Build Comment with Test Plan
152+
run: node -r esbuild-register scripts/build-announce/index.ts
153+
timeout-minutes: 8
159154
env:
160155
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
161156
GITHUB_REPOSITORY: ${{ github.repository }}
162-
PR_NUMBER: ${{ needs.validate-and-check-label.outputs.pr-number }}
163-
SEMVER: ${{ needs.validate-and-check-label.outputs.semver }}
157+
PR_NUMBER: ${{ needs.validate-and-find-pr.outputs.pr-number }}
158+
SEMVER: ${{ needs.validate-and-find-pr.outputs.semver }}
164159
IOS_BUILD_NUMBER: ${{ needs.trigger-ios-rc-build.outputs.ios_version_code }}
165160
ANDROID_BUILD_NUMBER: ${{ needs.trigger-android-rc-build.outputs.android_version_code }}
166161
BUILD_PIPELINE_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
162+
# AI Provider keys for test plan generation
163+
E2E_CLAUDE_API_KEY: ${{ secrets.E2E_CLAUDE_API_KEY }}
164+
E2E_OPENAI_API_KEY: ${{ secrets.E2E_OPENAI_API_KEY }}
165+
E2E_GEMINI_API_KEY: ${{ secrets.E2E_GEMINI_API_KEY }}
166+
167+
- name: Upload test plan JSON to GitHub Pages
168+
if: always()
169+
continue-on-error: true
170+
run: |
171+
VERSION="${{ needs.validate-and-find-pr.outputs.semver }}"
172+
JSON_FILE="release-test-plan.json"
173+
174+
# Check if JSON file was generated
175+
if [ ! -f "$JSON_FILE" ]; then
176+
echo "No test plan JSON file found, skipping upload"
177+
exit 0
178+
fi
179+
180+
# Save JSON to temp
181+
cp "$JSON_FILE" "/tmp/test-plan-${VERSION}.json"
182+
183+
# Configure git
184+
git config user.name "github-actions[bot]"
185+
git config user.email "github-actions[bot]@users.noreply.github.com"
186+
187+
# Fetch gh-pages branch or create it
188+
git fetch origin gh-pages:gh-pages 2>/dev/null || echo "gh-pages doesn't exist yet"
189+
190+
# Switch to gh-pages
191+
if git checkout gh-pages 2>/dev/null; then
192+
echo "Switched to existing gh-pages branch"
193+
else
194+
git checkout --orphan gh-pages
195+
git rm -rf . 2>/dev/null || true
196+
git clean -fd 2>/dev/null || true
197+
fi
198+
199+
# Create test-plans directory and copy JSON
200+
mkdir -p test-plans
201+
cp "/tmp/test-plan-${VERSION}.json" "test-plans/"
202+
203+
# Commit and push
204+
git add test-plans/
205+
git commit -m "Add test plan JSON for RC ${VERSION}" || echo "No changes to commit"
206+
git push origin gh-pages
167207
168208
slack-notification:
169209
name: Slack RC Notification
170210
needs:
171-
- validate-and-check-label
211+
- validate-and-find-pr
172212
- trigger-ios-rc-build
173213
- trigger-android-rc-build
174214
if: always() && needs.trigger-ios-rc-build.result == 'success' && needs.trigger-android-rc-build.result == 'success'
175215
uses: ./.github/workflows/slack-rc-notification.yml
176216
with:
177-
source_branch: ${{ needs.validate-and-check-label.outputs.branch-name }}
217+
source_branch: ${{ needs.validate-and-find-pr.outputs.branch-name }}
178218
semver: ${{ needs.trigger-ios-rc-build.outputs.semantic_version }}
179219
ios_build_number: ${{ needs.trigger-ios-rc-build.outputs.ios_version_code }}
180220
android_build_number: ${{ needs.trigger-android-rc-build.outputs.android_version_code }}

0 commit comments

Comments
 (0)