Skip to content

Commit e88fd67

Browse files
pnarayanaswamypnarayanaswamy
authored
fix: use bitrise builds same as performance tests (#29926)
<!-- Please submit this PR as a draft initially. Do not mark it as "Ready for review" until this PR meets the canonical Definition of Ready For Review in `docs/readme/ready-for-review.md`. In short: the template must be materially complete (not just section titles present), all status checks must be currently passing, and the only expected follow-up commits must be reviewer-driven. --> ## **Description** For nightly system tests use the same Bitrise build + BrowserStack upload pipeline same as performance tests ## **Changelog** <!-- If this PR is not End-User-Facing and should not show up in the CHANGELOG, you can choose to either: 1. Write `CHANGELOG entry: null` 2. Label with `no-changelog` If this PR is End-User-Facing, please write a short User-Facing description in the past tense like: `CHANGELOG entry: Added a new tab for users to see their NFTs` `CHANGELOG entry: Fixed a bug that was causing some NFTs to flicker` (This helps the Release Engineer do their job more quickly and accurately) --> CHANGELOG entry: ## **Related issues** Fixes: ## **Manual testing steps** ```gherkin Feature: my feature name Scenario: user [verb for user action] Given [describe expected initial app state] When user [verb for user action] Then [describe expected outcome] ``` ## **Screenshots/Recordings** <!-- If applicable, add screenshots and/or recordings to visualize the before and after of your change. --> ### **Before** <!-- [screenshots/recordings] --> ### **After** <!-- [screenshots/recordings] --> ## **Pre-merge author checklist** <!-- Every checklist item must be consciously assessed before marking this PR as "Ready for review". A checked box means you deliberately considered that responsibility, not that you literally performed every action listed. Unchecked boxes are ambiguous: they are not an implicit "N/A" and they are not a silent "skip". See `docs/readme/ready-for-review.md` for the full checklist semantics. --> - [ ] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [ ] I've completed the PR template to the best of my ability - [ ] I've included tests if applicable - [ ] I've documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I've applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. #### Performance checks (if applicable) - [ ] I've tested on Android - Ideally on a mid-range device; emulator is acceptable - [ ] I've tested with a power user scenario - Use these [power-user SRPs](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/edit-v2/401401446401?draftShareId=9d77e1e1-4bdc-4be1-9ebb-ccd916988d93) to import wallets with many accounts and tokens - [ ] I've instrumented key operations with Sentry traces for production performance metrics - See [`trace()`](/app/util/trace.ts) for usage and [`addToken`](/app/components/Views/AddAsset/components/AddCustomToken/AddCustomToken.tsx#L274) for an example For performance guidelines and tooling, see the [Performance Guide](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/400085549067/Performance+Guide+for+Engineers). ## **Pre-merge reviewer checklist** <!-- Reviewer checklist items follow the same semantics as the author checklist: an unchecked box is ambiguous, a checked box means the reviewer consciously assessed that responsibility. See `docs/readme/ready-for-review.md`. --> - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Medium risk because it rewires the nightly system test build/upload pipeline to depend on Bitrise workflow outputs; misconfigured inputs/outputs or permissions could cause tests to skip or run against wrong app builds. > > **Overview** > **Nightly system tests now use the same Bitrise → BrowserStack pipeline as performance tests.** The `run-system-tests.yml` workflow replaces the in-GitHub build + artifact upload flow with reusable `build-android-upload-to-browserstack.yml` / `build-ios-upload-to-browserstack.yml` calls and wires test jobs to consume their `with-srp`/`without-srp` BrowserStack URLs (or manual inputs). > > It also removes the dedicated `upload-to-browserstack` job and updates job gating/permissions to run/skip appropriately based on whether Bitrise builds were triggered or manual BrowserStack URLs were provided. > > Separately, `builds.yml` drops the `main-rc-with-srp` build configuration that previously baked SRP/password into the RC build for system tests. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8254c01. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent ca9df54 commit e88fd67

2 files changed

Lines changed: 32 additions & 168 deletions

File tree

‎.github/workflows/run-system-tests.yml‎

Lines changed: 31 additions & 150 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@
44
# no Sentry, no performance metrics). Validates that core user flows (login,
55
# onboarding, swaps, etc.) work on real devices via BrowserStack.
66
#
7+
# Builds are triggered on Bitrise (same pipeline as performance tests) and
8+
# uploaded to BrowserStack automatically.
9+
#
710
# Triggers:
811
# - Daily at 5 AM UTC (1 hour after nightly build starts)
912
# - Manually via workflow_dispatch (optionally with pre-built BrowserStack URLs)
@@ -33,171 +36,46 @@ on:
3336
type: string
3437

3538
permissions:
36-
contents: write
39+
contents: read
3740
id-token: write
3841

3942
concurrency:
4043
group: system-tests-${{ github.ref }}
4144
cancel-in-progress: true
45+
4246
jobs:
43-
build-with-srp:
44-
name: Build with-SRP (login tests)
47+
trigger-android-dual-versions:
48+
name: Build Android (Bitrise + BrowserStack)
49+
uses: ./.github/workflows/build-android-upload-to-browserstack.yml
4550
if: >-
4651
github.event_name != 'workflow_dispatch'
4752
|| (!inputs.browserstack_app_url_android
48-
&& !inputs.browserstack_app_url_ios
49-
&& !inputs.browserstack_clean_app_url_android
50-
&& !inputs.browserstack_clean_app_url_ios)
51-
uses: ./.github/workflows/build.yml
53+
&& !inputs.browserstack_clean_app_url_android)
5254
with:
53-
build_name: main-rc-with-srp
54-
platform: both
55-
skip_version_bump: true
56-
source_branch: ${{ github.ref_name }}
55+
branch_name: ${{ github.ref_name }}
5756
secrets: inherit
5857

59-
build-clean:
60-
name: Build clean RC (onboarding tests)
58+
trigger-ios-dual-versions:
59+
name: Build iOS (Bitrise + BrowserStack)
60+
uses: ./.github/workflows/build-ios-upload-to-browserstack.yml
6161
if: >-
6262
github.event_name != 'workflow_dispatch'
63-
|| (!inputs.browserstack_app_url_android
64-
&& !inputs.browserstack_app_url_ios
65-
&& !inputs.browserstack_clean_app_url_android
63+
|| (!inputs.browserstack_app_url_ios
6664
&& !inputs.browserstack_clean_app_url_ios)
67-
uses: ./.github/workflows/build.yml
6865
with:
69-
build_name: main-rc
70-
platform: both
71-
skip_version_bump: true
72-
source_branch: ${{ github.ref_name }}
66+
branch_name: ${{ github.ref_name }}
7367
secrets: inherit
7468

75-
upload-to-browserstack:
76-
name: Upload builds to BrowserStack
77-
needs: [build-with-srp, build-clean]
78-
if: always() && !cancelled()
79-
runs-on: ubuntu-latest
80-
outputs:
81-
android-login-url: ${{ steps.resolve.outputs.android-login-url }}
82-
ios-login-url: ${{ steps.resolve.outputs.ios-login-url }}
83-
android-onboarding-url: ${{ steps.resolve.outputs.android-onboarding-url }}
84-
ios-onboarding-url: ${{ steps.resolve.outputs.ios-onboarding-url }}
85-
env:
86-
BROWSERSTACK_USERNAME: ${{ secrets.BROWSERSTACK_USERNAME }}
87-
BROWSERSTACK_ACCESS_KEY: ${{ secrets.BROWSERSTACK_ACCESS_KEY }}
88-
steps:
89-
- name: Download Android APK (with-SRP)
90-
if: needs.build-with-srp.result == 'success'
91-
uses: actions/download-artifact@v8
92-
with:
93-
name: android-apk-main-rc-with-srp
94-
path: artifacts/android-with-srp
95-
96-
- name: Download iOS IPA (with-SRP)
97-
if: needs.build-with-srp.result == 'success'
98-
uses: actions/download-artifact@v8
99-
with:
100-
name: ios-ipa-main-rc-with-srp
101-
path: artifacts/ios-with-srp
102-
103-
- name: Download Android APK (clean)
104-
if: needs.build-clean.result == 'success'
105-
uses: actions/download-artifact@v8
106-
with:
107-
name: android-apk-main-rc
108-
path: artifacts/android-clean
109-
110-
- name: Download iOS IPA (clean)
111-
if: needs.build-clean.result == 'success'
112-
uses: actions/download-artifact@v8
113-
with:
114-
name: ios-ipa-main-rc
115-
path: artifacts/ios-clean
116-
117-
- name: Upload artifacts to BrowserStack and resolve URLs
118-
id: resolve
119-
run: |
120-
# Helper: resolve a single URL from manual input or build artifact.
121-
# Outputs the URL to GITHUB_OUTPUT, or empty string if unavailable.
122-
# Upload failures are logged but do not abort other resolutions.
123-
resolve_url() {
124-
local output_key="$1"
125-
local manual_url="$2"
126-
local artifact_dir="$3"
127-
local extension="$4"
128-
local custom_id="$5"
129-
130-
if [[ -n "$manual_url" ]]; then
131-
echo "${output_key}=${manual_url}" >> "$GITHUB_OUTPUT"
132-
echo "Using manual URL for ${output_key}: ${manual_url}"
133-
return 0
134-
fi
135-
136-
if [[ ! -d "$artifact_dir" ]]; then
137-
echo "No build artifact for ${output_key} and no manual URL — test will be skipped"
138-
echo "${output_key}=" >> "$GITHUB_OUTPUT"
139-
return 0
140-
fi
141-
142-
local file
143-
file=$(find "$artifact_dir" -name "*.${extension}" | head -1)
144-
if [[ -z "$file" ]]; then
145-
echo "::warning::No .${extension} found in ${artifact_dir} — ${output_key} will be empty"
146-
echo "${output_key}=" >> "$GITHUB_OUTPUT"
147-
return 0
148-
fi
149-
150-
echo "Uploading ${file} to BrowserStack..."
151-
local response
152-
response=$(curl -s -u "$BROWSERSTACK_USERNAME:$BROWSERSTACK_ACCESS_KEY" \
153-
-X POST "https://api-cloud.browserstack.com/app-automate/upload" \
154-
-F "file=@${file}" \
155-
-F "custom_id=${custom_id}")
156-
157-
local app_url
158-
app_url=$(echo "$response" | jq -r '.app_url')
159-
160-
if [[ -z "$app_url" || "$app_url" == "null" ]]; then
161-
echo "::error::BrowserStack upload failed for ${output_key} (${file}): ${response}"
162-
echo "${output_key}=" >> "$GITHUB_OUTPUT"
163-
return 0
164-
fi
165-
166-
echo "${output_key}=${app_url}" >> "$GITHUB_OUTPUT"
167-
echo "${output_key}: ${app_url}"
168-
}
169-
170-
resolve_url "android-login-url" \
171-
"${{ inputs.browserstack_app_url_android }}" \
172-
"artifacts/android-with-srp" "apk" \
173-
"system-test-android-login-${{ github.run_id }}"
174-
175-
resolve_url "ios-login-url" \
176-
"${{ inputs.browserstack_app_url_ios }}" \
177-
"artifacts/ios-with-srp" "ipa" \
178-
"system-test-ios-login-${{ github.run_id }}"
179-
180-
resolve_url "android-onboarding-url" \
181-
"${{ inputs.browserstack_clean_app_url_android }}" \
182-
"artifacts/android-clean" "apk" \
183-
"system-test-android-onboarding-${{ github.run_id }}"
184-
185-
resolve_url "ios-onboarding-url" \
186-
"${{ inputs.browserstack_clean_app_url_ios }}" \
187-
"artifacts/ios-clean" "ipa" \
188-
"system-test-ios-onboarding-${{ github.run_id }}"
189-
190-
echo "URL resolution complete"
191-
19269
run-android-login-tests:
19370
name: Android Login Tests
194-
needs: [upload-to-browserstack]
71+
needs: [trigger-android-dual-versions]
19572
if: >-
19673
always() && !cancelled()
197-
&& needs.upload-to-browserstack.outputs.android-login-url != ''
74+
&& (needs.trigger-android-dual-versions.result == 'skipped' || needs.trigger-android-dual-versions.result == 'success')
75+
&& (needs.trigger-android-dual-versions.outputs.with-srp-browserstack-url != '' || inputs.browserstack_app_url_android != '')
19876
runs-on: ubuntu-latest
19977
env:
200-
BROWSERSTACK_ANDROID_APP_URL: ${{ needs.upload-to-browserstack.outputs.android-login-url }}
78+
BROWSERSTACK_ANDROID_APP_URL: ${{ needs.trigger-android-dual-versions.outputs.with-srp-browserstack-url || inputs.browserstack_app_url_android }}
20179
steps:
20280
- name: Checkout code
20381
uses: actions/checkout@v6
@@ -289,13 +167,14 @@ jobs:
289167

290168
run-android-onboarding-tests:
291169
name: Android Onboarding Tests
292-
needs: [upload-to-browserstack]
170+
needs: [trigger-android-dual-versions]
293171
if: >-
294172
always() && !cancelled()
295-
&& needs.upload-to-browserstack.outputs.android-onboarding-url != ''
173+
&& (needs.trigger-android-dual-versions.result == 'skipped' || needs.trigger-android-dual-versions.result == 'success')
174+
&& (needs.trigger-android-dual-versions.outputs.without-srp-browserstack-url != '' || inputs.browserstack_clean_app_url_android != '')
296175
runs-on: ubuntu-latest
297176
env:
298-
BROWSERSTACK_ANDROID_CLEAN_APP_URL: ${{ needs.upload-to-browserstack.outputs.android-onboarding-url }}
177+
BROWSERSTACK_ANDROID_CLEAN_APP_URL: ${{ needs.trigger-android-dual-versions.outputs.without-srp-browserstack-url || inputs.browserstack_clean_app_url_android }}
299178
steps:
300179
- name: Checkout code
301180
uses: actions/checkout@v6
@@ -387,13 +266,14 @@ jobs:
387266

388267
run-ios-login-tests:
389268
name: iOS Login Tests
390-
needs: [upload-to-browserstack]
269+
needs: [trigger-ios-dual-versions]
391270
if: >-
392271
always() && !cancelled()
393-
&& needs.upload-to-browserstack.outputs.ios-login-url != ''
272+
&& (needs.trigger-ios-dual-versions.result == 'skipped' || needs.trigger-ios-dual-versions.result == 'success')
273+
&& (needs.trigger-ios-dual-versions.outputs.with-srp-browserstack-url != '' || inputs.browserstack_app_url_ios != '')
394274
runs-on: ubuntu-latest
395275
env:
396-
BROWSERSTACK_IOS_APP_URL: ${{ needs.upload-to-browserstack.outputs.ios-login-url }}
276+
BROWSERSTACK_IOS_APP_URL: ${{ needs.trigger-ios-dual-versions.outputs.with-srp-browserstack-url || inputs.browserstack_app_url_ios }}
397277
steps:
398278
- name: Checkout code
399279
uses: actions/checkout@v6
@@ -485,13 +365,14 @@ jobs:
485365

486366
run-ios-onboarding-tests:
487367
name: iOS Onboarding Tests
488-
needs: [upload-to-browserstack]
368+
needs: [trigger-ios-dual-versions]
489369
if: >-
490370
always() && !cancelled()
491-
&& needs.upload-to-browserstack.outputs.ios-onboarding-url != ''
371+
&& (needs.trigger-ios-dual-versions.result == 'skipped' || needs.trigger-ios-dual-versions.result == 'success')
372+
&& (needs.trigger-ios-dual-versions.outputs.without-srp-browserstack-url != '' || inputs.browserstack_clean_app_url_ios != '')
492373
runs-on: ubuntu-latest
493374
env:
494-
BROWSERSTACK_IOS_CLEAN_APP_URL: ${{ needs.upload-to-browserstack.outputs.ios-onboarding-url }}
375+
BROWSERSTACK_IOS_CLEAN_APP_URL: ${{ needs.trigger-ios-dual-versions.outputs.without-srp-browserstack-url || inputs.browserstack_clean_app_url_ios }}
495376
steps:
496377
- name: Checkout code
497378
uses: actions/checkout@v6

‎builds.yml‎

Lines changed: 1 addition & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -214,24 +214,7 @@ builds:
214214
secrets: *secrets
215215
code_fencing: *code_fencing_main
216216

217-
# Release candidate with pre-imported wallet (system tests / performance tests)
218-
# Same as main-rc but bakes in ADDITIONAL_SRP_1 + PREDEFINED_PASSWORD so the app
219-
# starts with an imported wallet, skipping onboarding. Used by run-system-tests.yml.
220-
main-rc-with-srp:
221-
script_name: main-rc
222-
github_environment: build-rc
223-
signing: *signing_rc
224-
env:
225-
<<: *public_envs
226-
METAMASK_ENVIRONMENT: 'rc'
227-
METAMASK_BUILD_TYPE: 'main'
228-
secrets:
229-
<<: *secrets
230-
ADDITIONAL_SRP_1: 'IN_BUILD_SRP'
231-
PREDEFINED_PASSWORD: 'E2E_PASSWORD'
232-
code_fencing: *code_fencing_main
233-
234-
# Test builds
217+
# Test builds (QA)
235218
main-test:
236219
github_environment: build-uat
237220
signing: *signing_uat

0 commit comments

Comments
 (0)