ci(MMQA-1667): skip native builds when PR only changes E2E/performance tests by javiergarciavera · Pull Request #33180 · MetaMask/metamask-mobile

javiergarciavera · 2026-07-13T08:36:44Z

Description

PRs that only touch E2E or performance test files currently trigger full iOS and Android native builds even though the app binary does not change. This adds test-only change detection so CI still runs Smart E2E Selection (including performance tag selection), but reuses the latest matching builds from main instead of compiling fresh native artifacts.

an example where the PR only touches test files and does not create builds for e2e, but reuse ones from main can be seen in
#33188

What changed

.github/rules/filter-rules.yml: new e2e_test_files and e2e_test_or_ignorable filters for smoke/regression/performance test paths and shared E2E infra.
.github/workflows/get-requirements.yml: new native_build_needed output; test-only PRs set native_build_needed=false while keeping E2E + Smart selection enabled.
.github/actions/find-reusable-build: main-branch-only mode to search only the base branch when reusing builds.
.github/workflows/build-android-e2e.yml / build-ios-e2e.yml: reuse-main-builds-only input skips repack and fails fast if no main artifact is found.
.github/workflows/ci.yml: wires native_build_needed into E2E and performance build jobs.
.github/workflows/run-performance-e2e.yml: skips fresh BrowserStack Android builds on test-only PRs and resolves the latest uploaded apps via BrowserStack API.
.github/guidelines/E2E_DECISION_TREE.md: documents the new test-only path.

force-builds label / [force-builds] commit tag still forces fresh native builds.

Changelog

CHANGELOG entry: null

Related issues

Fixes: MMQA-1667

Manual testing steps

Feature: test-only PR CI build reuse

  Scenario: PR with only E2E test file changes
    Given a PR that only modifies files under tests/smoke/, tests/regression/, or tests/performance/
    When CI runs on the PR
    Then Smart E2E Selection still runs
    And build-android-apks / build-ios-apps reuse main-branch artifacts (no native compile, no JS repack)
    And selected E2E / performance suites still execute

  Scenario: PR with app code changes
    Given a PR that modifies app/ or native project files
    When CI runs on the PR
    Then native iOS/Android builds run as before

N/A for on-device manual QA — CI workflow change only. Validated locally with compute-e2e-platform-flags unit tests and the run-compute-e2e-platform-flags GitHub Actions entrypoint.

Screenshots/Recordings

N/A — CI infrastructure change with no UI impact.

Before

N/A

After

N/A

Pre-merge author checklist

I've followed MetaMask Contributor Docs and MetaMask Mobile Coding Standards.
I've completed the PR template to the best of my ability
I've included tests if applicable
I've documented my code using JSDoc format if applicable
I've applied the right labels on the PR (see labeling guidelines). Not required for external contributors.

Performance checks (if applicable)

I've tested on Android
I've tested with a power user scenario
I've instrumented key operations with Sentry traces for production performance metrics

Pre-merge reviewer checklist

I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

When a PR only changes E2E or performance test files, CI still runs Smart E2E Selection and the selected test suites, but reuses the latest matching iOS/Android artifacts from main instead of compiling fresh native builds. Performance PR workflows also resolve existing BrowserStack apps instead of triggering new Android builds. Adds test-only change detection in get-requirements, a native_build_needed output, main-branch-only reuse in find-reusable-build, and unit tests for the platform flag decision logic. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

metamask-ci · 2026-07-13T08:37:34Z

PR template — items to address before "Ready for review"

Warnings — informational, address before merging:

Pre-merge author checklist has unchecked items (e.g. "I've tested on Android"). Every box must be consciously checked — see docs/readme/ready-for-review.md.

See docs/readme/ready-for-review.md for the full Definition of Ready for Review.

get-requirements runs before yarn install, so esbuild-register was not available. Move platform flag logic to plain CommonJS that runs with node only. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

Address CodeQL finding by validating app_url and custom_id against strict allowlists before persisting network data to the GitHub Actions output file. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

Test-only PRs were fingerprinting the PR merge tree, so find-reusable-build could not match completed main branch runs. Compute the fingerprint from main, relax metadata requirements for test-only reuse, and fall back to the main APK cache on Android. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 615857d. Configure here.}

When reuse-main-builds-only is active, force-builds should compile fresh instead of failing because no main artifact was found. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

…ceful fallback - Encode branch slug in BrowserStack custom_id uploads and resolve only main-prefixed apps for test-only performance E2E reuse - Use native_build_needed == 'false' for reuse mode (fail-safe default) - Fall back to fresh native build when main artifacts are not ready yet - Document race window and BrowserStack main-only resolution Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

github-actions · 2026-07-13T13:03:19Z

⚡ Performance Test Results

ℹ️ Performance test results are currently non-blocking and will not block this PR.

❌ 5 tests failed · 20 tests · 1 device

📱 Devices tested (1)

Android: Google Pixel 8 Pro (v14.0)

❌ Failed Tests (5)

@swap-bridge-dev-team

Test	Platform	Device	Reason	Recording
Cross-chain swap flow - ETH to SOL - 50+ accounts, SRP 1 + SRP 2 + SRP 3	Android	Google Pixel 8 Pro (v14.0)	Test error	📹 Watch

@metamask-mobile-platform

Test	Platform	Device	Reason	Recording
Connect to Uniswap dapp, edit accounts, choose another account, and skip Solana popup	Android	Google Pixel 8 Pro (v14.0)	Test error	📹 Watch
Cold Start after importing a wallet	Android	Google Pixel 8 Pro (v14.0)	Quality gates exceeded	📹 Watch

@metamask-onboarding-team

Test	Platform	Device	Reason	Recording
Account creation after fresh install	Android	Google Pixel 8 Pro (v14.0)	Quality gates exceeded	📹 Watch

@mm-perps-engineering-team

Test	Platform	Device	Reason	Recording
Perps open position and close it	Android	Google Pixel 8 Pro (v14.0)	Test error	📹 Watch

✅ Passed Tests (15)

Test	Platform	Device	Duration	Team	Recording
Aggregated Balance Loading Time, SRP 1 + SRP 2 + SRP 3	Android	Google Pixel 8 Pro (v14.0)	8.42s	@assets-dev-team	📹 Watch
Asset View, SRP 1 + SRP 2 + SRP 3	Android	Google Pixel 8 Pro (v14.0)	2.43s	@assets-dev-team	📹 Watch
Swap flow - ETH to LINK, SRP 1 + SRP 2 + SRP 3	Android	Google Pixel 8 Pro (v14.0)	4.66s	@swap-bridge-dev-team	📹 Watch
Import SRP with +50 accounts, SRP 1, SRP 2, SRP 3	Android	Google Pixel 8 Pro (v14.0)	4.04s	@Accounts-team	📹 Watch
Cold Start: Measure ColdStart To Login Screen	Android	Google Pixel 8 Pro (v14.0)	4.16s	@metamask-mobile-platform	📹 Watch
Measure Warm Start: Login To Wallet Screen	Android	Google Pixel 8 Pro (v14.0)	2.09s	@metamask-mobile-platform	📹 Watch
Measure Warm Start: Warm Start to Login Screen	Android	Google Pixel 8 Pro (v14.0)	1.57s	@metamask-mobile-platform	📹 Watch
Perps add funds	Android	Google Pixel 8 Pro (v14.0)	8.98s	@mm-perps-engineering-team	📹 Watch
Predict Available Balance - Complete Flow Performance	Android	Google Pixel 8 Pro (v14.0)	2.28s	@team-predict	📹 Watch
Predict Deposit - Complete Flow Performance	Android	Google Pixel 8 Pro (v14.0)	11.29s	@team-predict	📹 Watch
Predict Market Details - Complete Flow Performance	Android	Google Pixel 8 Pro (v14.0)	3.97s	@team-predict	📹 Watch
Measure Cold Start To Onboarding Screen	Android	Google Pixel 8 Pro (v14.0)	2.91s	@metamask-mobile-platform	📹 Watch
Onboarding Import SRP with +50 accounts, SRP 3	Android	Google Pixel 8 Pro (v14.0)	8.37s	@metamask-onboarding-team	📹 Watch
Seedless Onboarding: Apple Login New User	Android	Google Pixel 8 Pro (v14.0)	12.07s	@metamask-onboarding-team	📹 Watch
Seedless Onboarding: Google Login New User	Android	Google Pixel 8 Pro (v14.0)	11.64s	@metamask-onboarding-team	📹 Watch

Branch: MMQA-1667-reduce-builds-amount · Build: Normal · Commit: 95c382d · View full run

test:unit only matches .github/**/*.test.ts; rename the two new browserstack and e2e-platform-flags tests so they run in CI. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

github-actions · 2026-07-13T14:16:59Z

🔍 Smart E2E Test Selection

Selected E2E tags: SmokeAccounts, SmokeConfirmations, SmokeNetworkAbstractions, SmokeNetworkExpansion, SmokeSwap, SmokeStake, SmokeWalletPlatform, SmokeMoney, SmokePerps, SmokeMultiChainAPI, SmokePredictions, SmokeSeedlessOnboarding, SmokeBrowser, SmokeSnaps
Selected Performance tags: None (no tests recommended)
Risk Level: high
AI Confidence: 82%

click to see 🤖 AI reasoning details

E2E Test Selection:
This PR makes significant changes to the CI/CD pipeline infrastructure:

New "test-only PR" optimization: When a PR only changes E2E/performance test files, CI now reuses native builds from main instead of compiling fresh ones. This is a new code path that needs validation.
New scripts: compute-e2e-platform-flags.cjs (extracted from inline bash), resolve-main-browserstack-apps.cjs, browserstack-app-validation.cjs, and run-compute-e2e-platform-flags.cjs — all new CI scripts that affect how tests are triggered.
Modified workflows: build-android-e2e.yml, build-ios-e2e.yml, run-performance-e2e.yml, ci.yml, get-requirements.yml all have significant logic changes around build reuse and native build gating.
Modified action: find-reusable-build/action.yml has new main-branch-only mode that changes how build artifacts are discovered.
Filter rules: New e2e_test_files and e2e_test_or_ignorable filter groups in filter-rules.yml.

Since these are CI workflow files (.github/workflows/ files are in the e2e_relevant_workflows filter), the e2e_workflowsCount would be > 0, which means the new logic itself triggers a full native build + E2E run (not the test-only reuse path). This is appropriate — we need to validate the pipeline changes work correctly with a full build.

All E2E smoke tags are selected because:

The CI pipeline changes affect ALL test execution paths
We need to verify the new build reuse logic doesn't break any test suite
The changes to how builds are fingerprinted, found, and reused could affect any test category
CI infrastructure changes warrant broad coverage to ensure no regressions in test execution

No performance tests are selected because no app code was changed — only CI infrastructure was modified, and performance tests measure app behavior, not CI pipeline behavior.

Performance Test Selection:
No app code was changed in this PR — all changes are CI/CD infrastructure (GitHub Actions workflows, scripts, filter rules, and documentation). Performance tests measure app responsiveness and render times, which are not affected by CI pipeline changes. No performance tags are warranted.

View GitHub Actions results

github-actions · 2026-07-13T14:17:59Z

🧪 Flaky unit test detection

Run history flaky detection

View recent run history

Historical failure rate is a hint, not proof — review each suggestion in context. See the flaky-test-detection skill for the full pattern reference and manual audit workflow.

Failures / runs sampled per window:

File	7d	15d	30d
`.github/scripts/browserstack-app-validation.test.ts`	0/50	0/178	0/375

AI-detected flaky patterns

`.github/scripts/browserstack-app-validation.test.ts`

J7 — Non-deterministic data: Date.now(), Math.random(), unstubbed network (medium)

The test uses Date.now() to generate a unique filename for the temporary output file. This introduces non-deterministic behavior (J7). Although the test does not assert on the timestamp value itself, following the pattern can lead to intermittent issues in CI if filenames collide or under high load. Historical data shows no failures for this file, but the pattern is present in the modified test.

Suggested fix in .github/scripts/browserstack-app-validation.test.ts:

-    const outputPath = path.join(os.tmpdir(), `gh-output-${Date.now()}.txt`);
-    writeGithubOutputs(outputPath, {
-      found: 'true',
-      'with-srp-browserstack-url': 'bs://abc123',
-    });
-
-    expect(fs.readFileSync(outputPath, 'utf8')).toBe(
-      'found=true\nwith-srp-browserstack-url=bs://abc123\n',
-    );
-    fs.unlinkSync(outputPath);
+    const outputPath = path.join(os.tmpdir(), 'gh-output-test.txt');
+    writeGithubOutputs(outputPath, {
+      found: 'true',
+      'with-srp-browserstack-url': 'bs://abc123',
+    });
+
+    expect(fs.readFileSync(outputPath, 'utf8')).toBe(
+      'found=true\nwith-srp-browserstack-url=bs://abc123\n',
+    );
+    fs.unlinkSync(outputPath);

This check is informational only and does not block merging.

codecov-commenter · 2026-07-13T14:34:06Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.28%. Comparing base (b8d3e80) to head (00e3436).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #33180      +/-   ##
==========================================
- Coverage   84.28%   84.28%   -0.01%     
==========================================
  Files        6064     6067       +3     
  Lines      161494   161636     +142     
  Branches    39385    39432      +47     
==========================================
+ Hits       136121   136234     +113     
- Misses      15995    16001       +6     
- Partials     9378     9401      +23

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sonarqubecloud · 2026-07-13T14:34:34Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

metamask-ci Bot added the team-qa QA team label Jul 13, 2026

github-actions Bot added the size-L label Jul 13, 2026

github-advanced-security AI found potential problems Jul 13, 2026

View reviewed changes

Comment thread .github/scripts/resolve-main-browserstack-apps.mjs Fixed

cursoragent and others added 4 commits July 13, 2026 09:58

docs(ci): note main HEAD fingerprint for test-only reuse

615857d

Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

javiergarciavera marked this pull request as ready for review July 13, 2026 11:20

javiergarciavera requested review from a team as code owners July 13, 2026 11:20

racitores previously approved these changes Jul 13, 2026

View reviewed changes

cursor Bot reviewed Jul 13, 2026

View reviewed changes

Comment thread .github/workflows/build-android-e2e.yml

github-actions Bot added the risk:medium AI analysis: medium risk label Jul 13, 2026

fix(ci): honor force-builds on test-only PR native-build gate

63b4f62

When reuse-main-builds-only is active, force-builds should compile fresh instead of failing because no main artifact was found. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

cursor Bot dismissed racitores’s stale review via 63b4f62 July 13, 2026 11:31

cursor Bot temporarily deployed to build-e2e July 13, 2026 11:57 Inactive

test: rename .github script tests to .test.ts for CI coverage

00e3436

test:unit only matches .github/**/*.test.ts; rename the two new browserstack and e2e-platform-flags tests so they run in CI. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

Qbandev approved these changes Jul 13, 2026

View reviewed changes

javiergarciavera enabled auto-merge July 13, 2026 15:45

tommasini approved these changes Jul 14, 2026

View reviewed changes

jvbriones approved these changes Jul 14, 2026

View reviewed changes

javiergarciavera added this pull request to the merge queue Jul 14, 2026

Merged via the queue into main with commit 4fde051 Jul 14, 2026
175 checks passed

javiergarciavera deleted the MMQA-1667-reduce-builds-amount branch July 14, 2026 13:31

github-actions Bot locked and limited conversation to collaborators Jul 14, 2026

metamask-ci Bot added the release-8.4.0 Issue or pull request that will be included in release 8.4.0 label Jul 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

ci(MMQA-1667): skip native builds when PR only changes E2E/performance tests#33180

ci(MMQA-1667): skip native builds when PR only changes E2E/performance tests#33180
javiergarciavera merged 8 commits into
mainfrom
MMQA-1667-reduce-builds-amount

javiergarciavera commented Jul 13, 2026 •

edited

Loading

Uh oh!

metamask-ci Bot commented Jul 13, 2026

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

codecov-commenter commented Jul 13, 2026

Uh oh!

sonarqubecloud Bot commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Uh oh!

Conversation

javiergarciavera commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What changed

Changelog

Related issues

Manual testing steps

Screenshots/Recordings

Before

After

Pre-merge author checklist

Performance checks (if applicable)

Pre-merge reviewer checklist

Uh oh!

metamask-ci Bot commented Jul 13, 2026

PR template — items to address before "Ready for review"

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

⚡ Performance Test Results

❌ Failed Tests (5)

Uh oh!

github-actions Bot commented Jul 13, 2026

🔍 Smart E2E Test Selection

Uh oh!

github-actions Bot commented Jul 13, 2026

🧪 Flaky unit test detection

Run history flaky detection

AI-detected flaky patterns

.github/scripts/browserstack-app-validation.test.ts

Uh oh!

codecov-commenter commented Jul 13, 2026

Codecov Report

Uh oh!

sonarqubecloud Bot commented Jul 13, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

javiergarciavera commented Jul 13, 2026 •

edited

Loading

`.github/scripts/browserstack-app-validation.test.ts`