fix(ci): resilient BrowserStack main reuse + fallback upload (#33277)

## **Description**

On test-only PRs, performance E2E reuses BrowserStack Android apps
uploaded from `main` instead of compiling fresh dual builds. The resolve
step was fragile: it only scanned BrowserStack `recent_apps` for IDs
like `MetaMask-Android-*-main-<run_id>`, so main apps that existed could
still be missed when the short recent list was flooded by other uploads.
When resolve failed hard, dual upload was skipped (`reuse_main_builds`)
and performance suites ran with no app.

This PR makes main BrowserStack reuse resilient:

1. Upload with a **stable per-branch `custom_id`**
(`MetaMask-Android-With-SRP-<slug>` / `…-Without-SRP-<slug>`, no
`run_id`) so BrowserStack overwrites the previous app for that ID.
2. Resolve via **`recent_apps/{custom_id}`** for the stable main IDs
first, then fall back to scanning the recent list for stable or legacy
`…-main-<run_id>` IDs.
3. If nothing is found, emit `found=false` and **fall back to a fresh
dual Android build/upload** instead of failing empty.
4. Quiet the unit-test J7 flaky-pattern flag by using `fs.mkdtempSync`
instead of `Date.now()` in the temp output path.

Follow-up to MMQA-1667 (#33180). Observed on verification PR #33270.

## **Changelog**

CHANGELOG entry: null

## **Related issues**

Fixes: MMQA-1667

## **Manual testing steps**

```gherkin
Feature: BrowserStack main-app resolve for test-only performance PRs

  Scenario: Resolve finds stable main custom_id
    Given main has uploaded Android apps with custom_id MetaMask-Android-*-main
    When a test-only PR runs performance E2E with reuse_main_builds=true
    Then resolve-main-browserstack-urls sets found=true
    And dual Android upload is skipped
    And performance tests use the resolved BrowserStack app URLs

  Scenario: Resolve miss falls back to fresh upload
    Given BrowserStack has no resolvable main custom_id
    When a test-only PR runs performance E2E with reuse_main_builds=true
    Then resolve-main-browserstack-urls sets found=false
    And Trigger Android Dual Versions runs and uploads new apps
` ` `

## **Screenshots/Recordings**

### **Before**

N/A — CI-only change, no UI affected.

### **After**

N/A — CI-only change, no UI affected.

## **Pre-merge author checklist**

- [x] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md).
- [x] I've completed the PR template to the best of my ability
- [x] I've included tests if applicable
- [x] I've documented my code using [JSDoc](https://jsdoc.app/) format if applicable
- [x] I've applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors.

#### Performance checks (if applicable)

- [x] I've tested on Android
  - Ideally on a mid-range device; emulator is acceptable
- [x] I've tested with a power user scenario
  - Use these [power-user SRPs](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/edit-v2/401401446401?draftShareId=9d77e1e1-4bdc-4be1-9ebb-ccd916988d93) to import wallets with many accounts and tokens
- [x] I've instrumented key operations with Sentry traces for production performance metrics
  - See [`trace()`](/app/util/trace.ts) for usage and [`addToken`](/app/components/Views/AddAsset/components/AddCustomToken/AddCustomToken.tsx#L274) for an example

## **Pre-merge reviewer checklist**

- [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
- [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.
```

---

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>

Author: javiergarciavera

.github/guidelines/E2E_DECISION_TREE.md

@@ -32,7 +32,7 @@ When a PR only changes E2E/performance test files (and other ignorable files), C
 
 The native build fingerprint for test-only PRs is computed from **`main` HEAD** (not the PR merge tree) so the lookup key matches completed `ci.yml` runs on `main`. Reuse tries GitHub Actions artifacts first, then the Cirrus `main` APK cache on Android.
 
-If `main` has new native-changing commits but its CI build has not finished yet, reuse lookup may miss — CI logs a warning and **falls back to a fresh native build** instead of failing the workflow. Performance E2E on test-only PRs resolves BrowserStack apps by `custom_id` prefix `MetaMask-Android-*-main-*` so only uploads from `main` are reused (not other branches or PRs).
+If `main` has new native-changing commits but its CI build has not finished yet, reuse lookup may miss — CI logs a warning and **falls back to a fresh native build** instead of failing the workflow. Performance E2E on test-only PRs resolves BrowserStack apps via stable main `custom_id`s (`MetaMask-Android-*-main`) first, then legacy `…-main-<run_id>` IDs; if none are found it **falls back to a fresh dual Android upload** instead of failing.
 
 This applies when all changed files match `e2e_test_files` or `e2e_ignorable` filters in `.github/rules/filter-rules.yml`, with at least one E2E test file changed, and no E2E-relevant workflow files were modified.
 

.github/scripts/browserstack-app-validation.cjs

@@ -6,14 +6,24 @@
 /** BrowserStack app URLs use the bs:// scheme with an opaque app hash. */
 const BROWSERSTACK_APP_URL_PATTERN = /^bs:\/\/[A-Za-z0-9]+$/;
 
+/**
+ * custom_id format after MMQA-1667:
+ * - Stable per branch: MetaMask-Android-With-SRP-<branchSlug>
+ * - Legacy with run id: MetaMask-Android-With-SRP-<branchSlug>-<runId>
+ * Slug must include at least one letter so pure-numeric pre-MMQA IDs are rejected.
+ */
 const WITH_SRP_CUSTOM_ID_PATTERN =
-  /^MetaMask-Android-With-SRP-[A-Za-z0-9._-]+-\d+$/;
+  /^MetaMask-Android-With-SRP-(?=.*[A-Za-z])[A-Za-z0-9._-]+$/;
 const WITHOUT_SRP_CUSTOM_ID_PATTERN =
-  /^MetaMask-Android-Without-SRP-[A-Za-z0-9._-]+-\d+$/;
+  /^MetaMask-Android-Without-SRP-(?=.*[A-Za-z])[A-Za-z0-9._-]+$/;
 
 const MAIN_BRANCH_BROWSERSTACK_SLUG = 'main';
-const MAIN_WITH_SRP_CUSTOM_ID_PREFIX = `MetaMask-Android-With-SRP-${MAIN_BRANCH_BROWSERSTACK_SLUG}-`;
-const MAIN_WITHOUT_SRP_CUSTOM_ID_PREFIX = `MetaMask-Android-Without-SRP-${MAIN_BRANCH_BROWSERSTACK_SLUG}-`;
+/** Stable custom_id used for main uploads (overwritten on each main upload). */
+const MAIN_WITH_SRP_CUSTOM_ID = `MetaMask-Android-With-SRP-${MAIN_BRANCH_BROWSERSTACK_SLUG}`;
+const MAIN_WITHOUT_SRP_CUSTOM_ID = `MetaMask-Android-Without-SRP-${MAIN_BRANCH_BROWSERSTACK_SLUG}`;
+/** Prefix for older main uploads that still embed github.run_id. */
+const MAIN_WITH_SRP_CUSTOM_ID_PREFIX = `${MAIN_WITH_SRP_CUSTOM_ID}-`;
+const MAIN_WITHOUT_SRP_CUSTOM_ID_PREFIX = `${MAIN_WITHOUT_SRP_CUSTOM_ID}-`;
 
 /**
  * @param {unknown} value
@@ -50,6 +60,30 @@ function assertBrowserStackCustomId(value, kind) {
   return assertMatchesPattern(value, pattern, `${kind} custom_id`);
 }
 
+/**
+ * @param {unknown} customId
+ * @param {'with-srp' | 'without-srp'} kind
+ * @returns {boolean}
+ */
+function isMainBranchBrowserStackCustomId(customId, kind) {
+  if (typeof customId !== 'string') {
+    return false;
+  }
+  const stable =
+    kind === 'with-srp' ? MAIN_WITH_SRP_CUSTOM_ID : MAIN_WITHOUT_SRP_CUSTOM_ID;
+  const legacyPrefix =
+    kind === 'with-srp'
+      ? MAIN_WITH_SRP_CUSTOM_ID_PREFIX
+      : MAIN_WITHOUT_SRP_CUSTOM_ID_PREFIX;
+  if (customId === stable) {
+    return true;
+  }
+  return (
+    customId.startsWith(legacyPrefix) &&
+    /^\d+$/.test(customId.slice(legacyPrefix.length))
+  );
+}
+
 /**
  * @param {string} path
  * @param {Record<string, string>} outputs
@@ -70,9 +104,12 @@ module.exports = {
   WITH_SRP_CUSTOM_ID_PATTERN,
   WITHOUT_SRP_CUSTOM_ID_PATTERN,
   MAIN_BRANCH_BROWSERSTACK_SLUG,
+  MAIN_WITH_SRP_CUSTOM_ID,
+  MAIN_WITHOUT_SRP_CUSTOM_ID,
   MAIN_WITH_SRP_CUSTOM_ID_PREFIX,
   MAIN_WITHOUT_SRP_CUSTOM_ID_PREFIX,
   assertBrowserStackAppUrl,
   assertBrowserStackCustomId,
+  isMainBranchBrowserStackCustomId,
   writeGithubOutputs,
 };

.github/scripts/browserstack-app-validation.test.ts

@@ -5,6 +5,7 @@ import path from 'node:path';
 const {
   assertBrowserStackAppUrl,
   assertBrowserStackCustomId,
+  isMainBranchBrowserStackCustomId,
   writeGithubOutputs,
 } = require('./browserstack-app-validation.cjs');
 
@@ -24,7 +25,10 @@ describe('browserstack-app-validation', () => {
     ).toThrow('Invalid test');
   });
 
-  it('accepts expected custom_id formats with branch slug', () => {
+  it('accepts stable and run-scoped custom_id formats with branch slug', () => {
+    expect(
+      assertBrowserStackCustomId('MetaMask-Android-With-SRP-main', 'with-srp'),
+    ).toBe('MetaMask-Android-With-SRP-main');
     expect(
       assertBrowserStackCustomId(
         'MetaMask-Android-With-SRP-main-123',
@@ -45,14 +49,42 @@ describe('browserstack-app-validation', () => {
     ).toBe('MetaMask-Android-With-SRP-feature_x-789');
   });
 
-  it('rejects legacy custom_id formats without branch slug', () => {
+  it('rejects legacy custom_id formats without a letterful branch slug', () => {
     expect(() =>
       assertBrowserStackCustomId('MetaMask-Android-With-SRP-123', 'with-srp'),
     ).toThrow('Invalid with-srp custom_id');
   });
 
+  it('detects main-branch custom_ids only', () => {
+    expect(
+      isMainBranchBrowserStackCustomId(
+        'MetaMask-Android-With-SRP-main',
+        'with-srp',
+      ),
+    ).toBe(true);
+    expect(
+      isMainBranchBrowserStackCustomId(
+        'MetaMask-Android-With-SRP-main-999',
+        'with-srp',
+      ),
+    ).toBe(true);
+    expect(
+      isMainBranchBrowserStackCustomId(
+        'MetaMask-Android-With-SRP-feature_x',
+        'with-srp',
+      ),
+    ).toBe(false);
+    expect(
+      isMainBranchBrowserStackCustomId(
+        'MetaMask-Android-With-SRP-maintenance-1',
+        'with-srp',
+      ),
+    ).toBe(false);
+  });
+
   it('writes only validated single-line GitHub outputs', () => {
-    const outputPath = path.join(os.tmpdir(), `gh-output-${Date.now()}.txt`);
+    const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gh-output-'));
+    const outputPath = path.join(tempDir, 'gh-output-test.txt');
     writeGithubOutputs(outputPath, {
       found: 'true',
       'with-srp-browserstack-url': 'bs://abc123',
@@ -61,6 +93,6 @@ describe('browserstack-app-validation', () => {
     expect(fs.readFileSync(outputPath, 'utf8')).toBe(
       'found=true\nwith-srp-browserstack-url=bs://abc123\n',
     );
-    fs.unlinkSync(outputPath);
+    fs.rmSync(tempDir, { recursive: true, force: true });
   });
 });

.github/scripts/resolve-main-browserstack-apps.cjs

@@ -2,13 +2,19 @@
 /**
  * Resolves the latest BrowserStack app URLs for performance E2E when a PR only
  * changes tests and should reuse main-branch builds instead of compiling fresh.
+ *
+ * Prefer BrowserStack's recent_apps/{custom_id} endpoint with the stable main
+ * custom_id. Fall back to scanning recent_apps for stable or legacy main IDs.
+ * When nothing is found, exits 0 with found=false so CI can fall back to a fresh
+ * dual Android upload.
  */
 
 const {
-  MAIN_WITH_SRP_CUSTOM_ID_PREFIX,
-  MAIN_WITHOUT_SRP_CUSTOM_ID_PREFIX,
+  MAIN_WITH_SRP_CUSTOM_ID,
+  MAIN_WITHOUT_SRP_CUSTOM_ID,
   assertBrowserStackAppUrl,
   assertBrowserStackCustomId,
+  isMainBranchBrowserStackCustomId,
   writeGithubOutputs,
 } = require('./browserstack-app-validation.cjs');
 
@@ -26,53 +32,62 @@ if (!username || !accessKey) {
   process.exit(1);
 }
 
-async function main() {
-  const auth = Buffer.from(`${username}:${accessKey}`).toString('base64');
+/**
+ * @param {string} auth
+ * @param {string} path
+ * @returns {Promise<{ ok: boolean; status: number; payload: unknown }>}
+ */
+async function browserStackGet(auth, path) {
   const response = await fetch(
-    'https://api-cloud.browserstack.com/app-automate/recent_apps',
+    `https://api-cloud.browserstack.com/app-automate/${path}`,
     {
       headers: {
         Authorization: `Basic ${auth}`,
       },
     },
   );
-
-  if (!response.ok) {
-    console.error(
-      `BrowserStack recent_apps request failed: ${response.status} ${response.statusText}`,
-    );
-    process.exit(1);
+  let payload = null;
+  try {
+    payload = await response.json();
+  } catch {
+    payload = null;
   }
+  return { ok: response.ok, status: response.status, payload };
+}
 
-  const payload = await response.json();
-  if (!Array.isArray(payload)) {
-    console.error('BrowserStack recent_apps response was not an array');
-    process.exit(1);
+/**
+ * @param {unknown} payload
+ * @returns {Array<{ custom_id?: string; app_url?: string; uploaded_at?: string }>}
+ */
+function asAppList(payload) {
+  if (Array.isArray(payload)) {
+    return payload;
   }
-
-  /** @type {Array<{ custom_id?: string; app_url?: string; uploaded_at?: string }>} */
-  const apps = payload;
-
-  function findLatestApp(prefix, kind) {
-    const matches = apps
-      .filter(
-        (app) =>
-          typeof app.custom_id === 'string' && app.custom_id.startsWith(prefix),
-      )
-      .filter(
-        (app) => typeof app.app_url === 'string' && app.app_url.length > 0,
-      )
-      .sort((left, right) => {
-        const leftTime = Date.parse(left.uploaded_at || '') || 0;
-        const rightTime = Date.parse(right.uploaded_at || '') || 0;
-        return rightTime - leftTime;
-      });
-
-    const candidate = matches[0];
-    if (!candidate) {
-      return null;
+  if (payload && typeof payload === 'object') {
+    const maybeApps = /** @type {{ apps?: unknown }} */ (payload).apps;
+    if (Array.isArray(maybeApps)) {
+      return maybeApps;
     }
+  }
+  return [];
+}
 
+/**
+ * @param {Array<{ custom_id?: string; app_url?: string; uploaded_at?: string }>} apps
+ * @param {'with-srp' | 'without-srp'} kind
+ * @returns {{ customId: string; appUrl: string } | null}
+ */
+function pickLatestValidApp(apps, kind) {
+  const matches = apps
+    .filter((app) => isMainBranchBrowserStackCustomId(app.custom_id, kind))
+    .filter((app) => typeof app.app_url === 'string' && app.app_url.length > 0)
+    .sort((left, right) => {
+      const leftTime = Date.parse(left.uploaded_at || '') || 0;
+      const rightTime = Date.parse(right.uploaded_at || '') || 0;
+      return rightTime - leftTime;
+    });
+
+  for (const candidate of matches) {
     try {
       return {
         customId: assertBrowserStackCustomId(candidate.custom_id, kind),
@@ -85,24 +100,103 @@ async function main() {
       console.error(
         `Rejected BrowserStack ${kind} candidate ${candidate.custom_id || 'unknown'}: ${error.message}`,
       );
-      return null;
     }
   }
 
-  const withSrp = findLatestApp(MAIN_WITH_SRP_CUSTOM_ID_PREFIX, 'with-srp');
-  const withoutSrp = findLatestApp(
-    MAIN_WITHOUT_SRP_CUSTOM_ID_PREFIX,
+  return null;
+}
+
+/**
+ * @param {string} auth
+ * @param {string} customId
+ * @param {'with-srp' | 'without-srp'} kind
+ * @returns {Promise<{ customId: string; appUrl: string } | null>}
+ */
+async function resolveByCustomId(auth, customId, kind) {
+  const encodedId = encodeURIComponent(customId);
+  const { ok, status, payload } = await browserStackGet(
+    auth,
+    `recent_apps/${encodedId}`,
+  );
+
+  if (!ok) {
+    console.warn(
+      `BrowserStack recent_apps/${customId} returned ${status}; will try recent_apps list fallback.`,
+    );
+    return null;
+  }
+
+  const picked = pickLatestValidApp(asAppList(payload), kind);
+  if (picked) {
+    console.log(`Resolved ${kind} via recent_apps/${customId}: ${picked.customId}`);
+  }
+  return picked;
+}
+
+/**
+ * @param {string} auth
+ * @param {'with-srp' | 'without-srp'} kind
+ * @returns {Promise<{ customId: string; appUrl: string } | null>}
+ */
+async function resolveFromRecentAppsList(auth, kind) {
+  const { ok, status, payload } = await browserStackGet(auth, 'recent_apps');
+  if (!ok) {
+    console.warn(
+      `BrowserStack recent_apps list returned ${status}; cannot fall back for ${kind}.`,
+    );
+    return null;
+  }
+
+  const apps = asAppList(payload);
+  if (!Array.isArray(payload) && apps.length === 0) {
+    console.warn('BrowserStack recent_apps response was not a usable app list');
+    return null;
+  }
+
+  const picked = pickLatestValidApp(apps, kind);
+  if (picked) {
+    console.log(
+      `Resolved ${kind} via recent_apps list scan: ${picked.customId}`,
+    );
+  }
+  return picked;
+}
+
+async function main() {
+  const auth = Buffer.from(`${username}:${accessKey}`).toString('base64');
+
+  let withSrp = await resolveByCustomId(
+    auth,
+    MAIN_WITH_SRP_CUSTOM_ID,
+    'with-srp',
+  );
+  if (!withSrp) {
+    withSrp = await resolveFromRecentAppsList(auth, 'with-srp');
+  }
+
+  let withoutSrp = await resolveByCustomId(
+    auth,
+    MAIN_WITHOUT_SRP_CUSTOM_ID,
     'without-srp',
   );
+  if (!withoutSrp) {
+    withoutSrp = await resolveFromRecentAppsList(auth, 'without-srp');
+  }
 
   if (!withSrp || !withoutSrp) {
-    console.error(
+    console.warn(
       'Could not resolve latest BrowserStack Android apps for main-build reuse.',
     );
-    console.error(
+    console.warn(
       `Found with-SRP=${withSrp?.customId || 'none'}, without-SRP=${withoutSrp?.customId || 'none'}`,
     );
-    process.exit(1);
+    console.warn(
+      'Emitting found=false so CI can fall back to a fresh Android dual upload.',
+    );
+    writeGithubOutputs(githubOutputPath, {
+      found: 'false',
+    });
+    return;
   }
 
   console.log(`Reusing BrowserStack with-SRP app: ${withSrp.customId}`);