Skip to content

feat: add script to update default E2E fixture from generated report#26861

Merged
cmd-ob merged 25 commits into
mainfrom
fixture-update
Mar 6, 2026
Merged

feat: add script to update default E2E fixture from generated report#26861
cmd-ob merged 25 commits into
mainfrom
fixture-update

Conversation

@cmd-ob
Copy link
Copy Markdown
Contributor

@cmd-ob cmd-ob commented Mar 3, 2026

Description

This PR improves the reliability and security of the automated E2E fixture update workflow by fixing
race conditions, permission issues, and fork-handling logic.

Key changes:

update-e2e-fixtures.yml

  • Fix: prevent stale fixture caching on cancelled jobs — adds if: ${{ !cancelled() }} guard to the
    cache step so partial/failed runs don't overwrite valid cached fixtures
  • Fix: commit-fixtures job now handles upstream cancellations — adds !cancelled() to the job condition
    so it can still commit even when update-fixtures was cancelled
  • Fix: replace ACTIONS_WRITE_TOKEN with GITHUB_TOKEN — uses the built-in token with explicit contents:
    write and pull-requests: write permissions instead of a PAT, removing the dependency on the org
    secret
  • Add explicit permissions to commit-fixtures and check-status jobs

ci.yml

  • Fix: fork exclusion on all E2E jobs — moves !github.event.pull_request.head.repo.fork to the top of
    each condition so forks are excluded before any build/change checks
  • Fix: ai_confidence string-to-number comparison — wraps comparison in fromJSON(... || '0') to prevent
    string comparison bugs
  • Feat: force_run flag — propagates force_run output from smart-e2e-selection so E2E jobs can be
    manually forced regardless of file-change detection
  • Adds smart-e2e-selection as a needs dependency where missing (ios-tests-ready,
    validate-e2e-fixtures)

scripts/update-e2e-fixture.sh

  • Adds a helper script to update the default fixture from a generated E2E report

Default fixture (tests/framework/fixtures/json/default-fixture.json)

  • Updated fixture state to reflect current expected app state

Changelog

CHANGELOG entry:

Related issues

Fixes:

Manual testing steps

Feature: my feature name

  Scenario: user [verb for user action]
    Given [describe expected initial app state]

    When user [verb for user action]
    Then [describe expected outcome]

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

Note

Medium Risk
Medium risk because it changes CI gating/conditions for when E2E builds and fixture workflows run, and adjusts GitHub Action permissions/tokens for committing/commenting, which could impact test coverage or workflow execution if misconfigured.

Overview
Adds a manual override to always run E2E. The smart-e2e-selection composite action now outputs force_run when the skip-smart-e2e-selection label is present, and ci.yml uses this flag to trigger Android/iOS E2E builds/tests (and fixture validation) even when path filters say nothing changed; confidence parsing was also hardened via fromJSON(... || '0').

Tightens and streamlines E2E fixture updates/validation. The fixture-validation Detox spec now throws (failing CI) when the committed fixture is out of date and includes clearer remediation instructions; ignored keys were expanded for several runtime-added network configs, and a new unit test asserts unexpected new keys are detected. The Update E2E Fixtures workflow now avoids cache/commit steps when cancelled, sets explicit contents/pull-requests permissions, and uses GITHUB_TOKEN for PR checkout/comments; a new scripts/update-e2e-fixture.sh helps copy the generated fixture report into the committed fixture locally.

Written by Cursor Bugbot for commit afc3f5f. This will update automatically on new commits. Configure here.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 3, 2026

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@metamaskbot metamaskbot added the team-qa QA team label Mar 3, 2026
@github-actions github-actions Bot added the size-S label Mar 3, 2026
@cmd-ob cmd-ob added skip-smart-e2e-selection Skip Smart E2E selection, i.e. select all E2E tests to run no-changelog no-changelog Indicates no external facing user changes, therefore no changelog documentation needed no changelog required No changelog entry is required for this change and removed size-S skip-smart-e2e-selection Skip Smart E2E selection, i.e. select all E2E tests to run labels Mar 3, 2026
@cmd-ob cmd-ob requested a review from a team as a code owner March 3, 2026 07:57
@github-actions github-actions Bot added the size-S label Mar 3, 2026
@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

@github-actions github-actions Bot added size-M and removed size-S labels Mar 3, 2026
@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

3 similar comments
@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

2 similar comments
@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

@cmd-ob
Copy link
Copy Markdown
Contributor Author

cmd-ob commented Mar 3, 2026

@metamaskbot update-mobile-fixture

Comment thread .github/workflows/update-e2e-fixtures.yml Outdated
@github-actions github-actions Bot added the size-S label Mar 4, 2026
Comment thread .github/workflows/update-e2e-fixtures.yml
@github-actions github-actions Bot added size-M and removed size-S labels Mar 4, 2026
Comment thread tests/regression/fixtures/fixture-validation.spec.ts
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.50%. Comparing base (d67656d) to head (838d145).
⚠️ Report is 10 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26861      +/-   ##
==========================================
- Coverage   81.51%   81.50%   -0.01%     
==========================================
  Files        4627     4631       +4     
  Lines      121093   121816     +723     
  Branches    26453    26647     +194     
==========================================
+ Hits        98706    99288     +582     
- Misses      15480    15571      +91     
- Partials     6907     6957      +50     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Comment thread .github/workflows/update-e2e-fixtures.yml
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 6, 2026

🔍 Smart E2E Test Selection

⏭️ Smart E2E selection skipped - skip-smart-e2e-selection label found

All E2E tests pre-selected.

View GitHub Actions results

@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud Bot commented Mar 6, 2026

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 6, 2026

E2E Fixture Validation — Schema is up to date
6 value mismatches detected (expected — fixture represents an existing user).
View details

@cmd-ob cmd-ob enabled auto-merge March 6, 2026 15:18
@cmd-ob cmd-ob added this pull request to the merge queue Mar 6, 2026
Merged via the queue into main with commit 88da2c0 Mar 6, 2026
112 checks passed
@cmd-ob cmd-ob deleted the fixture-update branch March 6, 2026 16:01
@github-actions github-actions Bot locked and limited conversation to collaborators Mar 6, 2026
@metamaskbot metamaskbot added the release-7.70.0 Issue or pull request that will be included in release 7.70.0 label Mar 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

no changelog required No changelog entry is required for this change no-changelog no-changelog Indicates no external facing user changes, therefore no changelog documentation needed release-7.70.0 Issue or pull request that will be included in release 7.70.0 size-M skip-smart-e2e-selection Skip Smart E2E selection, i.e. select all E2E tests to run team-qa QA team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants