Prevent storage E2E retries from canceling CI - #4249
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adjusts Playwright CI sharding so storage-sensitive E2E tests run in a dedicated single-worker lane, allowing retries to re-run only failed tests (instead of replaying full serial groups) and reducing CI cancellations caused by late failures.
Changes:
- Adds CI test-group routing (
regularvsstorage) viaPLAYWRIGHT_TEST_GROUP, withworkers: 1for the storage lane. - Marks default-storage tests with
@storageand switches storage suites fromserialto Playwright’s default mode. - Updates the GitHub Actions matrix to add one storage lane per browser and to suffix storage artifacts.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| packages/playground/website/playwright/playwright.ci.config.ts | Adds PLAYWRIGHT_TEST_GROUP-based grep/grepInvert and workers: 1 to separate storage vs regular runs. |
| packages/playground/website/playwright/e2e/website-ui.spec.ts | Tags the default-storage suite as @storage and switches it off serial. |
| packages/playground/website/playwright/e2e/opfs.spec.ts | Switches OPFS tests from serial to default mode and updates comments. |
| .github/workflows/ci.yml | Adds per-browser storage matrix entries, passes PLAYWRIGHT_TEST_GROUP, and updates step/artifact naming. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // OPFS is browser-scoped, so `@storage` routes this suite to the one-worker CI lane. | ||
| test.describe('OPFS', { tag: '@storage' }, () => { | ||
| // Default mode retries only the failed test instead of replaying the suite. | ||
| test.describe.configure({ mode: 'default' }); |
There was a problem hiding this comment.
I think all the changes in the file from here down are the result of whitespace-only changes (indentation) that led to formatting changes.
|
This looks good to go. Let's merge. |
Why this change is needed
@adamziel reported that flaky end-to-end tests were causing more CI jobs to be
canceled.
The canceled jobs shared the same pattern: one storage test failed late in a
Playwright serial group, so Playwright retried the whole group from its first
test. In one case, a 33-test storage group restarted more than once and the job
reached the existing 30-minute limit before it could finish. The OPFS tests had
the same retry behavior.
Adding another ordinary shard would not help because Playwright keeps a serial
group together. Increasing the timeout or reducing the three retries would
also leave the whole-group replay in place.
Examples:
Why this approach
The storage tests must not run at the same time because they share browser
storage. A dedicated storage lane with one worker preserves that isolation.
Using Playwright's default test mode inside that lane means a failure retries
only the failed test instead of replaying every earlier test in its group.
This directly addresses the cancellation pattern while keeping the existing
30-minute job timeout and three retries.
What this changes
mode, allowing Playwright to retry one failed test.
@storageto the top-level OPFS suite and the default-storage group.CI routes tests only by this tag, so it does not need file-path rules.
workers: 1.normal three workers, then storage tests use one worker. Missing or unsupported
group values produce a clear error.
storagetest group. Entrieswithout a group use the ordinary-test behavior, so the matrix does not repeat
a
regularlabel nine times.-storagesuffix only to storage artifacts. Ordinary artifact namesremain unchanged.
The tradeoff is three additional install/build runners, one per browser. This
is the same job-count increase as changing from three to four ordinary shards,
but it removes the whole-group retry amplification that caused the cancellations.
Verification
each test is in exactly one lane: 182 ordinary tests and 59 storage tests.
The ordinary shards contain 61, 61, and 60 tests.
groups, and a missing or unsupported group exits with a clear error.
attempts passed. Two attempts had one flaky ZIP-import test, and Playwright
retried only that test instead of restarting the storage suite. This confirms
the new retry scope works as intended.
the
storagegroup.validation.
The repetitions also exposed a separate race in two ZIP-import tests. PR #4213
addresses that underlying race. It remains separate so this PR stays focused on
making retries safe and preventing CI cancellations.