Skip to content

Prevent storage E2E retries from canceling CI #4249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brandonpayton merged 7 commits into from
Aug 6, 2026
Merged

Prevent storage E2E retries from canceling CI #4249

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c0f1bad
Run storage E2E tests in a one-worker CI lane
brandonpayton Aug 5, 2026
7e4df7e
Simplify Playwright CI matrix labels
brandonpayton Aug 6, 2026
1ff061d
Route storage E2E tests by tag
brandonpayton Aug 6, 2026
bcb508e
Make Playwright CI routing explicit
brandonpayton Aug 6, 2026
e14ae80
Align local storage E2E runs with CI
brandonpayton Aug 6, 2026
97ea380
Clarify storage E2E routing
brandonpayton Aug 6, 2026
e8d2103
Tag OPFS tests at the suite level
brandonpayton Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 15 additions & 7 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,8 +277,10 @@ jobs:
strategy:
fail-fast: false
matrix:
# 3 shards per browser for balanced parallel execution
# (4 shards caused uneven distribution due to serial test files)
# Keep three balanced shards for non-storage tests. Storage tests
# have their own unsharded lane because OPFS is browser-scoped.
# That lane uses one worker, so Playwright can retry only the
# failed test without allowing storage tests to overlap.
include:
- browser: 'chromium'
shard: '1/3'
Expand Down Expand Up @@ -307,6 +309,12 @@ jobs:
- browser: 'webkit'
shard: '3/3'
shard_name: '3-of-3'
- browser: 'chromium'
test_group: 'storage'
- browser: 'firefox'
test_group: 'storage'
- browser: 'webkit'
test_group: 'storage'
steps:
- name: Free up runner disk space
shell: bash
Expand All @@ -329,27 +337,27 @@ jobs:
run: sudo npx playwright install ${{ matrix.browser }} --with-deps
- name: Build app for E2E tests
run: CORS_PROXY_URL=http://127.0.0.1:5263/cors-proxy.php? npx nx e2e:playwright:prepare-app-deploy-and-offline-mode playground-website
- name: Run Playwright tests - ${{ matrix.browser }}${{ matrix.shard && format(' (shard {0})', matrix.shard) || '' }}
- name: Run Playwright${{ matrix.test_group && format(' {0}', matrix.test_group) || '' }} tests - ${{ matrix.browser }}${{ matrix.shard && format(' (shard {0})', matrix.shard) || '' }}
run: |
SHARD_ARG=""
if [ -n "${{ matrix.shard }}" ]; then
SHARD_ARG="--shard=${{ matrix.shard }}"
fi
if [ "${{ matrix.browser }}" = "firefox" ]; then
sudo -E HOME=/root XDG_RUNTIME_DIR=/root CI=true npx playwright test --config=packages/playground/website/playwright/playwright.ci.config.ts --project=${{ matrix.browser }} $SHARD_ARG
sudo -E HOME=/root XDG_RUNTIME_DIR=/root CI=true PLAYWRIGHT_TEST_GROUP=${{ matrix.test_group || 'regular' }} npx playwright test --config=packages/playground/website/playwright/playwright.ci.config.ts --project=${{ matrix.browser }} $SHARD_ARG
else
sudo CI=true npx playwright test --config=packages/playground/website/playwright/playwright.ci.config.ts --project=${{ matrix.browser }} $SHARD_ARG
sudo CI=true PLAYWRIGHT_TEST_GROUP=${{ matrix.test_group || 'regular' }} npx playwright test --config=packages/playground/website/playwright/playwright.ci.config.ts --project=${{ matrix.browser }} $SHARD_ARG
fi
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report-${{ matrix.browser }}${{ matrix.shard_name && format('-shard-{0}', matrix.shard_name) || '' }}
name: playwright-report-${{ matrix.browser }}${{ matrix.test_group && format('-{0}', matrix.test_group) || '' }}${{ matrix.shard_name && format('-shard-{0}', matrix.shard_name) || '' }}
path: packages/playground/website/playwright-report/
if-no-files-found: ignore
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-snapshots-${{ matrix.browser }}${{ matrix.shard_name && format('-shard-{0}', matrix.shard_name) || '' }}
name: playwright-snapshots-${{ matrix.browser }}${{ matrix.test_group && format('-{0}', matrix.test_group) || '' }}${{ matrix.shard_name && format('-shard-{0}', matrix.shard_name) || '' }}
path: packages/playground/website/playwright/e2e/deployment.spec.ts-snapshots/
if-no-files-found: ignore

Expand Down
3 changes: 2 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,8 @@ npx nx build <package-name> # Build specific package
# Testing
npm test # Run all tests
npx nx test <package-name> # Test specific package
npx nx e2e playground-website # Run end-to-end tests
npx nx e2e playground-website # Run website Cypress E2E tests
npx nx run playground-website:e2e:playwright:ci # Run website Playwright E2E tests

# Running a single test file
npx nx test <package-name> --testFile=<test-file-name>
Expand Down
101 changes: 66 additions & 35 deletions packages/playground/website/playwright/e2e/opfs.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,16 +127,16 @@ Theme Name: Close Race Theme
return Buffer.from(zipBytes!);
}

// OPFS tests must run serially because OPFS storage is shared at the browser
// level, so tests would interfere with each other's saved sites if run in parallel.
test.describe.configure({ mode: 'serial' });
// OPFS is shared by tests in one browser. Default mode keeps this file ordered
// while retrying only the failed test.
test.describe.configure({ mode: 'default' });
Comment thread
brandonpayton marked this conversation as resolved.
Outdated

/**
* Returns a URL that opts this test out of default browser storage.
*
* `storage=temp` is what makes the site temporary. The random value keeps
* repeated navigations from reusing a temporary site created earlier in this
* serial OPFS test file.
* ordered OPFS test file.
*/
function getTemporaryPlaygroundUrl(hash = '') {
return `./?storage=temp&random=${Math.random().toString(36).slice(2)}${hash}`;
Expand Down Expand Up @@ -534,7 +534,8 @@ async function openPlaygroundPath(page: Page, path: string) {
path
);
}
test('should retry pending OPFS cleanup after another tab releases storage', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should retry pending OPFS cleanup after another tab releases storage', async ({
website,
context,
browserName,
Expand Down Expand Up @@ -594,7 +595,8 @@ test('should retry pending OPFS cleanup after another tab releases storage', asy
expect(storedSite.hasOldResetSentinel).toBe(false);
});

test('should start a new Playground after an initial OPFS sync was interrupted', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should start a new Playground after an initial OPFS sync was interrupted', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -630,7 +632,11 @@ test('should start a new Playground after an initial OPFS sync was interrupted',
await website.waitForNestedIframes();
});

test('should switch between sites', async ({ website, browserName }) => {
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should switch between sites', async ({
website,
browserName,
}) => {
test.skip(
browserName !== 'chromium',
`This test relies on OPFS which isn't available in Playwright's flavor of ${browserName}.`
Expand Down Expand Up @@ -693,7 +699,8 @@ test('should switch between sites', async ({ website, browserName }) => {
await expect(getPlaygroundTitle(website.page)).toContainText(firstSiteName);
});

test('should preserve PHP constants when saving a temporary site to OPFS', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should preserve PHP constants when saving a temporary site to OPFS', async ({
website,
browserName,
wordpress,
Expand Down Expand Up @@ -760,7 +767,8 @@ test('should preserve PHP constants when saving a temporary site to OPFS', async
await expect(wordpress.locator('body')).toContainText('E2E_TEST_VALUE');
});

test('should rename a saved Playground and persist after reload', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should rename a saved Playground and persist after reload', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -813,7 +821,8 @@ test('should rename a saved Playground and persist after reload', async ({
await website.closePlaygroundsPane();
});

test('should wait for a temporary OPFS metadata lock', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should wait for a temporary OPFS metadata lock', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -875,7 +884,8 @@ test('should wait for a temporary OPFS metadata lock', async ({
.toBe(newName);
});

test('should preserve metadata changes made in different tabs', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should preserve metadata changes made in different tabs', async ({
website,
context,
browserName,
Expand Down Expand Up @@ -931,7 +941,8 @@ test('should preserve metadata changes made in different tabs', async ({
expect(persistedMetadata.runtimeConfiguration.networking).toBe(false);
});

test('should show the Store permanently pane with the save controls', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should show the Store permanently pane with the save controls', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -963,7 +974,8 @@ test('should show the Store permanently pane with the save controls', async ({
await expect(pane).not.toBeVisible();
});

test('should close the Store permanently pane without saving', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should close the Store permanently pane without saving', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -998,7 +1010,8 @@ test('should close the Store permanently pane without saving', async ({
);
});

test('should have playground name input text selected by default', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should have playground name input text selected by default', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -1034,7 +1047,11 @@ test('should have playground name input text selected by default', async ({
await pane.getByRole('button', { name: 'Cancel' }).click();
});

test('should save site with custom name', async ({ website, browserName }) => {
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should save site with custom name', async ({
website,
browserName,
}) => {
test.skip(
browserName !== 'chromium',
`This test relies on OPFS which isn't available in Playwright's flavor of ${browserName}.`
Expand All @@ -1056,12 +1073,15 @@ test('should save site with custom name', async ({ website, browserName }) => {
// Verify the name also appears in the Playgrounds pane.
await website.openPlaygroundsPane();
await expect(
website.page.locator('[class*="siteRowName"]', { hasText: customName })
website.page.locator('[class*="siteRowName"]', {
hasText: customName,
})
).toBeVisible();
await website.closePlaygroundsPane();
});

test('should not persist the Store permanently pane through page refresh', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should not persist the Store permanently pane through page refresh', async ({
website,
browserName,
}) => {
Expand All @@ -1085,7 +1105,8 @@ test('should not persist the Store permanently pane through page refresh', async
await expect(pane).toHaveCount(0);
});

test('should display OPFS storage option as selected by default', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should display OPFS storage option as selected by default', async ({
website,
browserName,
}) => {
Expand All @@ -1108,7 +1129,8 @@ test('should display OPFS storage option as selected by default', async ({
await pane.getByRole('button', { name: 'Cancel' }).click();
});

test('should import ZIP into a fresh temporary site without browser storage', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should import ZIP into a fresh temporary site without browser storage', async ({
website,
browserName,
context,
Expand Down Expand Up @@ -1176,7 +1198,8 @@ test('should import ZIP into a fresh temporary site without browser storage', as
.toBe(marker);
});

test('should remove the saved site created for a failed ZIP import', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should remove the saved site created for a failed ZIP import', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -1220,7 +1243,8 @@ test('should remove the saved site created for a failed ZIP import', async ({
.toEqual(storedSiteSlugsBeforeImport);
});

test('should import a ZIP dropped on the page', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should import a ZIP dropped on the page', async ({
website,
wordpress,
browserName,
Expand Down Expand Up @@ -1256,7 +1280,8 @@ test('should import a ZIP dropped on the page', async ({
await expect(wordpress.locator('body')).toContainText(marker);
});

test('should show an inline error for a non-ZIP drop', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should show an inline error for a non-ZIP drop', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -1284,7 +1309,8 @@ test('should show an inline error for a non-ZIP drop', async ({
).toBeVisible();
});

test('should notify when a ZIP import loads before autosave finishes', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should notify when a ZIP import loads before autosave finishes', async ({
website,
browserName,
}) => {
Expand Down Expand Up @@ -1404,7 +1430,8 @@ test('should notify when a ZIP import loads before autosave finishes', async ({
.toEqual({ plugin: true, theme: true });
});

test('should import ZIP into a new saved site when a saved site exists', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should import ZIP into a new saved site when a saved site exists', async ({
website,
wordpress,
browserName,
Expand Down Expand Up @@ -1496,7 +1523,8 @@ test('should import ZIP into a new saved site when a saved site exists', async (
);
});

test('should create a saved site when importing ZIP while on a saved site with no existing temporary site', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should create a saved site when importing ZIP while on a saved site with no existing temporary site', async ({
website,
wordpress,
browserName,
Expand Down Expand Up @@ -1599,7 +1627,8 @@ test('should create a saved site when importing ZIP while on a saved site with n
);
});

test('should persist an imported ZIP saved site after switching away and back', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should persist an imported ZIP saved site after switching away and back', async ({
website,
wordpress,
browserName,
Expand Down Expand Up @@ -1689,7 +1718,8 @@ test('should persist an imported ZIP saved site after switching away and back',
await expect(wordpress.locator('body')).toContainText(importedMarker);
});

test('should retain files omitted from a legacy ZIP export', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should retain files omitted from a legacy ZIP export', async ({
website,
wordpress,
browserName,
Expand Down Expand Up @@ -1793,7 +1823,8 @@ test('should retain files omitted from a legacy ZIP export', async ({
);
});

test('should re-import an exported ZIP without switching sites', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should re-import an exported ZIP without switching sites', async ({
website,
context,
browserName,
Expand Down Expand Up @@ -1894,7 +1925,8 @@ test('should re-import an exported ZIP without switching sites', async ({
expect(importedSite.slug).not.toBe(occupiedSiteSlug);
});

test('should preserve a customized default-theme background through export and import', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should preserve a customized default-theme background through export and import', async ({
website,
wordpress,
browserName,
Expand Down Expand Up @@ -2024,12 +2056,10 @@ PHP;
);
});

// Missing site modal tests in a separate describe block to avoid state pollution
// Missing site modal tests in a separate describe block to avoid state pollution.
test.describe('Missing site modal', () => {
// These tests also need serial mode since they use OPFS
test.describe.configure({ mode: 'serial' });

test('should show modal when loading non-existent site slug', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should show modal when loading non-existent site slug', async ({
website,
wordpress,
browserName,
Expand All @@ -2056,7 +2086,8 @@ test.describe('Missing site modal', () => {
).toBeVisible({ timeout: 30000 });
});

test('should dismiss modal when clicking dismiss button', async ({
// `@storage` routes this browser-scoped test to the one-worker CI lane.
test('@storage should dismiss modal when clicking dismiss button', async ({
website,
wordpress,
browserName,
Expand Down
6 changes: 4 additions & 2 deletions packages/playground/website/playwright/e2e/website-ui.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1265,8 +1265,10 @@ test.describe('Database panel', () => {
});

// Test browser-saved Playgrounds by default and explicit temporary opt-outs.
test.describe('Default Playground storage', () => {
test.describe.configure({ mode: 'serial' });
// The `@storage` tag routes this group to the one-worker CI storage lane.
test.describe('Default Playground storage', { tag: '@storage' }, () => {
// Default mode prevents storage overlap while retrying only the failed test.
test.describe.configure({ mode: 'default' });

test('should create and finish autosaving a Playground from the root URL', async ({
website,
Expand Down
Loading
Loading