Merge pull request #18009 from ethereum/feat/playwright-chromatic-page-visual-tests

test(visual): add Playwright + Chromatic page visual regression suite

Author: pettinarip

.claude/skills/page-visual-tests/SKILL.md

@@ -0,0 +1,73 @@
+---
+name: page-visual-tests
+description: Playwright + Chromatic full-page visual tests for ethereum.org. Trigger on "add a page to the visual suite", "the snapshot keeps changing", "chromatic pages", "chromatic playwright", or edits to `tests/visual/`, `playwright.visual.config.ts`, or `.github/workflows/chromatic-pages.yml`. Skip for Storybook Chromatic (`chromatic.yml`), e2e (`tests/e2e/`), unit (`tests/unit/`).
+---
+
+# Page Visual Tests (Playwright + Chromatic)
+
+This repo has two Chromatic projects: Storybook (`chromatic.yml` + `pnpm chromatic`) and **page visual tests** (`chromatic-pages.yml` + `pnpm chromatic:pages`). This skill is the second one only.
+
+The Playwright suite captures DOM archives (not PNGs) per page × viewport; Chromatic re-renders them in the cloud to diff. A green local `pnpm test:visual` just means archives were produced — the diff happens after upload.
+
+## Files that matter
+
+- `playwright.visual.config.ts` — visual-only config (3 viewports + `webServer`)
+- `playwright.config.ts` — base (e2e + unit; **no `webServer`**)
+- `tests/visual/pages.spec.ts` — page list + readiness pattern
+- `.github/workflows/chromatic-pages.yml` — CI
+- `src/components/ui/skeleton.tsx`, `src/components/ui/spinner.tsx` — loading primitives
+- `package.json` scripts: `test:visual*`, `chromatic:pages`
+
+## Non-obvious constraints
+
+**Dual Playwright config.** `webServer` lives only in `playwright.visual.config.ts`. Moving it into the base config breaks `pnpm test:unit` and `pnpm test:e2e` in CI — they try to start Next against a missing `.next` build.
+
+**Desktop viewport is 1024, not 1280.** Chromatic caps snapshots at `width × height ≤ 25M` px. The tallest tested pages reach ~22.5k px; 1280 overflows, 1024 fits. Measure `document.documentElement.scrollHeight` before raising the viewport or adding a long page.
+
+**Loading contract: `data-slot="loading"`.** The shared `Skeleton` and `Spinner` primitives carry this attribute. Each test waits until `document.querySelectorAll('[data-slot="loading"]').length === 0` before snapshotting. Any bespoke loader — raw `animate-pulse-light`, a local Skeleton copy, a custom spinner — is invisible to the wait and will silently flake. Fix by routing through the shared primitive or adding `data-slot="loading"` to the bespoke loader's root.
+
+**Imports come from `@chromatic-com/playwright`, not `@playwright/test`.** The two packages re-export `expect` with skewed types, so `expect(...).toHaveCount(0)` misbehaves — prefer `page.waitForFunction` for the loading wait.
+
+**Environment.** `USE_MOCK_DATA=true` and `NEXT_PUBLIC_BUILD_LOCALES=en` are required at build and test time. Paths in the spec are unprefixed (`/wallets/`, not `/en/wallets/`) because `localePrefix: "as-needed"` serves English at the root — adding `/en` would just trigger a redirect.
+
+**Random ordering: `maybeShuffle`.** Lodash `shuffle` and `.sort(() => Math.random() - 0.5)` flake snapshots independently of loaders. Wrap them with `maybeShuffle` from `src/lib/utils/random.ts` — it returns the list unchanged when `IS_VISUAL_TEST=true`. Current call sites: `wallets.ts`, `apps.ts` (Highlights/Discover/AppOfTheWeek), `useStakingProductsCardGrid.ts`. The env var is exposed to the client bundle via `next.config.js`'s `env` block; without that, `process.env.IS_VISUAL_TEST` evaluates to `undefined` in client components and the shuffle still runs.
+
+**Use `domcontentloaded`, not `networkidle`.** Analytics and background fetches keep the network perpetually busy.
+
+## Canonical test
+
+```ts
+import { takeSnapshot, test } from "@chromatic-com/playwright"
+
+const pages: Array<{ name: string; path: string }> = [
+  { name: "Homepage", path: "/" },
+  { name: "Docs - Smart Contracts", path: "/developers/docs/smart-contracts/" },
+  // ...
+]
+
+test.describe("Page Visual Tests", () => {
+  for (const { name, path } of pages) {
+    test(name, async ({ page }, testInfo) => {
+      await page.goto(path, { waitUntil: "domcontentloaded" })
+      await page.waitForFunction(
+        () => document.querySelectorAll('[data-slot="loading"]').length === 0
+      )
+      await takeSnapshot(page, testInfo)
+    })
+  }
+})
+```
+
+## Common situations
+
+**Adding a page.** Each entry costs three snapshots (one per viewport) against Chromatic's budget, so check whether the page's layout (under `src/layouts/`) is already covered before adding. Scan the page subtree for bespoke loaders — they're the single biggest flake cause — and confirm full-page height stays under the 25M-pixel budget. Local loop: `pnpm test:visual:build` once, then `pnpm test:visual:desktop` for iteration, `pnpm test:visual` for the full sweep.
+
+**Flaky snapshot.** Two main causes. (1) A loader without `data-slot="loading"` — run with `--trace=on` and inspect the `waitForFunction` step; ~0 ms duration means it isn't being waited on. (2) Random ordering — grep the page subtree for `shuffle(`, `Math.random()`, or `.sort(() =>` and route through `safeShuffle`. If dynamic content is drifting, double-check `USE_MOCK_DATA=true` is set in both build and test steps.
+
+**Local `pnpm dev` masks a regression.** `playwright.visual.config.ts` sets `reuseExistingServer: true`, which is correct for CI but means a `pnpm dev` already running on :3000 will be used silently in place of the production build the suite assumes. If a snapshot diff doesn't reproduce in CI, kill the dev server and run `pnpm test:visual:build` to rebuild against the production output before retrying.
+
+**Pixel-limit error.** Measure the page's full-page height at 1024 px; if it exceeds ~24,400 px, the page needs shortening or removal from the suite. Cropping to viewport was considered and rejected — it defeats the below-the-fold regression coverage that justifies using Playwright over Storybook here.
+
+**Works locally, fails in CI.** Usually `HOME: /root` missing from the test step — GitHub Actions overrides `HOME` inside containers, and Playwright can no longer find the browsers baked into the `mcr.microsoft.com/playwright` image. Also check that the image tag matches `@playwright/test` in `package.json`.
+
+Branch: `feat/playwright-chromatic-page-visual-tests` · PR: <https://github.com/ethereum/ethereum-org-website/pull/18009>

.github/workflows/chromatic-pages.yml

@@ -0,0 +1,90 @@
+# Playwright + Chromatic full-page visual regression tests
+# Separate from chromatic.yml (Storybook component snapshots)
+name: "Chromatic: Page Visual Tests"
+
+on:
+  pull_request:
+    branches: [master, staging, "test/**"]
+    types: [opened, synchronize, ready_for_review]
+  workflow_dispatch:
+
+jobs:
+  playwright-visual:
+    name: Build & Capture Snapshots
+    runs-on: ubuntu-latest
+    container:
+      image: mcr.microsoft.com/playwright:v1.53.1-noble
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: 20
+          cache: pnpm
+
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      - name: Build Next.js with mock data (English only)
+        run: pnpm build
+        env:
+          USE_MOCK_DATA: "true"
+          IS_VISUAL_TEST: "true"
+          NEXT_PUBLIC_BUILD_LOCALES: "en"
+
+      - name: Run visual tests
+        run: pnpm test:visual
+        env:
+          HOME: /root
+          USE_MOCK_DATA: "true"
+          IS_VISUAL_TEST: "true"
+
+      - name: Upload test results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: chromatic-archives
+          path: test-results/
+          retention-days: 1
+
+  chromatic-upload:
+    name: Upload to Chromatic
+    needs: playwright-visual
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@v4
+
+      - name: Setup Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: 20
+          cache: pnpm
+
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+
+      - name: Download test results
+        uses: actions/download-artifact@v4
+        with:
+          name: chromatic-archives
+          path: test-results/
+
+      - name: Publish to Chromatic
+        uses: chromaui/action@v16
+        with:
+          projectToken: ${{ secrets.CHROMATIC_PAGES_TOKEN }}
+          playwright: true
+          exitZeroOnChanges: true

.gitignore

@@ -10,6 +10,7 @@
 /coverage
 tests/__results__/
 tests/__report__/
+test-results/
 
 # next.js
 /.next/
@@ -64,6 +65,7 @@ src/data/crowdin/bucketsAwaitingReviewReport.csv
 build-storybook.log
 build-archive.log
 storybook-static
+storybook-static-pages
 
 # Trigger
 .trigger

app/[locale]/_components/HomepageLazy.tsx

@@ -13,6 +13,7 @@ const Skeleton = ({
 }) => (
   <Section className={className}>
     <div
+      data-slot="loading"
       className={`w-full animate-pulse rounded-2xl bg-background-highlight ${heightClass}`}
     />
   </Section>

app/api/gas-eth-price/route.ts

@@ -20,5 +20,6 @@ export async function GET() {
   return NextResponse.json({
     gasPrice: gasPriceData.gasPrice,
     ethPriceUSD: ethPriceData.value,
+    ethPercentChange24h: ethPriceData.percentChange24h,
   })
 }

next.config.js

@@ -42,6 +42,10 @@ module.exports = (phase) => {
         process.env.DEPLOY_URL ||
         process.env.URL ||
         "https://ethereum.org",
+      // Inline IS_VISUAL_TEST into the client bundle so client-side shuffles
+      // (e.g. useStakingProductsCardGrid) can opt out of randomization during
+      // visual test builds. Server code reads it from process.env directly.
+      IS_VISUAL_TEST: process.env.IS_VISUAL_TEST,
     },
     webpack: (config) => {
       config.module.rules.push({

package.json

@@ -17,6 +17,7 @@
     "build-storybook": "storybook build",
     "build-storybook:chromatic": "storybook build --test",
     "chromatic": "chromatic --project-token fee8e66c9916",
+    "chromatic:pages": "chromatic --playwright --zip --exit-zero-on-changes",
     "lint:md": "markdownlint-cli2 \"public/content/**/*.md\" \"!public/content/translations/**\"",
     "lint:md:fix": "markdownlint-cli2 --fix \"public/content/**/*.md\" \"!public/content/translations/**\"",
     "update-tutorials": "ts-node -O '{ \"module\": \"commonjs\" }' src/scripts/update-tutorials-list.ts",
@@ -27,6 +28,9 @@
     "test:e2e:ui": "playwright test --project=e2e --ui",
     "test:e2e:debug": "playwright test --project=e2e --debug",
     "test:e2e:report": "playwright show-report tests/__report__",
+    "test:visual": "playwright test --config playwright.visual.config.ts",
+    "test:visual:desktop": "playwright test --config playwright.visual.config.ts --project=chromatic-desktop",
+    "test:visual:build": "USE_MOCK_DATA=true IS_VISUAL_TEST=true NEXT_PUBLIC_BUILD_LOCALES=en pnpm build",
     "trigger:dev": "trigger dev --env-file src/data-layer/.env.local",
     "trigger:deploy": "trigger deploy --env-file src/data-layer/.env.local"
   },
@@ -108,6 +112,7 @@
     "yaml-loader": "^0.8.0"
   },
   "devDependencies": {
+    "@chromatic-com/playwright": "^0.13.1",
     "@chromatic-com/storybook": "5.1.1",
     "@google/genai": "^1.46.0",
     "@netlify/plugin-nextjs": "^5.15.9",

playwright.visual.config.ts

@@ -0,0 +1,46 @@
+import type { ChromaticConfig } from "@chromatic-com/playwright"
+import { defineConfig } from "@playwright/test"
+
+import baseConfig from "./playwright.config"
+
+const visualUse: ChromaticConfig = {
+  disableAutoSnapshot: true,
+  assetDomains: ["s3-dcl1.ethquokkaops.io"],
+}
+
+// Append "Chromatic" to the default UA so `isChromatic()` returns true client-side.
+// Several components (QuizWidget, Simulator) use this signal to skip randomization.
+const userAgent =
+  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Chromatic"
+
+export default defineConfig<ChromaticConfig>({
+  ...baseConfig,
+
+  projects: [
+    {
+      name: "chromatic-desktop",
+      testDir: "./tests/visual",
+      outputDir: "./test-results",
+      // 1024 (Tailwind `lg`) keeps full-page snapshots under Chromatic's 25M pixel limit on our longest pages.
+      use: { ...visualUse, userAgent, viewport: { width: 1024, height: 720 } },
+    },
+    {
+      name: "chromatic-tablet",
+      testDir: "./tests/visual",
+      outputDir: "./test-results",
+      use: { ...visualUse, userAgent, viewport: { width: 768, height: 1024 } },
+    },
+    {
+      name: "chromatic-mobile",
+      testDir: "./tests/visual",
+      outputDir: "./test-results",
+      use: { ...visualUse, userAgent, viewport: { width: 375, height: 812 } },
+    },
+  ],
+
+  webServer: {
+    command: "pnpm start",
+    port: 3000,
+    reuseExistingServer: true,
+  },
+})