Skip to content

Commit d20536a

Browse files
ackeclaude
andcommitted
test(playwright): add browser tests + visual regression for HTML panels
Wires Playwright as an opt-in JS browser test suite alongside the existing node --test suite. Two specs: - tree-runtime.playwright.spec.mjs: behavior tests for the tree view runtime — IDE placeholder substitution, issue click → bridge call, product row expand/collapse — exercised in real Chromium against the actual domain/ide/treeview/template/tree.js. - panel-screenshots.playwright.spec.mjs: visual regression for 7 HTML panels (tree view, config page, code/oss/iac/secrets suggestion, scan summary) × 4 IDE themes (vscode, intellij, visualStudio, eclipse). Baselines committed under js-tests/screenshots/ as Darwin PNGs. scripts/panel-fixtures/main.go is a single Go program driven by --panel that builds one canonical example issue/state per panel and renders via the real renderers in infrastructure/{code,oss,iac,secrets} and domain/scanstates. Wired into a panel-fixtures Makefile target consumed by test-js / test-js-browser. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 1cc2994 commit d20536a

37 files changed

Lines changed: 697 additions & 4 deletions

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,8 @@ CLAUDE.md
8383

8484
# JS test dependencies
8585
node_modules/
86+
js-tests/playwright-report/
87+
js-tests/test-results/
8688

8789
.vscode
8890
.vault

Makefile

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -152,12 +152,28 @@ benchmark-real:
152152

153153
## test-js: Run all JavaScript tests (tree view + config dialog) and check ES5 compatibility.
154154
.PHONY: test-js
155-
test-js: tree-view-fixture config-dialog-fixture
155+
test-js: tree-view-fixture config-dialog-fixture panel-fixtures
156156
@echo "==> Running JS tests..."
157157
@cd js-tests && npm install --ignore-scripts && npm test
158158
@echo "==> Linting JS for ES5 compatibility..."
159159
@cd js-tests && npm run lint:es5
160160

161+
## test-js-browser-setup: Install Playwright browser runtime for optional JS browser tests.
162+
.PHONY: test-js-browser-setup
163+
test-js-browser-setup:
164+
@echo "==> Installing Playwright browser runtime..."
165+
@cd js-tests && npm install --ignore-scripts && npm run playwright:install
166+
167+
## test-js-browser: Run optional Playwright browser tests for tree view runtime.
168+
.PHONY: test-js-browser
169+
test-js-browser: tree-view-fixture config-dialog-fixture panel-fixtures
170+
@echo "==> Running JS browser tests (Playwright)..."
171+
@cd js-tests && npm install --ignore-scripts && npm run test:playwright
172+
173+
## test-js-all: Run all JS tests including optional Playwright browser tests.
174+
.PHONY: test-js-all
175+
test-js-all: test-js test-js-browser
176+
161177
.PHONY: race-test
162178
race-test:
163179
@echo "==> Running integration tests with race-detector..."
@@ -191,6 +207,16 @@ config-dialog-fixture:
191207
@go run scripts/config-dialog/main.go --dummy-data -no-panel > js-tests/fixtures/config-page.html
192208
@echo " Written to js-tests/fixtures/config-page.html"
193209

210+
## panel-fixtures: Regenerate code/oss/iac/secrets/scan-summary HTML fixtures used by JS tests.
211+
.PHONY: panel-fixtures
212+
panel-fixtures:
213+
@echo "==> Generating panel HTML fixtures..."
214+
@mkdir -p js-tests/fixtures
215+
@for p in code-suggestion oss-suggestion iac-suggestion secrets-suggestion scan-summary; do \
216+
echo " --> $$p" ; \
217+
go run ./scripts/panel-fixtures --panel $$p --output-file js-tests/fixtures/$$p.html ; \
218+
done
219+
194220
## generate: Regenerate generated files (e.g. mocks).
195221
.PHONY: generate
196222
generate:

js-tests/README.md

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# JS tests
2+
3+
Two suites live here:
4+
5+
- **Node tests** (`*.test.mjs`) — fast unit tests of webview helpers (tree runtime, dirty tracker, tabs, bridge, form/auth handlers). Run via `node --test` with jsdom.
6+
- **Playwright tests** (`*.playwright.spec.mjs`) — browser tests that load the real `domain/ide/treeview/template/tree.js` runtime in headless Chromium and assert on rendered DOM, computed CSS, and the `__ideExecuteCommand__` bridge calls each IDE would receive.
7+
8+
The Playwright suite is **opt-in** because it requires a Chromium download (~150 MB) and is slower than `node --test`.
9+
10+
## One-time setup
11+
12+
```sh
13+
make test-js-browser-setup
14+
```
15+
16+
This runs `npm install` in `js-tests/` and downloads the Chromium build Playwright drives.
17+
18+
## Running locally
19+
20+
| Command | What it runs |
21+
| ----------------------------- | ---------------------------------------------------------------- |
22+
| `make test-js` | Node tests + ES5 lint. Default for JS work. |
23+
| `make test-js-browser` | Playwright tests only (requires `test-js-browser-setup` first). |
24+
| `make test-js-all` | Node tests + Playwright tests. |
25+
26+
Direct npm equivalents (from `js-tests/`):
27+
28+
```sh
29+
npm test # node --test
30+
npm run test:playwright # playwright test (headless)
31+
npm run test:playwright:headed # playwright test --headed (watch the browser)
32+
```
33+
34+
## Visual regression baselines
35+
36+
`panel-screenshots.playwright.spec.mjs` renders each panel HTML fixture (`fixtures/tree-view.html`, `fixtures/config-page.html`) under each IDE's body bg/fg colors and compares against committed PNG baselines under `screenshots/`.
37+
38+
Baselines are **per-platform** (`<name>-{darwin,linux}.png`) because Chromium's font rendering differs across OSes. Tests use `threshold: 0.2` and `maxDiffPixelRatio: 0.02` to absorb minor antialiasing noise.
39+
40+
After an intentional UI change (panel HTML, CSS, or fixture content), regenerate the baselines for your platform and commit them:
41+
42+
```sh
43+
cd js-tests
44+
npx playwright test panel-screenshots.playwright.spec.mjs --update-snapshots
45+
git add screenshots/
46+
```
47+
48+
CI runs on Linux, so Linux baselines (`*-linux.png`) must also be committed when the visual changes. Generate them by running `--update-snapshots` in a Linux environment (CI container, Docker, etc.) and committing the result.
49+
50+
When a screenshot diff fails unintentionally, the HTML report (`npx playwright show-report`) shows the expected, actual, and diff PNGs side-by-side under "Attachments".
51+
52+
## Inspecting failures
53+
54+
When a Playwright test fails, the config retains a trace, screenshot, and video for that run:
55+
56+
```sh
57+
cd js-tests
58+
npx playwright show-report # opens playwright-report/ (HTML)
59+
npx playwright show-trace test-results/<failing-test>/trace.zip
60+
```
61+
62+
Use headed mode to step through interactively:
63+
64+
```sh
65+
cd js-tests
66+
npx playwright test --headed --debug
67+
```
68+
69+
Both `playwright-report/` and `test-results/` are gitignored.
70+
71+
## CI/CD
72+
73+
Today, `.github/workflows/build.yaml` runs `make clean test`, which depends on `test-js` (Node tests + ES5 lint). The Playwright suite is **not** in the default CI path.
74+
75+
To add it, run `test-js-browser` after the existing test step on Linux. Chromium download is cached by `actions/setup-node` between runs:
76+
77+
```yaml
78+
- name: Set up Node.js
79+
uses: actions/setup-node@v6
80+
with:
81+
node-version: '20'
82+
cache: 'npm'
83+
cache-dependency-path: js-tests/package-lock.json
84+
85+
- name: Install Playwright browser
86+
run: make test-js-browser-setup
87+
88+
- name: Run Playwright browser tests
89+
run: make test-js-browser
90+
```
91+
92+
Recommended placement: a new `js-browser-tests` job in `build.yaml` that runs in parallel with `integration-tests` (Linux-only — Chromium launch on macOS/Windows runners adds time without meaningful additional coverage for these tests).

js-tests/package-lock.json

Lines changed: 67 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

js-tests/package.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,13 @@
44
"type": "module",
55
"scripts": {
66
"test": "node --test --experimental-test-coverage",
7+
"test:playwright": "playwright test",
8+
"test:playwright:headed": "playwright test --headed",
9+
"playwright:install": "playwright install chromium",
710
"lint:es5": "eslint .."
811
},
912
"devDependencies": {
13+
"@playwright/test": "^1.57.0",
1014
"eslint": "^10.1.0",
1115
"jsdom": "^26.1.0"
1216
}
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
import { test, expect } from "@playwright/test";
2+
import { readFile } from "node:fs/promises";
3+
import { dirname, join } from "node:path";
4+
import { fileURLToPath } from "node:url";
5+
6+
const __filename = fileURLToPath(import.meta.url);
7+
const __dirname = dirname(__filename);
8+
9+
const ideVariants = [
10+
{ name: "vscode", background: "rgb(30, 30, 30)", foreground: "rgb(204, 204, 204)" },
11+
{ name: "intellij", background: "rgb(43, 43, 43)", foreground: "rgb(187, 187, 187)" },
12+
{ name: "visualStudio", background: "rgb(37, 37, 38)", foreground: "rgb(241, 241, 241)" },
13+
{ name: "eclipse", background: "rgb(27, 30, 35)", foreground: "rgb(216, 221, 229)" },
14+
];
15+
16+
const panels = [
17+
{ name: "tree-view", fixture: "tree-view.html" },
18+
{ name: "config-page", fixture: "config-page.html" },
19+
{ name: "code-suggestion", fixture: "code-suggestion.html" },
20+
{ name: "oss-suggestion", fixture: "oss-suggestion.html" },
21+
{ name: "iac-suggestion", fixture: "iac-suggestion.html" },
22+
{ name: "secrets-suggestion", fixture: "secrets-suggestion.html" },
23+
{ name: "scan-summary", fixture: "scan-summary.html" },
24+
];
25+
26+
for (const panel of panels) {
27+
for (const ide of ideVariants) {
28+
test(`screenshot: ${panel.name} × ${ide.name}`, async ({ page }, testInfo) => {
29+
const rawHtml = await readFile(join(__dirname, "fixtures", panel.fixture), "utf8");
30+
// Strip CSP meta tags — fixtures contain webview CSPs that block test-injected styles.
31+
const html = rawHtml.replace(/<meta[^>]*http-equiv=["']Content-Security-Policy["'][^>]*>/gi, "");
32+
await page.setViewportSize({ width: 1280, height: 900 });
33+
await page.setContent(html, { waitUntil: "domcontentloaded" });
34+
await page.addStyleTag({
35+
content: `body { background-color: ${ide.background} !important; color: ${ide.foreground} !important; }`,
36+
});
37+
const buf = await page.screenshot({ fullPage: true, animations: "disabled" });
38+
await testInfo.attach(`${panel.name}-${ide.name} (full page)`, {
39+
body: buf,
40+
contentType: "image/png",
41+
});
42+
await expect(page).toHaveScreenshot(`${panel.name}-${ide.name}.png`, {
43+
fullPage: true,
44+
threshold: 0.2,
45+
maxDiffPixelRatio: 0.02,
46+
animations: "disabled",
47+
});
48+
});
49+
}
50+
}

js-tests/playwright.config.mjs

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
import { defineConfig, devices } from "@playwright/test";
2+
3+
export default defineConfig({
4+
testDir: ".",
5+
testMatch: ["*.playwright.spec.mjs"],
6+
timeout: 30_000,
7+
expect: {
8+
timeout: 5_000,
9+
},
10+
fullyParallel: true,
11+
retries: process.env.CI ? 2 : 0,
12+
workers: process.env.CI ? 2 : undefined,
13+
reporter: [["list"], ["html", { open: "never", outputFolder: "playwright-report" }]],
14+
snapshotPathTemplate: "screenshots/{arg}-{platform}{ext}",
15+
use: {
16+
...devices["Desktop Chrome"],
17+
trace: process.env.PW_TRACE || "retain-on-failure",
18+
screenshot: process.env.PW_SCREENSHOT || "on",
19+
video: process.env.PW_VIDEO || "retain-on-failure",
20+
},
21+
projects: [
22+
{
23+
name: "chromium",
24+
},
25+
],
26+
});
34.4 KB
Loading
30.3 KB
Loading
32.7 KB
Loading

0 commit comments

Comments
 (0)