Skip to content

README.md

Latest commit

 

History

History
817 lines (631 loc) · 36.3 KB

README.md

File metadata and controls

817 lines (631 loc) · 36.3 KB
Raw

Browser (Playwright / E2E) Tests — Single Source of Truth

This is the canonical guide for Playwright browser tests in ComfyUI_frontend. Setup, running, structure, conventions, flake prevention, typed mocks, and the screenshot workflow all live here. Other test docs are short stubs that redirect to a section below.

Where guidance lives

Concern Location
Everything about writing/running E2E tests This file (canonical)
Agent auto-load in browser_tests/ browser_tests/AGENTS.md (stub → this file)
Agent auto-load on *.spec.ts edit docs/guidance/playwright.md (stub → this file)
Flake triage checklist Flake Prevention (was FLAKE_PREVENTION_RULES.md)
Mock data fixtures Test Data & Typed Mocks (was fixtures/data/README.md)
Authoring skill (agents) .claude/skills/writing-playwright-tests/SKILL.md
Flake-hardening skill (agents) .claude/skills/hardening-flaky-e2e-tests/SKILL.md
Review rulebook (subagent) .agents/checks/playwright-e2e.md (references this file)

Prerequisites

CRITICAL: Start the ComfyUI backend with --multi-user:

python main.py --multi-user

Without this flag, parallel tests conflict and fail randomly.

Setup

ComfyUI devtools

ComfyUI_devtools ships in this repo under tools/devtools/ and is copied into custom_nodes automatically during CI. It adds API endpoints and nodes used by browser tests. For local development, copy it into your ComfyUI install (create the target directory first — it won't exist on a fresh install):

mkdir -p /path/to/your/ComfyUI/custom_nodes/ComfyUI_devtools
cp -r tools/devtools/* /path/to/your/ComfyUI/custom_nodes/ComfyUI_devtools/

Containerized ComfyUI

To run a containerized ComfyUI backend instead of maintaining a separate install, install and start Docker, then run:

pnpm container:start

The command mounts tools/devtools and starts ComfyUI at localhost:8188. Leave it running. Use another terminal for pnpm dev and a third for pnpm test:browser:local.

Run browser tests against port 5173 when using this container. Port 8188 serves the frontend bundled into the container, not the current frontend checkout.

If the image is not cached, the launcher tries to pull it from GHCR. It uses GH_TOKEN and gets the matching username from gh api user. COMFY_CI_CONTAINER_TOKEN and COMFY_CI_CONTAINER_USER take precedence when set. If credentials are missing or the pull fails, the launcher says why and builds the matching release from source instead. This fallback may differ from the published CI image. The source build can take several minutes and uses about 10 GB, plus Docker build cache.

Remote agent containers

Remote development containers used by coding agents default to local mode. Local mode runs the containerized backend and one local frontend. Cloud mode skips Docker and the local backend, and runs one frontend against the Comfy test cloud. Set COMFYUI_FRONTEND_MODE=cloud in the container environment to switch modes.

If the container's GH_TOKEN can read the private package, no other credentials are needed. Otherwise, add a GitHub token with read:packages access to the container's secret settings. You can use COMFY_CI_CONTAINER_TOKEN with COMFY_CI_CONTAINER_USER instead of GH_TOKEN; these dedicated values take precedence. Recreate the container after changing its environment or secrets.

Node.js & Playwright

Install the Node version in .nvmrc, then the workspace dependencies and the Chromium and WebKit drivers (run from the repository root):

pnpm install
pnpm exec playwright install chromium webkit --with-deps

Environment

Create .env from the template and set the debugging keys:

cp .env_example .env
# Remove Vue dev overlay that blocks UI elements
DISABLE_VUE_PLUGINS=true

# Test against dev server (recommended) or backend directly
PLAYWRIGHT_TEST_URL=http://localhost:5173      # Dev server
# PLAYWRIGHT_TEST_URL=http://localhost:8188     # Direct backend
PLAYWRIGHT_SETUP_API_URL=http://localhost:8188 # Setup/auth API when using the dev server

# Path to ComfyUI for backing up user data/settings before tests
TEST_COMFYUI_DIR=/path/to/your/ComfyUI

Release API mocking

By default all tests mock api.comfy.org/releases so release popups don't cover UI elements. To test real release data:

await comfyPage.setup({ mockReleases: false })

See tests/releaseNotifications.spec.ts for release-specific tests.

Running Tests

pnpm test:browser:local                     # Run all E2E tests
pnpm test:browser:local widget.spec.ts      # Run a specific file
pnpm test:browser:local --ui                # Interactive UI mode (use for development)

Use UI mode while developing. It provides:

  • Locator picker — click an element to get its exact locator code
  • Step debugging — step through a test line by line via the Source tab
  • Time travel — click any step in Actions to see browser state at that moment
  • Console / Network / Attachments tabs — logs, API calls, and snapshot diffs

Slowing the browser down

With --headed or --ui, set SLOW_MO (ms) to slow every action. Applies only when PLAYWRIGHT_LOCAL is set (the default for test:browser:local):

SLOW_MO=250 pnpm test:browser:local --headed widget.spec.ts

Directory Structure

browser_tests/
├── assets/           - Test data (JSON workflows, images)
├── fixtures/
│   ├── ComfyPage.ts      - Main fixture (delegates to helpers)
│   ├── ComfyMouse.ts     - Mouse interaction helper
│   ├── VueNodeHelpers.ts - Vue Nodes 2.0 helpers
│   ├── selectors.ts      - Centralized TestIds
│   ├── data/             - Static test data (mock API responses, workflow JSONs, node definitions)
│   ├── components/       - Page object classes (locators, user interactions)
│   ├── helpers/          - Focused helper classes (domain-specific actions)
│   └── utils/            - Standalone utility functions (used by tests or fixtures)
└── tests/            - Test files (*.spec.ts), nested by feature

Spec File Placement

  • Nest specs in a folder that mirrors the feature under test. Do not dump new specs at the top level of tests/ — create/extend the matching feature subfolder (e.g. tests/templates/, tests/nodes/) so the layout mirrors the product surface. A flat tests/*.spec.ts pile is not acceptable for new work.
  • Keep spec filenames current. A spec's name must describe the coverage it actually holds. When a test's scope changes, rename the file — never leave a stale name that misdescribes what runs. Stale names send reviewers and future authors to the wrong file.

Architectural Separation

  • fixtures/data/ — Test data: mock API responses, workflow JSONs, node definitions, and the typed factory builders that produce them (e.g. createMockNodeDefinitions). No Playwright imports and no test logic — data and the functions that assemble it, nothing else.
  • fixtures/components/ — Page object components. Classes that own locators for a specific UI region (e.g. Actionbar, ContextMenu, SettingDialog, Templates).
  • fixtures/helpers/ — Helper classes that coordinate actions across multiple regions without owning a locator surface of their own (e.g. CanvasHelper, WorkflowHelper, NodeOperationsHelper).
  • fixtures/utils/ — Standalone exported functions (not classes) used by tests or fixtures (e.g. fitToView, clipboardSpy, builderTestUtils).

File Placement Rule

flowchart TD
    A[New file in browser_tests/fixtures/] --> B{Has any code?}
    B -- No, JSON/data only --> D[fixtures/data/]
    B -- Yes --> C{Is it a class?}
    C -- No, exported functions --> V{Pure data or a typed<br/>data factory?}
    V -- Yes --> D
    V -- No, test utility --> U[fixtures/utils/]
    C -- Yes --> E{Owns locators for a<br/>specific UI region?}
    E -- Yes --> P[fixtures/components/]
    E -- No, coordinates actions<br/>across the app --> H[fixtures/helpers/]
Loading

Custom-node regression suite

tests/customNodes/ holds the manifest-driven suite that proves community custom-node packs load, render in both renderers (LiteGraph canvas and Vue Nodes 2.0), and execute real workflows. It has its own prerequisites, pnpm scripts (pnpm test:custom-nodes and variants), and a one-JSON-row process for adding packs - see docs/custom-node-regression-suite.md.

Writing Tests

Golden rules

  1. Look at existing tests first. Search tests/ for similar patterns.
  2. Read the fixture code. APIs live in fixtures/ — read them directly instead of guessing.
  3. Use premade JSON workflow assets instead of building graphs programmatically. Assets live in assets/; create new ones by copying assets/default.json and editing the JSON.

Before changing an existing test

Run a regression-detection pass before you edit a test you did not write. git blame (or git log -p) the lines you are about to change and read the commit that introduced them — an assertion, wait, or timeout that looks redundant is often a deliberate fix for a past flake or bug. Confirm you are not silently undoing that fix. If the history shows the line guards a specific race/regression, preserve the guarantee (keep it, or replace it with an equivalent) rather than deleting it.

Test structure — Arrange / Act / Assert

  • All mock setup, state resets, and fixture arrangement belong in test.beforeEach() or Playwright fixtures.
  • Inside test(), only act (user actions) and assert.
  • Never call clearAllMocks or reset mock state mid-test.
  • Free-standing helpers, constants, and locator wiring don't belong inline in the spec — move them to a helper, fixture, or page object. Reusable dialog interactions (closing the templates dialog, dismissing a modal), setup routines, and locator wiring belong in fixtures/components/ (page objects), fixtures/helpers/, or a Playwright fixture — not as free-standing functions or constants at the top of the spec. Example: a "close the templates dialog" step becomes a method on a Templates page object, not an inline helper in the spec. Top-level Playwright hooks and configuration are fine to keep in the spec: test.use(), test.describe.configure(), and file-level test.beforeEach()/ test.afterEach() are Playwright's own API, not the free-standing code this rule targets. A spec file should otherwise contain only test.describe/test blocks, those Playwright hooks/config, and imports.
import {
  comfyPageFixture as test,
  comfyExpect as expect
} from '@e2e/fixtures/ComfyPage'

test.describe('Feature', { tag: ['@canvas'] }, () => {
  test.beforeEach(async ({ comfyPage }) => {
    await comfyPage.workflow.loadWorkflow('test.json')
  })

  test('should do something', async ({ comfyPage }) => {
    await comfyPage.menu.topbar.click()
    await expect(comfyPage.menu.nodeLibraryTab.root).toBeVisible()
  })
})

Import conventions

  • Prefer @e2e/* for imports within browser_tests/.
  • Use @/* for imports from src/.
  • Avoid new deep relative imports when an alias is available.

Leverage existing fixtures and helpers

Check for existing helpers before writing new ones — most needs are covered:

  • ComfyPage — main fixture; delegates to helper objects: comfyPage.workflow.loadWorkflow(), comfyPage.settings.setSetting(), comfyPage.command.executeCommand(), comfyPage.nodeOps.getNodeRefById(), comfyPage.canvasOps.resetView(), comfyPage.vueNodes.waitForNodes().
  • Component page objects — fixtures/components/ (e.g. Actionbar, ContextMenu, Templates).
  • Helper classes — fixtures/helpers/ (e.g. CanvasHelper, WorkflowHelper).
  • Utilities — fixtures/utils/ (e.g. fitToView, clipboardSpy).

Creating new test helpers

Register new domain-specific helpers as Playwright fixtures via base.extend() — do not attach them as new properties on ComfyPage. Fixtures get automatic setup/teardown and compose via mergeTests.

// browser_tests/fixtures/assetFixture.ts
import { test as base } from '@playwright/test'

export const test = base.extend<{ assetHelper: AssetHelper }>({
  assetHelper: async ({ page }, use) => {
    const helper = new AssetHelper(page)
    await helper.setup()
    await use(helper)
    await helper.cleanup() // automatic teardown
  }
})

Extend @playwright/test base (not comfyPageFixture) so helpers can be composed. The existing comfyPage.* helpers predate this rule and remain on ComfyPage; new ones should be fixtures.

Page object locator style

Define locators as public readonly properties assigned in the constructor — not as getter methods. Getters that just return a locator add indirection and hide the object shape from IDE auto-complete.

// ✅ public readonly, assigned in constructor
export class MyDialog extends BaseDialog {
  public readonly submitButton: Locator
  constructor(page: Page) {
    super(page)
    this.submitButton = this.root.getByRole('button', { name: 'Submit' })
  }
}

Keep getters only for lazy init (this._tab ??= new Tab(this.page)), runtime-computed values, or private convenience accessors. Reuse cached locator properties in methods rather than rebuilding locators.

Selectors

  • Prefer role, label, or test-id selectors over CSS/DOM-structure selectors.
  • Hard-coded data-testid strings should reference fixtures/selectors.ts when a matching entry exists.
  • When multiple nodes share a title, disambiguate: vueNodes.getNodeByTitle(name).nth(n) — strict mode fails on ambiguous locators.

Node references over coordinates

Node references from fixtures/utils/litegraphUtils.ts are stable; raw coordinates are fragile.

// ✅ Prefer — target by id, or by type when exactly one match is expected
const node = await comfyPage.nodeOps.getNodeRefById('12')
await node.click('title')

// ❌ Avoid
await comfyPage.canvas.click({ position: { x: 100, y: 100 } })

When selecting by type and more than one node can match, don't index [0] on the returned array — the order is non-deterministic (see the Flake Prevention patterns table). Use getNodeRefById(), or guard with toHaveLength(1) first.

Vue Nodes vs LiteGraph — decision guide

Choose based on what you're testing:

Testing… Use Why
Vue-rendered node UI, DOM widgets, CSS states comfyPage.vueNodes.* Nodes are DOM elements; use locators
Canvas interactions, connections, legacy nodes comfyPage.nodeOps.* Canvas-based; use coordinates/refs
Both in one test Pick primary, minimize switching Mixing both is a smell

Vue Nodes requires explicit opt-in:

await comfyPage.settings.setSetting('Comfy.VueNodes.Enabled', true)
await comfyPage.vueNodes.waitForNodes()

Vue Node state is expressed via CSS classes:

const BYPASS_CLASS = /before:bg-bypass\/60/
await expect(node).toHaveClass(BYPASS_CLASS)

Canvas timing

  • After canvas ops (drag, canvas click, resizeNode, pan, zoom, or a page.evaluate graph mutation that changes visuals), call await comfyPage.nextFrame() before asserting. loadWorkflow() does not need it. Prefer encapsulating nextFrame() inside page-object methods.
  • Focus the canvas before keyboard input: await comfyPage.canvas.click() before page.keyboard.press(...), or keys go nowhere.
  • Mark the canvas dirty after programmatic state changes: window['app'].graph.setDirtyCanvas(true, true).
  • dblclick() on canvas needs a small { delay: 5 }; drags need { steps: 10 } not { steps: 1 }.

Custom assertions

Prefer adding assertion methods directly on the page object or helper class — they're discoverable via IntelliSense without special imports.

// ✅ Page-object assertions for new code
await node.expectPinned()

The existing comfyExpect matchers (toBePinned, toBeBypassed, toBeCollapsed, toHaveFocus, toHaveScreenshot) in fixtures/utils/customMatchers.ts remain supported and are used across the suite — keep using them. Do not extend comfyExpect with new matchers; add new assertions as page-object methods instead.

Test tags

Two tiers. Project-routing tags are load-bearing — playwright.config.ts selects which project/run a test lands in by grepping these, so a test only runs where its tags place it:

Tag Effect (per playwright.config.ts)
@mobile Runs in the mobile-chrome (Pixel 5) project
@mobile-ios Runs in the mobile-safari (iPhone 15 / WebKit) project
@2x Runs in the 2x-scale project
@0.5x Runs in the 0.5x-scale project
@perf Runs in the perf project
@audit Runs in the audit project
@cloud Runs in the cloud project
@oss Excluded from the cloud project

Use @mobile-ios sparingly — only for regressions that reproduce under iOS-shaped conditions (WKWebView bridge exposure, WebKit-only quirks). Playwright's WebKit engine does not expose embedded-WKWebView globals such as window.webkit.messageHandlers; inject them via page.addInitScript() and set the context userAgent. See browser_tests/tests/cloudLoginIosWebview.spec.ts for the reference pattern.

Organizational tags are used for manual --grep filtering (not project routing). Common ones in the suite: @smoke, @slow, @screenshot, @canvas, @node, @widget, @vue-nodes, @subgraph, @ui. Apply them so a test is findable by its area; add a project-routing tag whenever the test must run in that project.

test.describe('Feature', { tag: ['@screenshot', '@canvas'] }, () => {

Type safety

Never use any or as any in E2E code.

Browser tests read window.app, window.graph, window.LiteGraph (optional in app types). Use non-null assertions in E2E tests only:

window.app!.graph!.nodes
window.LiteGraph!.registered_node_types

Acceptable type assertions:

window.app!.extensionManager
id: 'TestSetting' as TestSettingId
type TestSettingId = keyof Settings

Forbidden: settings: testData as any, data as unknown as SomeType. Read internal state via page.evaluate and stores directly — don't widen public API types to expose internals.

Never inline-declare types in tests. Do not hand-write interface/type shapes for API payloads, node definitions, or store data inside a spec or mock file. Import the real type instead — the generated packages (@comfyorg/ingest-types, @comfyorg/registry-types, generatedManagerTypes.ts) or the src/ Zod schemas (see Test Data & Typed Mocks for the full source-of-truth table). An inline type is a copy that silently drifts from the real shape; a generated type or schema fails to compile the moment the contract changes.

Assertion messages & soft assertions

Attach a message to precondition checks so a failure points at the broken assumption, and use expect.soft() to verify several invariants without aborting on the first failure:

expect(node.widgets, 'Widget count changed — update test fixture').toHaveLength(
  4
)

expect.soft(menuItem1).toBeVisible()
expect.soft(menuItem2).toBeVisible()

When to use page.evaluate

Acceptable (sparingly) — reading internal state with no UI representation, or setting up fixtures:

const nodeCount = await page.evaluate(() => window.app!.graph!.nodes.length)
await page.evaluate(() => useWorkflowStore().activeWorkflow)
await page.evaluate(() => window.app!.registerExtension({ name: 'TestExt' }))

Avoid — performing actions that have a UI equivalent; use locators instead:

// ❌ node.widgets![0].value = 512     ✅ widgetLocator.click(); widgetLocator.fill('512')
// ❌ btn.dispatchEvent(new MouseEvent('click'))  ✅ page.getByTestId('...').click()
// ❌ app.queuePrompt(0)               ✅ page.getByRole('button', { name: 'Queue' }).click()

Preferred — helper methods from fixtures/helpers/ that wrap real user interactions.

Minimal workflows & cleanup

  • Load the smallest workflow the test needs (loadWorkflow('single_ksampler')), not the full default graph.
  • Server-persisted state (settings, uploaded files, saved workflows) leaks across tests. Reset it in afterEach (or a fixture) and clean up files:
await comfyPage.settings.setSetting('Comfy.ColorPalette', 'dark')
comfyPage.deleteFileAfterTest({ filename: 'image.png' })
  • Tests that manipulate canvas view should resetView() in afterEach:
test.afterEach(async ({ comfyPage }) => {
  await comfyPage.canvasOps.resetView()
})

Debug helpers (remove before committing)

comfyPage.debugAddMarker(pos), debugAttachScreenshot(testInfo, name), debugShowCanvasOverlay(), debugGetCanvasDataURL() are for local debugging only. Never commit them.

Test Data & Typed API Mocks

Mock data in fixtures/data/ exports typed objects that conform to generated types or Zod schemas — never ad-hoc inline JSON. Typing mocks catches shape drift at compile time instead of through flaky runtime failures.

comfyPageFixture navigates during setup(), before the test body. Destructuring { comfyPage } in a hook triggers that setup before the hook body runs, so a page.route() placed there misses the first navigation. Register routes before navigation — from a custom fixture, or a beforeEach that takes only { page } and registers routes before comfyPage is ever set up — to intercept initial page-load requests.

import { createMockNodeDefinitions } from '@e2e/fixtures/data/nodeDefinitions'

const nodeDefs = createMockNodeDefinitions({ MyCustomNode: {/* ... */} })
await page.route('**/api/object_info', (route) =>
  route.fulfill({ json: nodeDefs })
)

Sources of truth for mock types

The three generated-type packages are auto-generated from OpenAPI specs — prefer them for any mock targeting their endpoints:

Endpoint category Type source
Cloud-only (hub, billing, workflows) @comfyorg/ingest-types (packages/ingest-types)
Registry (releases, nodes, publishers) @comfyorg/registry-types (packages/registry-types)
Manager (queue tasks, packages) generatedManagerTypes.ts (src/workbench/extensions/manager/types/)
Python backend (queue, history, settings, features) Manual Zod schemas in src/schemas/apiSchema.ts
Node definitions src/schemas/nodeDefSchema.ts, src/schemas/nodeDef/nodeDefSchemaV2.ts
Templates src/platform/workflow/templates/types/template.ts
Jobs API src/platform/remote/comfyui/jobs/jobTypes.ts (zJobDetail, zJobsListResponse)
Workflow validation src/platform/workflow/validation/schemas/workflowSchema.ts
Asset metadata src/types/metadataTypes.ts 
// ✅ Import the type and annotate mock data
import type { ReleaseNote } from '@/platform/updates/common/releaseService'
const mockRelease: ReleaseNote = {
  id: 1,
  project: 'comfyui',
  version: 'v0.3.44' /* ... */
}

// ❌ Untyped inline JSON — schema drift goes unnoticed
body: JSON.stringify([{ id: 1, project: 'comfyui', version: 'v0.3.44' }])

The example annotates with the app-facing ReleaseNote (which the release service derives from the registry response) to show the pattern. When you mock the raw endpoint response body, annotate with the generated type from the table above — for releases that is @comfyorg/registry-types — so the mock tracks the API shape directly.

Keep fixture values realistic but stable — avoid dates, random IDs, or anything that would cause flakiness. When adding a fixture, locate the generated type or Zod schema first, then create a .ts file that satisfies it.

Flake Prevention

Distilled from the PR 10817 stabilization thread. Reference this section when triaging or editing flaky tests.

Quick checklist

Before merging a flaky-test fix, confirm all of these:

  • the latest CI artifact was inspected directly
  • the root cause is stated as a race or readiness mismatch
  • the fix waits on the real readiness boundary
  • the assertion primitive matches the job
  • the fix stays local unless a shared helper truly owns the race
  • local verification uses a targeted rerun

1. Start with CI evidence

  • Don't trust the top-level GitHub check alone. Inspect the latest Playwright report.json directly, even on a green run.
  • Treat tests marked flaky in report.json as real work.
  • Use error-context.md, traces, and page snapshots before editing code.
  • Pull the newest run after each push — the flaky set changes.

2. Wait for the real readiness boundary

  • Visible is not always ready. If behavior depends on internal state, wait on that state.
  • After canvas interactions, await comfyPage.nextFrame() unless the helper already guarantees a settled frame.
  • After workflow reloads or node-def refreshes, wait for the reload to finish.
  • Common boundaries: node.imgs populated before image context menus; settings cleanup finished before asserting persisted state; locale-triggered reload finished before selecting nodes; real builder UI ready (not transient helper metadata).

3. Choose the smallest correct assertion

  • Locator state → built-in retrying assertions (toBeVisible, toHaveText, toHaveCount, toHaveClass).
  • Single async value → expect.poll(() => asyncFn()).toBe(expected).
  • Multiple assertions that must settle together → expect(async () => { ... }).toPass().
  • Never make immediate assertions right after async UI mutations, settings writes, clipboard writes, or graph updates.
  • Never use waitForTimeout() to hide a race — it is always wrong.

Prefer expect.poll() over toPass() for a single async call + single assertion (better error messages). Use the default poll timeout (5000 ms); never tighten below ~2000 ms.

await expect
  .poll(() => comfyPage.settings.getSetting('Comfy.NodeLibrary.Bookmarks.V2'))
  .toEqual([])

4. Prefer behavioral assertions

Use screenshots only when appearance is the behavior under test. Otherwise assert on link counts, positions, visible menu items, persisted settings, or node state.

5. Keep helper changes narrow

  • Shared helpers should drive setup to a stable boundary.
  • Don't encode one-spec timing assumptions into generic helpers — prefer a local wait in that spec.
  • If a helper fails before the real test begins, relax the brittle precondition and let downstream UI interaction prove readiness.

6. Verify narrowly

  • Targeted reruns via pnpm test:browser:local.
  • On Windows, prefer file:line or whole-spec args over --grep (quoting issues).
  • Use --repeat-each 5 for targeted flake verification.

Common flake patterns

Pattern Bad Fix
Snapshot-then-assert expect(await evaluate()).toBe(x) await expect.poll(() => evaluate()).toBe(x)
Immediate boundingBox/layout read const box = await loc.boundingBox(); expect(box!.width).toBe(w) await expect.poll(() => loc.boundingBox().then(b => b?.width)).toBe(w)
Immediate graph state after drop expect(await getLinkCount()).toBe(1) await expect.poll(() => getLinkCount()).toBe(1)
Fake readiness helper Helper that clicks but doesn't assert state Remove; poll the actual value
nextFrame after menu click clickMenuItem(x); nextFrame() clickMenuItem(x); contextMenu.waitForHidden()
Tight poll timeout expect.poll(..., { timeout: 250 }) ≥2000 ms; prefer default (5000 ms)
Immediate count() const n = await loc.count(); expect(n).toBe(3) await expect(loc).toHaveCount(3)
Immediate evaluate after mutation setSetting(); expect(await evaluate()).toBe(x) await expect.poll(() => evaluate()).toBe(x)
Screenshot without readiness loadWorkflow(); nextFrame(); toHaveScreenshot() waitForNodes() or poll state first
Non-deterministic node order getNodeRefsByType('X')[0] with >1 match getNodeRefById(id) or guard toHaveLength(1)

Local noise (not automatic CI root causes)

Missing local input fixtures/models, teardown EPERM while restoring the local user-data dir, and local screenshot baseline diffs on Windows are local distractions. Confirm whether they block the exact flaky path before acting. Never commit temporary local assets or local screenshot baselines.

Gotchas

Symptom Cause Fix
subtree intercepts pointer events on DOM widgets Canvas z-999 overlay intercepts click() Use locator.dispatchEvent('contextmenu', { bubbles: true, cancelable: true, button: 2 })
Context menu empty or wrong items Node not selected Select node first: vueNodes.selectNode() or nodeRef.click('title')
navigateIntoSubgraph timeout Node too small in test asset JSON Use node size [400, 200] minimum
Keyboard shortcuts don't work Missing focus await comfyPage.canvas.click() first
Widget value wrong after drag-drop Upload incomplete Add { waitForUpload: true }

Screenshot / Visual Regression Testing

Screenshot expectations are platform-specific (font rendering). We maintain Linux baselines because CI runs on Linux.

  • Do not commit local screenshot baselines.
  • Prefer functional assertions; use screenshots only when appearance is the behavior under test.
await expect(comfyPage.canvas).toHaveScreenshot('state.png')
// Mask dynamic content (timestamps, versions) to avoid flakes:
await expect(comfyPage.canvas).toHaveScreenshot('page.png', {
  mask: [page.locator('.timestamp')]
})

Working with screenshots locally

Skip screenshot tests locally:

// playwright.config.ts
export default defineConfig({
  grep: process.env.CI ? undefined : /^(?!.*screenshot).*$/
})

grep matches the test title and its tags, not the test body — a call to toHaveScreenshot() alone won't be detected. Tag every screenshot test @screenshot (or put "screenshot" in its title) for this filter to exclude it.

Or generate local baselines for comparison (do not commit them). If you added the screenshot-skipping grep above, remove it (or override --grep for this run) — otherwise --update-snapshots won't regenerate the excluded baselines:

pnpm test:browser:local --update-snapshots

Creating new baselines in CI

  1. Write the test with toHaveScreenshot('filename.png').
  2. Open the PR and add the New Browser Test Expectations label.
  3. CI generates and commits the Linux baselines.

Fork PRs can't auto-commit screenshots — a maintainer commits them for you.

Debugging in CI

  1. Download artifacts from the failed run.
  2. View the trace: pnpm dlx playwright show-trace trace.zip.
  3. CI deploys an HTML report to Cloudflare Pages (link in the PR comment).
  4. Reproduce CI locally: CI=true pnpm test:browser.

Test Reports (Cloudflare Pages)

Reports deploy automatically for every PR and push to main branches, one Cloudflare Pages project per browser configuration (comfyui-playwright-chromium, -mobile-chrome, -chromium-2x, -chromium-0-5x), with branch-specific URLs (https://[branch].comfyui-playwright-[browser].pages.dev). PR comments carry ✅/❌ status and direct report links per browser.

New-Test Video Walkthrough

CI re-runs any Playwright spec file newly added in a PR (detected via git diff --diff-filter=A against the base branch) with video recording enabled and deploys the resulting HTML report — Playwright's report embeds the video inline per test — to the same comfyui-playwright-chromium Cloudflare Pages project, on its own branch. The link is posted alongside the other browser report links in the PR comment. It's scoped to added files only — the existing suite always runs without video — to keep CI cost and time bounded. Recording is opt-in via RECORD_VIDEO=true (set only by that CI job; see playwright.config.ts) and slowed down with SLOW_MO so the result is legible for reviewers. Files containing no tests eligible for the chromium project, such as @perf-only specs, are skipped with an explanation in the workflow summary. Newly added @audit tests fail this job instead because the audit project has no regular CI coverage.

After Making Changes

  • pnpm typecheck:browser after modifying TypeScript in this directory.
  • pnpm exec eslint browser_tests/path/to/file.ts to lint specific files.
  • pnpm exec oxlint browser_tests/path/to/file.ts for oxlint.

Resources