test: custom-node E2E regression suite - Core depth and Cloud breadth (#15225)

## Summary

Adds a local-backend custom-node E2E suite with two complementary
populations. **Not a PR gate yet**: the workflow runs on a nightly
schedule and manual dispatch only - PRs neither trigger it nor wait on
any of its checks, and none of its checks are required in branch
protection. Within a run it fails closed (an install failure or skipped
tier is red).

- **Core depth:** the original six pinned packs retain S1-S13 and S15.
S14 geometry snapshots remain removed by team decision.
- **Cloud breadth:** the pinned Cloud snapshot has 87 joined manifest
rows; 83 run across five fixed shards on a local CPU ComfyUI backend.
One source row is unjoined and four rows are explicitly quarantined.

S15 is restored inside the six Core curated workflow tests that already
execute. It adds output comparison, not another prompt or Playwright
test, so its incremental runtime is negligible.

## Changes

- Pins ComfyUI core, pack sources, registry artifacts, staged inputs,
worker count, retry count, and shard composition.
- Keeps each cloud pack in a stable shared Python environment; changing
a quarantine entry cannot reshuffle other packs.
- Runs Core as its own matrix entry and Cloud as five weight-balanced
breadth shards.
- Fails on any Playwright failure, skip, flaky result, count mismatch,
dirty backend teardown, or stale exact expectation.
- Records visible errors for the page lifetime, so a transient toast
that clears before the assertion still fails S7.
- Reports every coverage exclusion in bold in the GitHub Actions summary
with its mechanism and removal condition.
- Includes two bounded frontend fixes surfaced by the suite: unset
image-upload combos no longer request filename=undefined, and empty
audio-upload sentinels no longer request a preview. These are the only
live src behavior changes in this PR.
- Provides deliberate-break proofs for S1, S2, S3, S9, and S15.

## Tier coverage and applicability

| Tier | Core | Cloud breadth | Assertion |
|---|---|---|---|
| S1 | 6 packs | rows declaring `load` | Every enrolled registered node
instantiates in LiteGraph with exact declared slot materialization. |
| S2 | 6 packs | rows declaring `load` | The enrolled S1 corpus mounts
under Vue Nodes with its visible widgets and slots represented in the
DOM. |
| S3 | 6 packs | rows declaring `load` | Enrolled node identity and
type, initialized live widget topology, and serialized widget values
survive save/reload; exact pinned pack divergences are two-way ledgered.
|
| S4 | retained | retained | Representative type-correct connections
among enrolled nodes survive graph connection and round-trip validation.
|
| S5 | retained | retained | Curated anchor links and one materialized
in-pack link per applicable pack use real drag/connection APIs in both
renderers, excluding only explicitly reported nodes. |
| S6 | retained | retained | Connectivity round-trips reach prompt
conversion and validate the serialized edge contract. |
| S7 | retained, strengthened | retained, strengthened | All
user-visible error surfaces are sampled every animation frame from
initial navigation; transient and final-state errors fail. |
| S8 | retained | retained | Console errors and uncaught page errors are
collected across startup and operations; only exact attributed
signatures are accepted. |
| S9 | all Core `run` rows | VideoHelperSuite, the only Cloud `run` row
| Exact calibrated model-free corpora queue against the real backend and
must execute or produce an observable output. |
| S10 | retained | retained | Manifest shape, exact local node counts,
registered-pack attribution, and collection counts are sentinels. |
| S11 | retained | retained where declared | Expected frontend
extensions and served web-directory assets must register. |
| S12 | Impact case | Impact case | Dynamic list input grows and shrinks
through programmatic and real drag connections in both renderers. |
| S13 | 6 pinned Core profiles | not enrolled | Existing Core
interaction-delta profiles compare at the exact recorded pack refs.
Cloud expansion is
[FE-1659](https://linear.app/comfyorg/issue/FE-1659/define-scalable-s13-interaction-regression-coverage-beyond-core).
|
| S14 | removed | removed | Team-approved removal of full node
geometry/position/size snapshots. |
| S15 | 6 Core curated workflows | not enrolled | Deterministic sink
payload hashes detect valid-but-wrong serialized output. Full-pack
expansion is
[FE-1657](https://linear.app/comfyorg/issue/FE-1657/extend-s15-output-regression-coverage-to-every-custom-node-pack).
|

The suite contains no `test.skip` or `test.fixme`, uses one worker and
`--retries=0`, and independently rejects Playwright-reported skips or
flaky results.

## Explicit coverage debt

- `comfyui-fl-path-animator` does not join the pinned Cloud snapshot.
- LivePortraitKJ has an unfetchable SHA and radiance has an
unsatisfiable `Imath` requirement. Their upstream fixes are tracked by
[FE-1660](https://linear.app/comfyorg/issue/FE-1660/fix-upstream-pack-metadata-and-remove-custom-node-e2e-quarantine).
- SeedVR2 and NVIDIA RTX register zero nodes on a CPU runner. GPU-backed
restoration is
[FE-1658](https://linear.app/comfyorg/issue/FE-1658/add-gpu-backed-custom-node-e2e-coverage-and-remove-cpu-runner).
- `comfyui-itools@0.6.8` is a banned registry artifact whose
`iToolsCropImage` hook has two terminal race outcomes under the same
pin. Only that node is excluded from S1-S8; the pack count remains exact
and its other 21 nodes run. Restoration is
[FE-1675](https://linear.app/comfyorg/issue/FE-1675/e2e-nodes-tests-fix-itools-crop-lifecycle-race-and-restore-s1-s8).
- `VHS_SelectLatest` requires the pack-owned prompt transformation and
is the one model-free node not executed by Cloud S9. Restoration is
[FE-1661](https://linear.app/comfyorg/issue/FE-1661/restore-vhs_selectlatest-s9-execution-coverage).
- `was-node-suite-comfyui/Text Random Prompt` performs an unbounded
public Lexica API request and is excluded only from Core S9.
Deterministic restoration is
[FE-1682](https://linear.app/comfyorg/issue/FE-1682/e2e-nodes-tests-restore-was-text-random-prompt-s9-execution-coverage).
- Exact known pack defects remain exercised under two-way stale ledgers;
they are not skipped. A fixed or changed outcome fails until the
expectation is removed or recalibrated with evidence.

## Review focus

- Whether the two disclosed preview guards are correct and appropriately
scoped; all remaining changes are tests, fixtures, scripts, tooling,
documentation, or CI.
- Whether every tier claim above matches its assertion and
applicability.
- Whether each temporary exclusion is specific, visible, owned, and
removable.
- Whether exact expectation ledgers describe attributable pack behavior
without weakening the asserted contract.
- Whether representative-per-slot connectivity plus per-pack
two-renderer drags is the right bounded surface; this does not claim a
producer-by-consumer cross-product or cross-shard pairing.
- Whether the fixed-shard dependency environment and manifest provenance
are sufficiently deterministic.

Supersedes #13389 and #15200. Existing review follow-ups remain tracked
in
[FE-1611](https://linear.app/comfyorg/issue/FE-1611/custom-node-e2e-the-10-review-findings-that-survive-the-cloud-cut).

---------

Co-authored-by: Nathaniel Parson Koroso <tetratrade@zoho.com>
Co-authored-by: IAMtheIAM <iamtheiam@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: CodeJuggernaut <81205671+CodeJuggernaut@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: GitHub Action <action@github.com>
Co-authored-by: github-actions <github-actions@github.com>

Author: 7 people

.gitattributes

@@ -6,3 +6,4 @@ packages/registry-types/src/comfyRegistryTypes.ts linguist-generated=true
 packages/ingest-types/src/types.gen.ts linguist-generated=true
 packages/ingest-types/src/zod.gen.ts linguist-generated=true
 src/workbench/extensions/manager/types/generatedManagerTypes.ts linguist-generated=true
+browser_tests/fixtures/data/customNodeManifest.cloud.json linguist-generated=true

.github/actions/changes-filter/action.yaml

@@ -51,7 +51,7 @@ runs:
     - name: Filter typed changes
       if: ${{ github.event_name == 'pull_request' }}
       id: filter
-      uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+      uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
       with:
         filters: |
           app_website:
@@ -75,7 +75,7 @@ runs:
     - name: Filter relevant changes
       if: ${{ github.event_name == 'pull_request' }}
       id: relevant
-      uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+      uses: dorny/paths-filter@ceb8a2b8f2d89434be7ff52d3de7ec3738c5cc9d # v4.0.3
       with:
         predicate-quantifier: 'every'
         filters: |

.github/actions/setup-comfyui-server/action.yaml

@@ -9,6 +9,10 @@ inputs:
     description: 'Whether to launch the server after setup'
     required: false
     default: 'false'
+  comfyui_ref:
+    description: 'ComfyUI git ref to check out (tag/branch/SHA). Empty = default branch (master).'
+    required: false
+    default: ''
 runs:
   using: 'composite'
   steps:
@@ -19,6 +23,7 @@ runs:
       uses: actions/checkout@v7
       with:
         repository: 'comfyanonymous/ComfyUI'
+        ref: ${{ inputs.comfyui_ref }}
         path: 'ComfyUI'
 
     - name: Install ComfyUI_devtools from frontend repo

.github/actions/setup-playwright/action.yaml

@@ -1,5 +1,10 @@
 name: Setup Playwright
 description: Cache and install Playwright browsers with dependencies
+inputs:
+  install_deps:
+    description: Install Playwright operating-system dependencies
+    required: false
+    default: 'true'
 runs:
   using: composite
   steps:
@@ -17,12 +22,33 @@ runs:
         path: '~/.cache/ms-playwright'
         key: ${{ runner.os }}-playwright-browsers-${{ steps.detect-version.outputs.playwright-version }}
 
-    - name: Install Playwright Browsers
-      if: steps.cache-playwright-browsers.outputs.cache-hit != 'true'
+    - name: Bound Playwright dependency downloads
+      if: inputs.install_deps == 'true'
       shell: bash
-      run: pnpm exec playwright install chromium --with-deps
+      run: |
+        set -euo pipefail
+        printf '%s\n' \
+          'Acquire::Retries "3";' \
+          'Acquire::http::Timeout "60";' \
+          'Acquire::https::Timeout "60";' \
+          | sudo tee /etc/apt/apt.conf.d/80-playwright-ci >/dev/null
 
-    - name: Install Playwright Browsers (operating system dependencies)
-      if: steps.cache-playwright-browsers.outputs.cache-hit == 'true'
+    - name: Install Playwright browser and dependencies
       shell: bash
-      run: pnpm exec playwright install-deps
+      env:
+        BROWSER_CACHE_HIT: ${{ steps.cache-playwright-browsers.outputs.cache-hit }}
+        INSTALL_DEPS: ${{ inputs.install_deps }}
+      run: |
+        set -euo pipefail
+        if [ "$INSTALL_DEPS" != true ]; then
+          playwright_args=(install chromium)
+        elif [ "$BROWSER_CACHE_HIT" = true ]; then
+          playwright_args=(install-deps chromium)
+        else
+          playwright_args=(install chromium --with-deps)
+        fi
+        for attempt in 1 2; do
+          pnpm exec playwright "${playwright_args[@]}" && exit 0
+          [ "$attempt" -lt 2 ] || exit 1
+          sleep 2
+        done