test: custom-node E2E regression suite #13389
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,203 @@ | ||
| # Records the CLOUD custom-node geometry baselines | ||
| # (browser_tests/fixtures/customNode/geometry/cloud/<pack>.json) and uploads | ||
| # them as an artifact to commit. The cloud sibling of | ||
| # record-custom-nodes-geometry.yaml: same record mechanism (mount sweep in | ||
| # CN_GEOMETRY=record mode, which throws after writing each pack's file so a | ||
| # record can never look like a green test run), with the backend swapped for | ||
| # Cloud. Kept as a SEPARATE file rather than a suite_env input on the core | ||
| # record workflow so the core record path stays byte-for-byte unchanged and | ||
| # neither file has to thread conditionals through every step - the same | ||
| # core/cloud split the two gate workflows already use. | ||
| # | ||
| # Cloud swaps out the whole local-backend half: no ComfyUI checkout, no pack | ||
| # install, no torch constraints, no `python main.py`. The dist is served by | ||
| # `vite preview` with /api proxied to Cloud (preview:cloud-e2e), and the mount | ||
| # sweep runs under CUSTOM_NODES_ENV=cloud so geometry.ts records into | ||
| # geometry/cloud/. | ||
| # | ||
| # SECRETS ARE REQUIRED here (unlike the always-on cloud gate, which no-ops when | ||
| # they are absent). A record run is a deliberate operator dispatch, so missing | ||
| # secrets are an operator error, not an expected pre-calibration state - the | ||
| # job FAILS LOUD rather than silently recording nothing. | ||
| # | ||
| # Runs for real only once the smoke user credentials AND the generated | ||
| # customNodeManifest.cloud.json exist (without the manifest the mount sweep | ||
| # throws at collection under cloud env). Dispatch it during Phase 5 | ||
| # calibration, after the manifest lands, on the branch carrying it. | ||
| name: 'Record Custom Nodes Geometry (Cloud)' | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| # workflow_dispatch cannot see a workflow that has never been on the | ||
| # default branch; pushing a record/custom-nodes-cloud-* ref runs this | ||
| # file from that ref (the core record workflow's escape hatch). Delete | ||
| # the ref after collecting the artifact. | ||
| push: | ||
| branches: ['record/custom-nodes-cloud-*'] | ||
|
|
||
| concurrency: | ||
| # Shared literal group with ci-tests-custom-nodes-cloud.yaml. The reason | ||
| # is queue-scoped, not instance-topology (the earlier "one shared Cloud | ||
| # instance" wording here was unsourced and is retracted): both workflows | ||
| # sign in as the ONE smoke account (smokeAuth.ts memoizes a single | ||
| # SMOKE_ACCOUNT_EMAIL identity), so they share one ComfyUI queue. Per-test | ||
| # cleanup is now scoped to each page's own prompt ids, but the shared | ||
| # queue still makes a record run and a gate run interfere: waitForQueueQuiet | ||
| # (customNodeSuite.ts) waits on the whole queue and cannot tell whose work | ||
| # it is. Retire this group when the smoke-account pool lands (see the | ||
| # gate's concurrency comment for the full semantics). Every gate event - | ||
| # including workflow_dispatch - now shares this group, so a record run and | ||
| # a gate run can no longer overlap. | ||
| group: custom-nodes-cloud-instance | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| record-geometry-cloud: | ||
| if: github.repository == 'Comfy-Org/ComfyUI_frontend' | ||
| runs-on: ubuntu-latest | ||
| # 60 was inherited from the core record job and is far too small for | ||
| # cloud: run 30445909728 was killed at exactly 60.25 minutes, mid | ||
| # geometry record, and a job-level timeout CANCELS - which reads as a | ||
| # queue eviction and sent four attempts chasing the wrong cause. Cloud | ||
| # pays per-test app load over the network (the cloud gate needs 287+ | ||
| # minutes for the same corpus), so this sits just under GitHub's 360 | ||
| # hard wall. The two record steps carry their own lower ceilings so an | ||
| # overrun FAILS the step - keeping the upload steps reachable - instead | ||
| # of cancelling the job and losing the partial artifact. | ||
| timeout-minutes: 350 | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| # Fail loud if the operator dispatched without the cloud secrets: a record | ||
| # that silently produced no baselines would look like a crashed run. | ||
| - name: Require smoke user credentials | ||
| env: | ||
| SMOKE_ACCOUNT_EMAIL: ${{ secrets.SMOKE_ACCOUNT_EMAIL }} | ||
| SMOKE_ACCOUNT_PASSWORD: ${{ secrets.SMOKE_ACCOUNT_PASSWORD }} | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| missing=() | ||
| [ -n "${SMOKE_ACCOUNT_EMAIL:-}" ] || missing+=(SMOKE_ACCOUNT_EMAIL) | ||
| [ -n "${SMOKE_ACCOUNT_PASSWORD:-}" ] || missing+=(SMOKE_ACCOUNT_PASSWORD) | ||
| if [ ${#missing[@]} -gt 0 ]; then | ||
| echo "::error::cloud geometry record needs these repo secrets: ${missing[*]}" | ||
| exit 1 | ||
| fi | ||
|
|
||
| - name: Checkout repository | ||
| uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup frontend | ||
| uses: ./.github/actions/setup-frontend | ||
| with: | ||
| include_build_step: false | ||
|
|
||
| # DISTRIBUTION is a build-time constant: the recorded baselines must come | ||
| # from the cloud dist, not a localhost build (see the gate's build step). | ||
| - name: Build cloud dist | ||
| shell: bash | ||
| run: pnpm build:cloud-e2e | ||
|
|
||
| - name: Setup Playwright | ||
| uses: ./.github/actions/setup-playwright | ||
|
|
||
| # Same served-dist-against-Cloud mechanism as the cloud gate. | ||
| - name: Start cloud preview server | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| pnpm preview:cloud-e2e --port 4173 & | ||
| for _ in $(seq 1 60); do | ||
| if curl -sf -o /dev/null http://localhost:4173/; then | ||
| echo "cloud preview server ready on :4173" | ||
| exit 0 | ||
| fi | ||
| sleep 2 | ||
| done | ||
| echo "::error::cloud preview server did not become ready on :4173" | ||
| exit 1 | ||
|
|
||
| # The checkout brings any committed cloud baselines along; delete them so | ||
| # the uploaded artifact is provably a product of THIS run and the | ||
| # if-no-files-found guard can fire on a crashed record instead of shipping | ||
| # stale bytes. `rm -f` no-ops cleanly the first time (no cloud/ dir yet). | ||
| - name: Clear committed cloud baselines | ||
| shell: bash | ||
| run: rm -f browser_tests/fixtures/customNode/geometry/cloud/*.json | ||
|
|
||
| # CUSTOM_NODES_ENV=cloud routes geometry.ts's record writes into | ||
| # geometry/cloud/. Record mode throws by design after writing each file, | ||
| # so continue-on-error and retries=0 - the artifact is the product, not a | ||
| # green run. No CN_GEOMETRY_CORE: cloud has no pinned core SHA, so | ||
| # provenance stamps 'unrecorded' (allNodes.spec.ts); the row's deployRef | ||
| # is recorded as the pin. | ||
| - name: Record cloud geometry baselines | ||
| continue-on-error: true | ||
|
Contributor
There was a problem hiding this comment. Record artifacts can be partial or stale while looking complete. The geometry destination is cleared, but interaction profiles are not. All three producers use Clear every destination first, write into a fresh run-specific directory, and validate the exact manifest pack set, schema, provenance, and node counts before upload. Also note that the three step ceilings total 380 minutes before setup while the job ceiling is 350, so the job can be cancelled before uploads despite comments promising reachability. |
||
| timeout-minutes: 160 | ||
| env: | ||
| CUSTOM_NODES_ENV: cloud | ||
| CN_GEOMETRY: record | ||
| PLAYWRIGHT_TEST_URL: http://localhost:4173 | ||
| SMOKE_ACCOUNT_EMAIL: ${{ secrets.SMOKE_ACCOUNT_EMAIL }} | ||
| SMOKE_ACCOUNT_PASSWORD: ${{ secrets.SMOKE_ACCOUNT_PASSWORD }} | ||
| run: | | ||
| pnpm exec playwright test browser_tests/tests/customNodes/allNodes.spec.ts \ | ||
| --project=custom-nodes --reporter=list --workers=1 --retries=0 \ | ||
| --grep "all nodes:" | ||
|
|
||
| # S13 cloud profiles: same probes, same session; CUSTOM_NODES_ENV=cloud | ||
| # routes writes into interactionProfiles/cloud/. | ||
| - name: Record cloud interaction profiles (S13) | ||
| continue-on-error: true | ||
| timeout-minutes: 160 | ||
| env: | ||
| CUSTOM_NODES_ENV: cloud | ||
| CN_INTERACTION: record | ||
| PLAYWRIGHT_TEST_URL: http://localhost:4173 | ||
| SMOKE_ACCOUNT_EMAIL: ${{ secrets.SMOKE_ACCOUNT_EMAIL }} | ||
| SMOKE_ACCOUNT_PASSWORD: ${{ secrets.SMOKE_ACCOUNT_PASSWORD }} | ||
| run: | | ||
| pnpm exec playwright test browser_tests/tests/customNodes/interactionProfiles.spec.ts \ | ||
| --project=custom-nodes --reporter=list --workers=1 --retries=0 | ||
|
|
||
| # S15 cloud hashes: the download now runs in-page so it carries the | ||
| # signed-in session, which is what kept this tier core-only. Writes | ||
| # test-results/curatedOutputHashes.recorded.json; commit it as | ||
| # browser_tests/fixtures/data/curatedOutputHashes.cloud.json. | ||
| - name: Record cloud output hashes (S15) | ||
| continue-on-error: true | ||
| timeout-minutes: 60 | ||
| env: | ||
| CUSTOM_NODES_ENV: cloud | ||
| RECORD_OUTPUT_HASHES: '1' | ||
| PLAYWRIGHT_TEST_URL: http://localhost:4173 | ||
| SMOKE_ACCOUNT_EMAIL: ${{ secrets.SMOKE_ACCOUNT_EMAIL }} | ||
| SMOKE_ACCOUNT_PASSWORD: ${{ secrets.SMOKE_ACCOUNT_PASSWORD }} | ||
| run: | | ||
| pnpm exec playwright test browser_tests/tests/customNodes/customNode.regression.spec.ts \ | ||
| --project=custom-nodes --reporter=list --workers=1 --retries=0 | ||
|
|
||
| - name: Upload recorded cloud output hashes | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: custom-nodes-output-hashes-cloud | ||
| path: test-results/curatedOutputHashes.recorded.json | ||
| retention-days: 7 | ||
| if-no-files-found: warn | ||
|
|
||
| - name: Upload recorded cloud interaction profiles | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: custom-nodes-interaction-profiles-cloud | ||
| path: browser_tests/fixtures/customNode/interactionProfiles/cloud/ | ||
| retention-days: 7 | ||
| if-no-files-found: error | ||
|
|
||
| - name: Upload recorded cloud baselines | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: custom-nodes-geometry-baselines-cloud | ||
| path: browser_tests/fixtures/customNode/geometry/cloud/ | ||
| retention-days: 7 | ||
| if-no-files-found: error | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,164 @@ | ||
| # Records the custom-node geometry baselines | ||
| # (browser_tests/fixtures/customNode/geometry/<pack>.json) in the CI | ||
| # environment and uploads them as an artifact to commit. Recording MUST | ||
| # happen here, not on a dev machine: font metrics differ across platforms | ||
| # by whole pixels, and the baselines also encode pack-JS-built layout, so | ||
| # only this environment produces the numbers the gate compares against. | ||
| # Run it (manual dispatch) whenever a deliberate pin/core bump or an | ||
| # intended restyle legitimately moves layout, then commit the artifact in | ||
| # the same PR as the change that moved it. Procedure: | ||
| # docs/custom-node-regression-suite.md Step 5b. | ||
| name: 'Record Custom Nodes Geometry' | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| # workflow_dispatch cannot see a workflow that has never been on the | ||
| # default branch; pushing a record/custom-nodes-* ref runs this file | ||
| # from that ref instead. Delete the ref after collecting the artifact. | ||
| push: | ||
| branches: ['record/custom-nodes-*'] | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }} | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| record-geometry: | ||
| if: github.repository == 'Comfy-Org/ComfyUI_frontend' | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 60 | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup frontend | ||
| uses: ./.github/actions/setup-frontend | ||
| with: | ||
| include_build_step: true | ||
|
|
||
| - name: Setup Playwright | ||
| uses: ./.github/actions/setup-playwright | ||
|
|
||
| # Same pinned core as the gate: baselines must describe the exact | ||
| # world the gate tests. Bump this SHA together with the gate's | ||
| # comfyui_ref (ci-tests-custom-nodes.yaml) and the canary Job B copy. | ||
| - name: Setup ComfyUI server | ||
| uses: ./.github/actions/setup-comfyui-server | ||
| with: | ||
| launch_server: 'false' | ||
| comfyui_ref: 'b08e6cf35fac50d3ca8470dffb3f9a1fbb7187d2' | ||
|
|
||
| # Identical pinned-pack install to the gate. | ||
| - name: Install manifest custom nodes | ||
| shell: bash | ||
| run: | | ||
| set -euo pipefail | ||
| pip freeze | grep -iE '^(torch|torchvision|torchaudio)==' \ | ||
| > /tmp/torch-constraints.txt || true | ||
| manifest=browser_tests/fixtures/data/customNodeManifest.core.json | ||
| mapfile -t entries < <(jq -c '.[]' "$manifest") | ||
| for entry in "${entries[@]}"; do | ||
| repo=$(jq -r '.repo' <<<"$entry") | ||
| pin=$(jq -r '.pin' <<<"$entry") | ||
| pack=$(jq -r '.pack' <<<"$entry") | ||
| if ! [[ "$pack" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]]; then | ||
| echo "::error::unsafe pack name: '$pack'"; exit 1 | ||
| fi | ||
| if ! [[ "$pin" =~ ^[0-9a-f]{40}$ ]]; then | ||
| echo "::error::$pack: pin must be a full commit SHA (got '$pin')"; exit 1 | ||
| fi | ||
| dir="ComfyUI/custom_nodes/$pack" | ||
| echo "::group::install $pack" | ||
| git clone --depth 1 "$repo" "$dir" | ||
| git -C "$dir" fetch --depth 1 origin "$pin" | ||
| git -C "$dir" checkout "$pin" | ||
| if [ -f "$dir/requirements.txt" ]; then | ||
| pip install -r "$dir/requirements.txt" -c /tmp/torch-constraints.txt | ||
| fi | ||
| echo "::endgroup::" | ||
| done | ||
|
|
||
| - name: Stage run-tier assets | ||
| shell: bash | ||
| run: cp browser_tests/assets/plain_video.mp4 ComfyUI/input/plain_video.mp4 | ||
|
|
||
| - name: Start ComfyUI server | ||
| shell: bash | ||
| working-directory: ComfyUI | ||
| run: | | ||
| python main.py --cpu --multi-user --cache-none --front-end-root ../dist & | ||
| wait-for-it --service 127.0.0.1:8188 -t 600 | ||
|
|
||
| # The checkout brings the committed baselines along; delete them so | ||
| # the uploaded artifact is provably a product of THIS run and the | ||
| # if-no-files-found guard below can actually fire on a crashed | ||
| # record instead of shipping stale bytes as a fresh recording. | ||
| - name: Clear committed baselines | ||
| shell: bash | ||
| run: rm -f browser_tests/fixtures/customNode/geometry/*.json | ||
|
|
||
| # Record mode throws by design after writing each pack's file (a | ||
| # record must never look like a green test run), so the step is | ||
| # expected to exit nonzero; the artifact is the product. Retries are | ||
| # pointless against a deliberate throw, so they are off. | ||
| # CN_GEOMETRY_CORE stamps provenance into each baseline file and | ||
| # must match the comfyui_ref above. | ||
| - name: Record geometry baselines | ||
| continue-on-error: true | ||
| env: | ||
| CN_GEOMETRY: record | ||
| CN_GEOMETRY_CORE: b08e6cf35fac50d3ca8470dffb3f9a1fbb7187d2 | ||
| run: | | ||
| pnpm exec playwright test browser_tests/tests/customNodes/allNodes.spec.ts \ | ||
| --project=custom-nodes --reporter=list --workers=1 --retries=0 \ | ||
| --grep "all nodes:" | ||
|
|
||
| # S15 output hashes ride the same backend boot: curated workflows in | ||
| # record mode write test-results/curatedOutputHashes.recorded.json and | ||
| # deliberately fail each recording test (a record is not a pass). | ||
| - name: Record output hashes (S15) | ||
| continue-on-error: true | ||
| env: | ||
| RECORD_OUTPUT_HASHES: '1' | ||
| # Stamped into the fixture's recordedAt.core; must match comfyui_ref. | ||
| CN_OUTPUT_HASHES_CORE: b08e6cf35fac50d3ca8470dffb3f9a1fbb7187d2 | ||
| run: | | ||
| pnpm exec playwright test browser_tests/tests/customNodes/customNode.regression.spec.ts \ | ||
| --project=custom-nodes --reporter=list --workers=1 --retries=0 \ | ||
| --grep "Curated workflow execution: completes without error" | ||
|
|
||
| - name: Upload recorded baselines | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: custom-nodes-geometry-baselines | ||
| path: browser_tests/fixtures/customNode/geometry/ | ||
| retention-days: 7 | ||
| if-no-files-found: error | ||
|
|
||
| - name: Upload recorded output hashes | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: custom-nodes-output-hashes | ||
| path: test-results/curatedOutputHashes.recorded.json | ||
| retention-days: 7 | ||
| if-no-files-found: error | ||
| # S13 interaction profiles ride the same boot: record mode writes the | ||
| # per-pack fixture files and deliberately fails each test. | ||
| - name: Record interaction profiles (S13) | ||
| continue-on-error: true | ||
| env: | ||
| CN_INTERACTION: record | ||
| CN_INTERACTION_CORE: b08e6cf35fac50d3ca8470dffb3f9a1fbb7187d2 | ||
| run: | | ||
| pnpm exec playwright test browser_tests/tests/customNodes/interactionProfiles.spec.ts \ | ||
| --project=custom-nodes --reporter=list --workers=1 --retries=0 | ||
|
|
||
| - name: Upload recorded interaction profiles | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: custom-nodes-interaction-profiles | ||
| path: browser_tests/fixtures/customNode/interactionProfiles/ | ||
| retention-days: 7 | ||
| if-no-files-found: error |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Blocking — secret-bearing workflow executes branch-controlled code.
A collaborator can push
record/custom-nodes-cloud-*; this workflow then checks out that ref and runs its local actions, package scripts, Vite build, and Playwright code withSMOKE_ACCOUNT_EMAILandSMOKE_ACCOUNT_PASSWORDavailable. Repository write access does not imply permission to read shared account credentials.Hypothetical: a branch modifies
setup-frontendorbuild:cloud-e2eto POST environment values elsewhere. The run still looks like a recording run, and the exfiltration may leave no useful failure signal. Reviewers cannot differentiate a legitimate branch implementation from credential-stealing code after the fact.Remove the
pushescape hatch. Keep the workflow implementation on the trusted default branch and accept only inert inputs/artifacts, or use an environment with required approval and no arbitrary ref checkout. See GitHub's secure use reference.