Skip to content

Commit 7cbfc7e

Browse files
committed
test: report pack pin age, and give the bump a command and an action
The suite already pinned every pack at a commit, which is what stops a pack author's push from redding an unrelated PR. What it had no answer for was the cost of pinning: breakage stays invisible between bumps, there was no way to bump other than editing forty characters of JSON by hand, and nothing said when a pin had gone stale. Each pin now carries pinnedAt. `pnpm custom-node-pins` reports the age of each and whether upstream has moved; `pnpm custom-node-pins:update` rewrites them; update-custom-node-pins.yaml does the same and opens the PR. Every suite run reports pin age and warns past 30 days - never fails, because blocking every PR over a pin nobody's diff touched is how a gate gets routed around. No pin changes here, only dates: all six were verified today, and five are at their upstream HEAD. KJNodes is 36 commits behind and is left for a deliberate bump, since expectedNodeCount and expectedExtensions are calibrated against the pinned source and would need recalibrating with it.
1 parent bfa059b commit 7cbfc7e

7 files changed

Lines changed: 327 additions & 0 deletions

File tree

.github/workflows/ci-tests-custom-nodes.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,16 @@ jobs:
130130
with:
131131
include_build_step: true
132132

133+
# Pinning is what stops a pack author's 3am push from redding an
134+
# unrelated PR. The cost is that pack breakage stays invisible until
135+
# someone bumps, and nothing used to say when - so this reports the age
136+
# of every pin and warns past 30 days. Never fails the job: blocking
137+
# every PR over a pin nobody's diff touched is how a gate gets routed
138+
# around.
139+
- name: How old are the pack pins
140+
continue-on-error: true
141+
run: pnpm custom-node-pins
142+
133143
- name: Setup Playwright
134144
uses: ./.github/actions/setup-playwright
135145

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Bump the custom-node suite's pack pins.
2+
#
3+
# Manual, and that is the design. customNodeManifest.core.json names the exact
4+
# commit of every pack the suite installs, so no git surface moves underneath a
5+
# PR: unpinned, every pack author is a committer to this repo's CI and a bug
6+
# pushed to any of them reds the next unrelated PR.
7+
#
8+
# The cost is the mirror image - pack breakage stays invisible until someone
9+
# bumps - so this opens the PR where it all lands at once, reviewed, by someone
10+
# who expected it. A red suite on THAT pr means the ecosystem moved. A red
11+
# anywhere else means the diff did.
12+
name: 'Update custom-node pins'
13+
14+
on:
15+
workflow_dispatch:
16+
17+
permissions:
18+
contents: read
19+
20+
jobs:
21+
update-pins:
22+
if: github.repository == 'Comfy-Org/ComfyUI_frontend'
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 15
25+
permissions:
26+
contents: write
27+
pull-requests: write
28+
steps:
29+
- name: Checkout repository
30+
uses: actions/checkout@v7
31+
32+
- name: Setup frontend
33+
uses: ./.github/actions/setup-frontend
34+
with:
35+
include_build_step: false
36+
37+
- name: Report the pins being replaced
38+
run: pnpm custom-node-pins
39+
40+
- name: Bump every pin to upstream HEAD
41+
run: pnpm custom-node-pins:update | tee bump.txt
42+
43+
- name: Open the bump PR
44+
uses: peter-evans/create-pull-request@22a9089034f40e5a961c8808d113e2c98fb63676 # v7.0.11
45+
with:
46+
branch: chore/custom-node-pins
47+
commit-message: 'chore: bump custom-node suite pack pins'
48+
title: 'chore: bump custom-node suite pack pins'
49+
body: |
50+
Regenerates the `pin` and `pinnedAt` fields in
51+
`browser_tests/fixtures/data/customNodeManifest.core.json`.
52+
53+
**`CI: Custom Nodes` is expected to red on this PR, and that is the
54+
point.** `expectedNodeCount` and `expectedExtensions` are calibrated
55+
against the pinned source, and the manifest is deliberate that any
56+
delta — either direction — fails until it is recalibrated. A red here
57+
is the suite telling you what changed in the ecosystem since the last
58+
bump; a red anywhere else means the diff did it.
59+
60+
To land this: read the failures, decide whether each is real breakage
61+
worth an upstream issue or acceptable churn, recalibrate the counts,
62+
and merge. Leaving the pins stale is strictly worse — it makes the
63+
suite measure an ecosystem that no longer exists.
64+
65+
See the triggering run's log for which pins moved.
66+
add-paths: browser_tests/fixtures/data/customNodeManifest.core.json

browser_tests/fixtures/data/customNodeManifest.core.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
"pack": "ComfyUI-Impact-Pack",
44
"repo": "https://github.com/ltdrdata/ComfyUI-Impact-Pack",
55
"pin": "429d0159ad429e64d2b3916e6e7be9c22d025c3c",
6+
"pinnedAt": "2026-08-14",
67
"tiers": ["load", "connectivity", "run"],
78
"workflow": "assets/customNodes/impact_primitives_run.json",
89
"expectedNodes": ["ImpactInt", "ImpactFloat"],
@@ -25,6 +26,7 @@
2526
"pack": "ComfyUI-VideoHelperSuite",
2627
"repo": "https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite",
2728
"pin": "4ee72c065db22c9d96c2427954dc69e7b908444b",
29+
"pinnedAt": "2026-08-14",
2830
"tiers": ["load", "connectivity", "run"],
2931
"workflow": "assets/customNodes/vhs_video_pipeline_run.json",
3032
"expectedNodes": ["VHS_LoadVideoPath", "VHS_VideoInfo"],
@@ -46,6 +48,7 @@
4648
"pack": "ComfyUI_essentials",
4749
"repo": "https://github.com/cubiq/ComfyUI_essentials",
4850
"pin": "9d9f4bedfc9f0321c19faf71855e228c93bd0dc9",
51+
"pinnedAt": "2026-08-14",
4952
"tiers": ["load", "connectivity", "run"],
5053
"workflow": "assets/customNodes/essentials_math_display_run.json",
5154
"expectedNodes": ["SimpleMathInt+", "DisplayAny"],
@@ -69,6 +72,7 @@
6972
"pack": "ComfyUI-KJNodes",
7073
"repo": "https://github.com/kijai/ComfyUI-KJNodes",
7174
"pin": "e27a505b3ba6ce42687fe00500deda103d9d6071",
75+
"pinnedAt": "2026-08-14",
7276
"tiers": ["load", "connectivity", "run"],
7377
"workflow": "assets/customNodes/kjnodes_constants_run.json",
7478
"expectedNodes": ["INTConstant", "FloatConstant"],
@@ -113,6 +117,7 @@
113117
"pack": "ComfyUI-Custom-Scripts",
114118
"repo": "https://github.com/pythongosssss/ComfyUI-Custom-Scripts",
115119
"pin": "609f3afaa74b2f88ef9ce8d939626065e3247469",
120+
"pinnedAt": "2026-08-14",
116121
"tiers": ["load", "connectivity", "run"],
117122
"workflow": "assets/customNodes/customscripts_string_show_run.json",
118123
"expectedNodes": ["StringFunction|pysssss", "ShowText|pysssss"],
@@ -127,6 +132,7 @@
127132
"pack": "was-node-suite-comfyui",
128133
"repo": "https://github.com/WASasquatch/was-node-suite-comfyui",
129134
"pin": "ea935d1044ae5a26efa54ebeb18fe9020af49a45",
135+
"pinnedAt": "2026-08-14",
130136
"tiers": ["load", "connectivity", "run"],
131137
"workflow": "assets/customNodes/was_number_text_run.json",
132138
"expectedNodes": ["Constant Number", "Number to Text", "Text to Console"],

browser_tests/tests/customNodes/README.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,35 @@ pack onboarding, and the detection proof - lives in the Technical Design
99
Doc: [docs/custom-node-regression-suite.md](../../../docs/custom-node-regression-suite.md).
1010

1111
Quick start: `pnpm test:custom-nodes:local`.
12+
13+
## Pack pins
14+
15+
Every pack is installed at a fixed commit, checked in as `pin` in
16+
[`customNodeManifest.core.json`](../../fixtures/data/customNodeManifest.core.json)
17+
alongside `pinnedAt`, the date it was last verified. ComfyUI core is pinned
18+
the same way, via `comfyui_ref` in `ci-tests-custom-nodes.yaml`.
19+
20+
This is what stops a pack author's push from redding an unrelated PR.
21+
Unpinned, all six pack maintainers are effectively committers to this repo's
22+
CI. The cost is the mirror image - pack breakage is invisible between bumps -
23+
so bumping is a deliberate, reviewed act rather than something that happens to
24+
you.
25+
26+
```bash
27+
pnpm custom-node-pins # age of each pin, and whether upstream moved
28+
pnpm custom-node-pins:update # rewrite every pin to upstream HEAD
29+
```
30+
31+
Or run [`update-custom-node-pins.yaml`](https://github.com/Comfy-Org/ComfyUI_frontend/actions/workflows/update-custom-node-pins.yaml)
32+
(`workflow_dispatch`), which does the same thing and opens the PR.
33+
34+
Every suite run reports pin age and warns past **30 days**. It never fails on
35+
staleness alone: blocking every PR over a pin nobody's diff touched is how a
36+
gate gets routed around.
37+
38+
**A bump is expected to red the suite, and that is the point.**
39+
`expectedNodeCount` and `expectedExtensions` are calibrated against the pinned
40+
source, and the manifest is deliberate that any delta - either direction -
41+
fails until it is recalibrated. A red on a bump PR is the suite telling you
42+
what changed in the ecosystem since the last one. A red anywhere else means
43+
the diff did it. Keeping those two apart is the entire reason to pin.

package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@
5656
"test:browser": "pnpm exec playwright test",
5757
"test:browser:coverage": "cross-env COLLECT_COVERAGE=true pnpm test:browser",
5858
"test:browser:local": "cross-env PLAYWRIGHT_LOCAL=1 PLAYWRIGHT_TEST_URL=http://localhost:5173 pnpm test:browser",
59+
"custom-node-pins": "tsx scripts/customNodePins.ts",
60+
"custom-node-pins:update": "tsx scripts/customNodePins.ts --write",
5961
"test:custom-nodes": "cross-env PLAYWRIGHT_TEST_URL=http://localhost:5173 pnpm exec playwright test browser_tests/tests/customNodes/ --config playwright.chrome.config.ts --workers=1",
6062
"test:custom-nodes:ci": "cross-env PLAYWRIGHT_TEST_URL=http://localhost:8188 pnpm exec playwright test browser_tests/tests/customNodes/ --config playwright.chrome.config.ts --workers=1",
6163
"test:custom-nodes:watch": "cross-env PLAYWRIGHT_TEST_URL=http://localhost:5173 PLAYWRIGHT_LOCAL=1 SLOW_MO=300 pnpm exec playwright test browser_tests/tests/customNodes/customNode.regression.spec.ts browser_tests/tests/customNodes/connectivity.spec.ts --config playwright.chrome.config.ts --workers=1 --headed",

scripts/customNodePins.test.ts

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import { describe, expect, it } from 'vitest'
2+
3+
import { MAX_AGE_DAYS, ageInDays, stalest } from './customNodePins'
4+
5+
const at = (iso: string) => new Date(`${iso}T00:00:00Z`)
6+
7+
describe('ageInDays', () => {
8+
it('counts whole days between the pin and today', () => {
9+
expect(ageInDays('2026-08-01', at('2026-08-31'))).toBe(30)
10+
})
11+
12+
it('reports null rather than 0 for an undated pin', () => {
13+
expect(ageInDays(undefined, at('2026-08-31'))).toBeNull()
14+
})
15+
16+
it('reports null for a date it cannot parse', () => {
17+
expect(ageInDays('last tuesday', at('2026-08-31'))).toBeNull()
18+
})
19+
})
20+
21+
describe('stalest', () => {
22+
const dated = (pinnedAt?: string) => ({
23+
pack: 'p',
24+
repo: 'r',
25+
pin: 'sha',
26+
pinnedAt
27+
})
28+
29+
it('is the oldest pin, not the newest, because that is what freshness is worth', () => {
30+
expect(
31+
stalest(
32+
[dated('2026-08-30'), dated('2026-07-01'), dated('2026-08-29')],
33+
at('2026-08-31')
34+
)
35+
).toBe(61)
36+
})
37+
38+
it('refuses to answer when any pin is undated', () => {
39+
expect(stalest([dated('2026-08-30'), dated()], at('2026-08-31'))).toBeNull()
40+
})
41+
42+
it('breaches the limit one day past it, not on it', () => {
43+
const onLimit = stalest([dated('2026-08-01')], at('2026-08-31'))
44+
const pastLimit = stalest([dated('2026-08-01')], at('2026-09-01'))
45+
expect(onLimit).toBe(MAX_AGE_DAYS)
46+
expect(pastLimit).toBeGreaterThan(MAX_AGE_DAYS)
47+
})
48+
})

scripts/customNodePins.ts

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
/**
2+
* Reports and bumps the commit each manifest pack is pinned at.
3+
*
4+
* The suite installs custom-node packs at fixed commits so no git surface
5+
* moves underneath a PR. That is the right call - unpinned, every pack author
6+
* becomes a committer to this repo's CI, and a bug pushed to any of them reds
7+
* the next unrelated PR. The cost is the mirror image: pack breakage is
8+
* invisible until someone bumps, and nothing was telling anyone when to.
9+
*
10+
* pnpm custom-node-pins report age and whether upstream moved
11+
* pnpm custom-node-pins:update rewrite the pins to upstream HEAD
12+
*
13+
* A bump is expected to red the suite. `expectedNodeCount` and
14+
* `expectedExtensions` are calibrated against the pinned source, and the
15+
* manifest is explicit that any delta fails until it is deliberately
16+
* recalibrated. That failure is the suite telling you what changed in the
17+
* ecosystem, which is the whole reason to bump on purpose rather than drift.
18+
*/
19+
import { execFile } from 'node:child_process'
20+
import { readFileSync, writeFileSync } from 'node:fs'
21+
import { fileURLToPath } from 'node:url'
22+
import { promisify } from 'node:util'
23+
24+
const run = promisify(execFile)
25+
26+
// Lazy: import.meta.url is not a file: URL under vitest, so resolving this at
27+
// module scope makes the file unimportable by its own test.
28+
function manifestPath(): string {
29+
return fileURLToPath(
30+
new URL(
31+
'../browser_tests/fixtures/data/customNodeManifest.core.json',
32+
import.meta.url
33+
)
34+
)
35+
}
36+
37+
export const MAX_AGE_DAYS = 30
38+
const UPDATE_WORKFLOW =
39+
'https://github.com/Comfy-Org/ComfyUI_frontend/actions/workflows/update-custom-node-pins.yaml'
40+
41+
interface PinnedPack {
42+
pack: string
43+
repo: string
44+
pin: string
45+
pinnedAt?: string
46+
}
47+
48+
/** Whole days elapsed, or null when the pin carries no date yet. */
49+
export function ageInDays(
50+
pinnedAt: string | undefined,
51+
today: Date
52+
): number | null {
53+
if (!pinnedAt) return null
54+
const then = Date.parse(`${pinnedAt}T00:00:00Z`)
55+
if (Number.isNaN(then)) return null
56+
return Math.floor((today.getTime() - then) / 86_400_000)
57+
}
58+
59+
/** The oldest pin is what the suite's freshness is actually worth. */
60+
export function stalest(packs: PinnedPack[], today: Date): number | null {
61+
const ages = packs.map((p) => ageInDays(p.pinnedAt, today))
62+
if (ages.some((a) => a === null)) return null
63+
return Math.max(...(ages as number[]))
64+
}
65+
66+
async function headSha(repo: string): Promise<string> {
67+
try {
68+
const { stdout } = await run('git', ['ls-remote', repo, 'HEAD'], {
69+
timeout: 60_000
70+
})
71+
return stdout.split(/\s/)[0] ?? ''
72+
} catch {
73+
return ''
74+
}
75+
}
76+
77+
function load(): PinnedPack[] {
78+
return JSON.parse(readFileSync(manifestPath(), 'utf8')) as PinnedPack[]
79+
}
80+
81+
function today(): string {
82+
return new Date().toISOString().slice(0, 10)
83+
}
84+
85+
function say(line: string): void {
86+
process.stdout.write(`${line}\n`)
87+
}
88+
89+
async function report(): Promise<number> {
90+
const packs = load()
91+
const now = new Date()
92+
const heads = await Promise.all(packs.map((p) => headSha(p.repo)))
93+
94+
say('='.repeat(72))
95+
const age = stalest(packs, now)
96+
say(
97+
age === null
98+
? 'CUSTOM-NODE PINS: no pin dates recorded - freshness unknown'
99+
: `custom-node pins: oldest is ${age} day(s) old (limit ${MAX_AGE_DAYS})`
100+
)
101+
say(` update via ${UPDATE_WORKFLOW}`)
102+
say(' or locally pnpm custom-node-pins:update')
103+
say('='.repeat(72))
104+
105+
for (const [i, p] of packs.entries()) {
106+
const days = ageInDays(p.pinnedAt, now)
107+
const moved = heads[i] && heads[i] !== p.pin
108+
say(
109+
` ${p.pack.padEnd(28)} ${p.pin.slice(0, 10)}` +
110+
` ${p.pinnedAt ?? 'undated'}` +
111+
`${days === null ? '' : ` (${days}d)`}` +
112+
` ${!heads[i] ? 'upstream unreachable' : moved ? 'UPSTREAM MOVED' : 'at upstream HEAD'}`
113+
)
114+
}
115+
116+
if (age !== null && age <= MAX_AGE_DAYS) return 0
117+
const summary =
118+
age === null
119+
? `custom-node pins carry no date - bump them at ${UPDATE_WORKFLOW}`
120+
: `custom-node pins are ${age} days old - bump them at ${UPDATE_WORKFLOW}`
121+
say(`::warning title=Custom-node pins are stale::${summary}`)
122+
return 0
123+
}
124+
125+
async function update(): Promise<number> {
126+
const packs = load()
127+
const stamp = today()
128+
const heads = await Promise.all(packs.map((p) => headSha(p.repo)))
129+
130+
const unreachable = packs.filter((_, i) => !heads[i]).map((p) => p.pack)
131+
if (unreachable.length) {
132+
process.stderr.write(`could not resolve: ${unreachable.join(', ')}\n`)
133+
return 1
134+
}
135+
136+
const moved = packs.filter((p, i) => heads[i] !== p.pin)
137+
const next = packs.map((p, i) => ({
138+
...p,
139+
pin: heads[i],
140+
pinnedAt: heads[i] === p.pin ? (p.pinnedAt ?? stamp) : stamp
141+
}))
142+
writeFileSync(manifestPath(), `${JSON.stringify(next, null, 2)}\n`)
143+
144+
for (const p of moved) {
145+
const to = heads[packs.indexOf(p)]
146+
say(` ${p.pack.padEnd(28)} ${p.pin.slice(0, 10)} -> ${to.slice(0, 10)}`)
147+
}
148+
say(
149+
`${moved.length} of ${packs.length} pins moved; recalibrate` +
150+
' expectedNodeCount / expectedExtensions if the suite reds'
151+
)
152+
return 0
153+
}
154+
155+
const invokedDirectly =
156+
process.argv[1] !== undefined &&
157+
import.meta.url === new URL(`file://${process.argv[1]}`).href
158+
159+
if (invokedDirectly) {
160+
process.exitCode = await (process.argv.includes('--write')
161+
? update()
162+
: report())
163+
}

0 commit comments

Comments
 (0)