Skip to content

Commit 0e3eb1e

Browse files
benjcooleybenjcooley
committed
ci: pin the corpus to commits, and make a stale pin impossible to miss
Tracking pack HEADs made every one of ~5,100 pack authors a committer to this repo's CI: one of them pushes a bug and the next unrelated PR goes red for it, with no signal telling whoever opened that PR it was not their diff. corpus.pins.json is checked in and names the commit of every pack the matrix measures, so a pack can only change through a reviewed bump. Bumping is manual - update-corpus-pins.yaml resolves every pack via git ls-remote (the REST API would exhaust its hourly budget in one run) and opens the PR where a month of ecosystem churn lands at once. A red there means the ecosystem moved; a red anywhere else means the diff did. The cost of pinning is that pack breakage is invisible until someone bumps, so that is the loudest thing in the run: a dependency-free pin-status job leads every run with the pin date, the age and the bump URL, and raises a warning annotation past 30 days. Not a hard failure - blocking every PR for something none of them caused is how a check gets routed around. The cache key is now the pins hash rather than the run id, so a PR gets an exact hit for the same commits and a guaranteed miss the moment they move. Measured over a 40-pack sample: 38 of 38 resolvable packs fetched at exactly their pinned commit, and zero fetch failures where the same sample at HEAD had five.
1 parent cdc846c commit 0e3eb1e

7 files changed

Lines changed: 5364 additions & 47 deletions

File tree

‎.github/workflows/ci-ecosystem-matrix.yaml‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,25 @@ env:
5858
CORPUS_CACHE_PREFIX: registry-corpus-v3-
5959

6060
jobs:
61+
# First job in the run, and deliberately dependency-free so its annotation
62+
# and step summary land at the top. Stale pins are the one failure mode this
63+
# instrument cannot detect from the inside: everything stays green while the
64+
# ecosystem it claims to measure moves on without it.
65+
pin-status:
66+
runs-on: ubuntu-latest
67+
timeout-minutes: 5
68+
steps:
69+
- name: Checkout repository
70+
uses: actions/checkout@v7
71+
72+
- name: Setup Python
73+
uses: actions/setup-python@v6
74+
with:
75+
python-version: ${{ env.PYTHON_VERSION }}
76+
77+
- name: How old is the corpus, and how do I bump it
78+
run: python3 scripts/registry-census/pins.py
79+
6180
corpus:
6281
runs-on: ubuntu-latest
6382
timeout-minutes: 30
@@ -77,14 +96,15 @@ jobs:
7796
with:
7897
python-version: ${{ env.PYTHON_VERSION }}
7998

80-
# Restore the newest corpus regardless of key: the primary key never
81-
# hits (run-scoped), so restore-keys always serves the latest save.
99+
# Keyed on the pins, so it is an exact hit for every run measuring the
100+
# same commits and a guaranteed miss the moment they are bumped. The
101+
# prefix fallback serves a cold PR something usable rather than nothing.
82102
- name: Restore corpus cache
83103
id: restore
84104
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
85105
with:
86106
path: .census
87-
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ github.run_id }}
107+
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('scripts/registry-census/corpus.pins.json') }}
88108
restore-keys: ${{ env.CORPUS_CACHE_PREFIX }}
89109

90110
# Re-pin the registry snapshot every run: it is the corpus's target
@@ -99,16 +119,13 @@ jobs:
99119
# and unpacks archives; pack code executes in ecosystem-matrix, which
100120
# checks out with persist-credentials: false.
101121
#
102-
# --revalidate only off main. On a PR the restored corpus is the
103-
# measurement's control: refetching drifted packs there would move the
104-
# population out from under the baseline it is being compared to.
122+
# Fetches at the commits named in corpus.pins.json, so this only does
123+
# real work on a cold cache or a pin bump. Prints the pin banner first.
105124
- name: Fetch corpus
106125
id: fetch
107126
env:
108127
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
109-
run: |
110-
python3 scripts/registry-census/fetch_corpus.py \
111-
${{ github.event_name == 'pull_request' && ' ' || '--revalidate' }}
128+
run: python3 scripts/registry-census/fetch_corpus.py
112129

113130
# corpus.lock.json records the ETag and resolved tree of every pack this
114131
# run measured - the identity a published figure has to be cited with.

‎.github/workflows/update-corpus-pins.yaml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Bump the ecosystem matrix's corpus pins.
2+
#
3+
# Manual only, and that is the design. scripts/registry-census/corpus.pins.json
4+
# names the exact commit of every registry pack the matrix measures, so the
5+
# gate is a constant: a pack author pushing a bug cannot red an unrelated PR.
6+
# The cost of that is that pack breakage is invisible until someone bumps the
7+
# pins - so this workflow opens the PR where a whole month of ecosystem churn
8+
# lands at once, reviewed, by someone who expected it.
9+
#
10+
# A red on THAT pr means the ecosystem moved. A red anywhere else means the
11+
# diff did. Keeping those two apart is the entire point.
12+
name: 'Update corpus pins'
13+
14+
on:
15+
workflow_dispatch:
16+
17+
permissions:
18+
contents: read
19+
20+
jobs:
21+
update-pins:
22+
if: github.repository == 'Comfy-Org/ComfyUI_frontend'
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 45
25+
permissions:
26+
contents: write
27+
pull-requests: write
28+
steps:
29+
- name: Checkout repository
30+
uses: actions/checkout@v7
31+
32+
- name: Setup Python
33+
uses: actions/setup-python@v6
34+
with:
35+
python-version: '3.11'
36+
37+
- name: Report the pins being replaced
38+
run: python3 scripts/registry-census/pins.py
39+
40+
- name: Refresh registry snapshot
41+
run: python3 scripts/registry-census/refresh_registry.py
42+
43+
# git ls-remote, not the REST API: ~5,100 packs would exhaust the
44+
# authenticated 5,000/hour REST budget in a single run.
45+
- name: Resolve every pack to a commit
46+
env:
47+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
48+
run: python3 scripts/registry-census/fetch_corpus.py --write-pins --workers 32
49+
50+
- name: Summarize the bump
51+
id: summary
52+
run: |
53+
set -euo pipefail
54+
python3 - <<'PY' >> "$GITHUB_STEP_SUMMARY"
55+
import json, subprocess
56+
path = 'scripts/registry-census/corpus.pins.json'
57+
new = json.load(open(path))['packs']
58+
old = json.loads(
59+
subprocess.run(['git', 'show', f'HEAD:{path}'],
60+
capture_output=True, text=True).stdout or '{"packs":{}}'
61+
)['packs']
62+
moved = sorted(k for k in new.keys() & old.keys() if new[k] != old[k])
63+
added = sorted(new.keys() - old.keys())
64+
dropped = sorted(old.keys() - new.keys())
65+
print(f'## Corpus pin bump\n')
66+
print(f'- {len(new)} packs pinned')
67+
print(f'- {len(moved)} moved, {len(added)} added, {len(dropped)} dropped\n')
68+
if moved:
69+
print('<details><summary>moved</summary>\n')
70+
print('\n'.join(f'- `{k}` {old[k][:8]} -> {new[k][:8]}' for k in moved))
71+
print('\n</details>')
72+
PY
73+
74+
- name: Open the bump PR
75+
uses: peter-evans/create-pull-request@v7
76+
with:
77+
branch: chore/corpus-pins
78+
commit-message: 'chore: bump ecosystem matrix corpus pins'
79+
title: 'chore: bump ecosystem matrix corpus pins'
80+
body: |
81+
Regenerates `scripts/registry-census/corpus.pins.json`, which names the
82+
exact commit of every registry pack the ecosystem matrix measures.
83+
84+
**A red `CI: Ecosystem Matrix` on this PR means the ecosystem moved, not
85+
that this diff broke something.** That is what this PR is for. Read the
86+
verdict's breached criterion and the per-pack signature drift, decide
87+
whether it is real breakage worth an upstream issue or acceptable churn,
88+
and merge either way — leaving the pins stale is strictly worse, because
89+
it makes the gate measure an ecosystem that no longer exists.
90+
91+
See the step summary on the triggering run for the moved/added/dropped
92+
breakdown.
93+
add-paths: scripts/registry-census/corpus.pins.json

‎.oxfmtrc.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
"ignorePatterns": [
99
"packages/registry-types/src/comfyRegistryTypes.ts",
1010
"public/materialdesignicons.min.css",
11+
"scripts/registry-census/corpus.pins.json",
1112
"scripts/registry-census/detection-proof/**",
1213
"src/__ecs_matrix__/**",
1314
"src/types/generatedManagerTypes.ts",

‎scripts/registry-census/README.md‎

Lines changed: 49 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -27,20 +27,52 @@ shared 10GB Actions budget, which evicted itself mid-run — run 31753242605
2727
saved the entry, shard 2 restored it, and shards 1/3/4 found no corpus at
2828
all.
2929

30-
Each pack's tarball ETag and resolved tree name are recorded in
31-
`corpus.lock.json`. codeload names the archive root `<repo>-<committish>`, so
32-
for a HEAD fetch that name is the commit actually measured — which is what
33-
makes a red delta diffable ("these 31 packs moved since the last green run")
34-
rather than an unattributable number.
30+
## Pins — read this before wondering why a pack looks stale
31+
32+
`corpus.pins.json` is **checked in** and names the exact commit of every pack
33+
the matrix measures. The fetch uses those commits, not `HEAD`.
34+
35+
This is the difference between a PR gate and a liability. Tracking pack HEADs
36+
makes every one of ~5,100 pack authors a committer to this repo's CI: one of
37+
them pushes a bug at 3am and the next unrelated PR goes red for it, and the
38+
person who has to work that out is whoever opened it. Pinned, a pack can only
39+
change through a reviewed bump.
40+
41+
**Bumping is manual, and currently the only thing keeping the pins honest.**
42+
43+
- Workflow: [`update-corpus-pins.yaml`](https://github.com/Comfy-Org/ComfyUI_frontend/actions/workflows/update-corpus-pins.yaml) (`workflow_dispatch`) — resolves every
44+
pack, opens a PR, and summarizes what moved.
45+
- Locally: `python3 scripts/registry-census/fetch_corpus.py --write-pins`
46+
47+
**A red `CI: Ecosystem Matrix` on a pin-bump PR means the ecosystem moved, not
48+
that the diff broke something.** That is what the bump PR is for. Anywhere
49+
else, a red means the diff. Keeping those two apart is the whole point.
50+
51+
Every matrix run opens with a `pin-status` job printing the pin date, the age,
52+
and the bump URL, and raises a `::warning::` annotation once the pins are more
53+
than **30 days** old. It does not fail the build — a stale-pin failure would
54+
block everyone for something nobody's PR caused — but it is the loudest thing
55+
in the run, because a stale corpus is the one defect this instrument cannot
56+
detect from the inside. Everything stays green while the ecosystem it claims
57+
to measure moves on without it.
58+
59+
Packs registered since the last bump have no pin and track their registry ref
60+
until the next one, rather than dropping out of the population.
61+
62+
Each pack's tarball ETag and the ref actually fetched are recorded in
63+
`corpus.lock.json`, which ships as a run artifact — that is the identity a
64+
published figure has to be cited with, and diffing two runs' lockfiles gives
65+
you the packs that moved between them.
3566

3667
## Running locally
3768

3869
```bash
39-
python3 scripts/registry-census/refresh_registry.py # pin the registry snapshot
40-
python3 scripts/registry-census/fetch_corpus.py # fetch missing packs
41-
python3 scripts/registry-census/fetch_corpus.py --revalidate # also refetch drifted packs
42-
python3 scripts/registry-census/fetch_corpus.py --limit 50 # smoke test
43-
python3 -m unittest discover -s scripts/registry-census # verdict unit tests
70+
python3 scripts/registry-census/pins.py # how old is the corpus
71+
python3 scripts/registry-census/refresh_registry.py # re-pin the registry snapshot
72+
python3 scripts/registry-census/fetch_corpus.py # fetch at the pinned commits
73+
python3 scripts/registry-census/fetch_corpus.py --limit 50 # smoke test
74+
python3 scripts/registry-census/fetch_corpus.py --write-pins # bump the pins
75+
python3 -m unittest discover -s scripts/registry-census # verdict unit tests
4476
```
4577

4678
Pure stdlib; needs `curl` and `tar` on PATH.
@@ -50,13 +82,14 @@ Pure stdlib; needs `curl` and `tar` on PATH.
5082
`.github/workflows/ci-ecosystem-matrix.yaml`. **Corpus freshness is a
5183
push-to-main concern, not a PR concern:**
5284

53-
| event | corpus | metrics |
54-
| -------------- | ----------------------------------------- | -------------------- |
55-
| `push` to main | ETag-revalidated, drifted packs refetched | baseline **written** |
56-
| `pull_request` | main's corpus restored and used as-is | baseline **read** |
85+
| event | corpus | metrics |
86+
| -------------- | ---------------------------------------------- | -------------------- |
87+
| `push` to main | fetched at the pinned commits, cache **saved** | baseline **written** |
88+
| `pull_request` | main's cached corpus, restored as-is | baseline **read** |
5789

58-
Both sides of the comparison then measure the same packs, so a red delta is
59-
attributable to the diff rather than to a pack that moved overnight. Actions
90+
Both sides measure the same packs — the pins guarantee it across cache
91+
evictions, the restore-without-refetch guarantees it within a run — so a red
92+
delta is attributable to the diff rather than to a pack that moved. Actions
6093
cache scoping also means only a default-branch run can produce an entry other
6194
branches restore, so a PR never writes one.
6295

0 commit comments

Comments
 (0)