Skip to content

Commit 471bc99

Browse files
committed
ci: one cache key for the corpus, and let a PR populate it
Three keys disagreed. The corpus job restored on the pins hash and saved on the lockfile hash, so the exact key could never hit; the shards restored on a run-scoped key and fell back to the bare prefix. And the save was gated to the default branch, so on a PR the shards received nothing at all - run 31827717441, all four shards dead on 'corpus missing'. All three now key on the pin set, which is the only thing that identifies which packs a corpus contains. The prefix fallbacks are gone with them: a fallback hands back a corpus pinned to different commits, and since a pinned fetch treats an already-present pack as done, nothing downstream would correct it. The shards fail closed on a miss instead. Saving from a PR is affordable now for reasons that were not true of the version that evicted itself: ~0.32GB rather than 6.0GB, and one entry per pin bump rather than one per run that saw drift.
1 parent 43b2f9c commit 471bc99

1 file changed

Lines changed: 30 additions & 22 deletions

File tree

.github/workflows/ci-ecosystem-matrix.yaml

Lines changed: 30 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -96,16 +96,17 @@ jobs:
9696
with:
9797
python-version: ${{ env.PYTHON_VERSION }}
9898

99-
# Keyed on the pins, so it is an exact hit for every run measuring the
100-
# same commits and a guaranteed miss the moment they are bumped. The
101-
# prefix fallback serves a cold PR something usable rather than nothing.
99+
# Exact key only, deliberately no restore-keys. The key IS the pin set,
100+
# so a prefix fallback would hand back a corpus pinned to different
101+
# commits - and since a pinned fetch treats an already-present pack as
102+
# done, nothing downstream would correct it. Better a 4-minute cold
103+
# fetch than a green verdict over the wrong packs.
102104
- name: Restore corpus cache
103105
id: restore
104106
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
105107
with:
106108
path: .census
107109
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('scripts/registry-census/corpus.pins.json') }}
108-
restore-keys: ${{ env.CORPUS_CACHE_PREFIX }}
109110

110111
# Re-pin the registry snapshot every run: it is the corpus's target
111112
# list, and new packs only enter the corpus through it.
@@ -139,29 +140,33 @@ jobs:
139140
retention-days: 30
140141
if-no-files-found: warn
141142

142-
# Key derived from the lockfile: unchanged corpus finds its key taken
143-
# and the server rejects the duplicate; any pack drift writes a new
144-
# entry. Gated on the FETCH step and a present lockfile: an early
145-
# failure must not save a partial .census - least of all under the
146-
# empty-hash key, which equals the restore-keys prefix and would
147-
# become the best match for every later run.
143+
# Same key as the restore, which is the PIN SET - so there is exactly
144+
# one entry per pin bump rather than one per run that saw drift, and
145+
# concurrent runs on the same pins race for one key with the loser
146+
# rejected as a duplicate.
148147
#
149-
# Default branch only. A PR that saved would spend the repo's shared
150-
# cache budget on an entry no other branch can restore.
148+
# Saved on any event, not just main. The shards are separate jobs and
149+
# this cache is the only thing that carries the corpus to them: gating
150+
# the save to the default branch meant a PR could never run the matrix
151+
# at all, which is what reddened run 31827717441. Affordable now that
152+
# the corpus is ~0.32GB rather than the 6.0GB that evicted itself.
153+
#
154+
# Gated on the FETCH step and a present lockfile: an early failure must
155+
# not save a partial .census.
151156
- name: Save corpus cache
152157
if: |
153-
github.event_name != 'pull_request'
154-
&& github.ref_name == github.event.repository.default_branch
158+
steps.restore.outputs.cache-hit != 'true'
155159
&& steps.fetch.outcome == 'success'
156160
&& hashFiles('.census/corpus.lock.json') != ''
157161
uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
158162
with:
159163
path: .census
160-
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('.census/corpus.lock.json') }}
164+
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('scripts/registry-census/corpus.pins.json') }}
161165

162-
# Each drift writes a new entry against the repo's shared 10GB budget; keep
163-
# the newest only. Its own job so `corpus`, which holds a token while
164-
# extracting untrusted archives, stays at contents: read.
166+
# One entry per pin bump against the repo's shared 10GB budget; keep the
167+
# newest only. Branch-scoped, so a PR's entry is never deleted out from
168+
# under it - those expire with the PR. Its own job so `corpus`, which holds
169+
# a token while extracting untrusted archives, stays at contents: read.
165170
prune-corpus-cache:
166171
needs: corpus
167172
if: |
@@ -226,14 +231,17 @@ jobs:
226231
with:
227232
python-version: ${{ env.PYTHON_VERSION }}
228233

229-
# The corpus job above saved the cache this run; same-run caches are
230-
# visible to later jobs. Restore-only - the matrix never writes it.
234+
# The corpus job saved this exact key; same-run caches are visible to
235+
# later jobs. Restore-only, exact key, and fail-on-cache-miss: a prefix
236+
# fallback here would hand the shard a corpus pinned to different
237+
# commits, and `corpus missing` further down is a much clearer failure
238+
# than a green verdict over the wrong packs.
231239
- name: Restore corpus cache
232240
uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
233241
with:
234242
path: .census
235-
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ github.run_id }}-matrix
236-
restore-keys: ${{ env.CORPUS_CACHE_PREFIX }}
243+
key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('scripts/registry-census/corpus.pins.json') }}
244+
fail-on-cache-miss: true
237245

238246
# node_cache: false - setup-node's post step writes the pnpm store
239247
# AFTER this job has executed ~1,900 packs' code, and the publish

0 commit comments

Comments
 (0)