ci: detection proof - poison corpus counter-evidence for the matrix #7

Workflow file for this run

.github/workflows/ci-ecosystem-matrix.yaml at 43b2f9c

	# PR gate: execute every registry pack's frontend JS against this commit's
	# frontend runtime and gate on the population result.
	#
	# The corpus is a mirror of the frontend JS of EVERY registry pack (~5,100
	# packs). Only the executable surface is kept - .js/.mjs plus the text assets
	# a module can import - and binary assets are staged as empty placeholders so
	# their imports still resolve. That is what makes it cacheable: ~0.3GB
	# compressed against the repo's shared 10GB budget. Keeping every asset byte
	# put it at 6.0GB, which evicted itself mid-run (run 31753242605: three of
	# four shards found no corpus at all).
	#
	# Corpus freshness is a push-to-main concern, not a PR concern. main
	# revalidates every pack by ETag and saves the cache; a PR restores that
	# corpus and uses it as-is. Both sides of the comparison then measure the
	# same packs, so a red delta is attributable to the diff rather than to a
	# pack that moved overnight.
	#
	# The upstream static idiom scan (L1) is deliberately absent: its
	# false-positive/negative rate was judged too high to act on. Packs are
	# measured by executing their code - the ecosystem matrix - instead.
	#
	# Provenance: scripts/registry-census/ is vendored from
	# Comfy-Org/ComfyUI_ECS_Compat_Check (private, migration-phase instrument
	# slated for archival) - see scripts/registry-census/README.md.
	name: 'CI: Ecosystem Matrix'

	on:
	# A frontend change is the thing this gate exists to catch, so src/** runs
	# it. The corpus is restored, not refetched, so a PR run is bounded by
	# execution rather than by the network.
	pull_request:
	paths:
	- 'src/**'
	- 'scripts/registry-census/**'
	- 'vitest.matrix.config.mts'
	- '.github/workflows/ci-ecosystem-matrix.yaml'
	# Refreshes the corpus and writes the baseline every PR is measured against.
	# Actions cache scoping means only a default-branch run can produce an entry
	# that other branches restore.
	push:
	branches: [main]
	paths:
	- 'src/**'
	- 'scripts/registry-census/**'
	- 'vitest.matrix.config.mts'
	- '.github/workflows/ci-ecosystem-matrix.yaml'
	workflow_dispatch:

	permissions:
	contents: read

	concurrency:
	group: ecosystem-matrix-${{ github.ref }}
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

	env:
	PYTHON_VERSION: '3.11'
	CORPUS_CACHE_PREFIX: registry-corpus-v3-

	jobs:
	# First job in the run, and deliberately dependency-free so its annotation
	# and step summary land at the top. Stale pins are the one failure mode this
	# instrument cannot detect from the inside: everything stays green while the
	# ecosystem it claims to measure moves on without it.
	pin-status:
	runs-on: ubuntu-latest
	timeout-minutes: 5
	steps:
	- name: Checkout repository
	uses: actions/checkout@v7

	- name: Setup Python
	uses: actions/setup-python@v6
	with:
	python-version: ${{ env.PYTHON_VERSION }}

	- name: How old is the corpus, and how do I bump it
	run: python3 scripts/registry-census/pins.py

	corpus:
	runs-on: ubuntu-latest
	timeout-minutes: 30
	# contents: read only. The fetch step below holds a token (for the
	# codeload rate limit) while extracting untrusted archives, so this job
	# must not also carry actions: write - cache pruning runs as its own job.
	permissions:
	contents: read
	steps:
	- name: Checkout repository
	uses: actions/checkout@v7

	# An ubuntu-latest image bump must not silently change the interpreter
	# under a verdict whose whole value is run-to-run comparability.
	- name: Setup Python
	uses: actions/setup-python@v6
	with:
	python-version: ${{ env.PYTHON_VERSION }}

	# Keyed on the pins, so it is an exact hit for every run measuring the
	# same commits and a guaranteed miss the moment they are bumped. The
	# prefix fallback serves a cold PR something usable rather than nothing.
	- name: Restore corpus cache
	id: restore
	uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
	with:
	path: .census
	key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('scripts/registry-census/corpus.pins.json') }}
	restore-keys: ${{ env.CORPUS_CACHE_PREFIX }}

	# Re-pin the registry snapshot every run: it is the corpus's target
	# list, and new packs only enter the corpus through it.
	- name: Refresh registry snapshot
	run: python3 scripts/registry-census/refresh_registry.py

	# Authenticated codeload: anonymous fetches are rate-limited per IP, and
	# a run where many packs drift at once exhausts that budget partway
	# through. Run 31738365496 failed 145 of 150 attempted fetches this way
	# and reported it as an ecosystem-wide outage. This job only downloads
	# and unpacks archives; pack code executes in ecosystem-matrix, which
	# checks out with persist-credentials: false.
	#
	# Fetches at the commits named in corpus.pins.json, so this only does
	# real work on a cold cache or a pin bump. Prints the pin banner first.
	- name: Fetch corpus
	id: fetch
	env:
	GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
	run: python3 scripts/registry-census/fetch_corpus.py

	# corpus.lock.json records the ETag and resolved tree of every pack this
	# run measured - the identity a published figure has to be cited with.
	# registry-stale.json is present only when the snapshot is a fallback.
	- name: Upload corpus provenance
	if: always()
	uses: actions/upload-artifact@v6
	with:
	name: corpus-lock
	path: .census/*.json
	retention-days: 30
	if-no-files-found: warn

	# Key derived from the lockfile: unchanged corpus finds its key taken
	# and the server rejects the duplicate; any pack drift writes a new
	# entry. Gated on the FETCH step and a present lockfile: an early
	# failure must not save a partial .census - least of all under the
	# empty-hash key, which equals the restore-keys prefix and would
	# become the best match for every later run.
	#
	# Default branch only. A PR that saved would spend the repo's shared
	# cache budget on an entry no other branch can restore.
	- name: Save corpus cache
	if: \|
	github.event_name != 'pull_request'
	&& github.ref_name == github.event.repository.default_branch
	&& steps.fetch.outcome == 'success'
	&& hashFiles('.census/corpus.lock.json') != ''
	uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
	with:
	path: .census
	key: ${{ env.CORPUS_CACHE_PREFIX }}${{ hashFiles('.census/corpus.lock.json') }}

	# Each drift writes a new entry against the repo's shared 10GB budget; keep
	# the newest only. Its own job so `corpus`, which holds a token while
	# extracting untrusted archives, stays at contents: read.
	prune-corpus-cache:
	needs: corpus
	if: \|
	needs.corpus.result == 'success'
	&& github.event_name != 'pull_request'
	&& github.repository == 'Comfy-Org/ComfyUI_frontend'
	&& github.ref_name == github.event.repository.default_branch
	runs-on: ubuntu-latest
	timeout-minutes: 10
	permissions:
	actions: write
	steps:
	# --ref, or this enumerates every branch's entries and deletes a PR's
	# corpus out from under it. --limit, because the default of 30 leaves a
	# larger backlog permanently unpruned. pipefail, because the exit status
	# otherwise comes from the loop, which succeeds over empty input - so a
	# failing `gh cache list` would let pruning stop while the step reads
	# green.
	- name: Prune superseded corpus caches
	env:
	GH_TOKEN: ${{ github.token }}
	run: \|
	set -euo pipefail
	gh cache list --repo "$GITHUB_REPOSITORY" \
	--key "$CORPUS_CACHE_PREFIX" \
	--ref "refs/heads/${GITHUB_REF_NAME}" \
	--limit 100 --sort created_at --order desc \
	--json id --jq '.[1:][].id' > ids.txt
	while read -r id; do
	gh cache delete "$id" --repo "$GITHUB_REPOSITORY" \|\| true
	done < ids.txt

	# The execution rung between the corpus (every pack fetched) and the E2E
	# gates (browser + backend): every JS-shipping pack's real extension code
	# is imported and driven against the real frontend runtime under vitest -
	# registration lifecycle, a user-operation battery, serialize/reload.
	# Sharded 4 ways; vitest additionally parallelizes per core within a
	# shard (~4.7s of test time per pack, measured).
	#
	# A shard checks harness integrity only: every built spec must write its
	# per-pack row. vitest's own exit code is ignored here because pack code
	# leaks unhandled rejections. The ecosystem PASS/FAIL verdict is applied
	# once, over all shards combined, by matrix-verdict below.
	ecosystem-matrix:
	needs: corpus
	runs-on: ubuntu-latest
	timeout-minutes: 30
	strategy:
	fail-fast: false
	matrix:
	shard: [1, 2, 3, 4]
	steps:
	# This job EXECUTES unreviewed third-party pack code under vitest;
	# the checkout must not leave a usable git credential behind for it.
	- name: Checkout repository
	uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Setup Python
	uses: actions/setup-python@v6
	with:
	python-version: ${{ env.PYTHON_VERSION }}

	# The corpus job above saved the cache this run; same-run caches are
	# visible to later jobs. Restore-only - the matrix never writes it.
	- name: Restore corpus cache
	uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
	with:
	path: .census
	key: ${{ env.CORPUS_CACHE_PREFIX }}${{ github.run_id }}-matrix
	restore-keys: ${{ env.CORPUS_CACHE_PREFIX }}

	# node_cache: false - setup-node's post step writes the pnpm store
	# AFTER this job has executed ~1,900 packs' code, and the publish
	# workflows restore that same lockfile-derived key while holding
	# NPM_TOKEN and id-token: write. Slower installs here; no write path
	# out of the job that runs unreviewed code.
	- name: Setup frontend
	uses: ./.github/actions/setup-frontend
	with:
	include_build_step: false
	node_cache: 'false'

	- name: Build shard fixture
	run: python3 scripts/registry-census/build_matrix.py --shard ${{ matrix.shard }}/4

	# Every built spec gets its stub row BEFORE vitest starts: no failure
	# mode - worker crash, collection kill, hang - can produce a missing
	# row, only an incomplete one, which the verdict counts and reports.
	- name: Pre-create stub rows
	run: \|
	python3 - <<'PY'
	import json, os
	manifest = json.load(open('src/__ecs_matrix__/manifest.json'))
	os.makedirs('matrix-rows', exist_ok=True)
	for pack, entry in manifest.items():
	if 'entries' in entry:
	json.dump(
	{'pack': pack, 'incomplete': True},
	open(os.path.join('matrix-rows', entry['safe'] + '.json'), 'w'),
	)
	PY

	- name: Run the matrix
	run: \|
	set -euo pipefail
	shopt -s nullglob
	specs=(src/__ecs_matrix__/*.matrix.test.ts)
	built=${#specs[@]}
	if [ "$built" -eq 0 ]; then
	echo "::error::no pack specs built - corpus missing or empty"
	exit 1
	fi
	MATRIX_OUT="$PWD/matrix-rows" pnpm exec vitest run \
	--config vitest.matrix.config.mts > vitest-shard.log 2>&1 \
	\|\| echo "vitest exited nonzero - tolerated, pack code may leak unhandled errors"
	tail -25 vitest-shard.log
	cp src/__ecs_matrix__/manifest.json "matrix-rows/_manifest-shard-${{ matrix.shard }}.json"

	# nullglob makes an empty match an empty array rather than the
	# literal pattern, so these counts stay honest when nothing ran.
	spec_names=()
	for f in "${specs[@]}"; do
	spec_names+=("$(basename "$f" .matrix.test.ts)")
	done
	row_names=()
	for f in matrix-rows/*.json; do
	name=$(basename "$f" .json)
	case "$name" in _manifest*) continue ;; esac
	row_names+=("$name")
	done
	echo "specs=$built rows=${#row_names[@]}"
	if [ "${#row_names[@]}" -ne "$built" ]; then
	echo "::error::matrix wrote ${#row_names[@]} rows for $built specs - harness failure, not pack noise"
	comm -23 \
	<(printf '%s\n' "${spec_names[@]}" \| sort) \
	<(printf '%s\n' "${row_names[@]}" \| sort) \
	\| sed 's/^/::error::missing row: /'
	exit 1
	fi

	# The shard log is the only record of WHY a shard died. Without it a
	# 06:30 failure leaves the last 25 lines of a truncated tail and nothing
	# to go back to.
	- name: Upload rows
	if: always()
	uses: actions/upload-artifact@v6
	with:
	name: ecosystem-matrix-shard-${{ matrix.shard }}
	path: \|
	matrix-rows/
	vitest-shard.log
	retention-days: 30
	if-no-files-found: warn

	# The single combined result: all four shards' rows merge into one table
	# and one gating verdict - this job's exit code IS the ecosystem verdict
	# (0 PASS, 1 FAIL, 2 harness failure / no rows / short population).
	#
	# MATRIX_EXPECT_SHARDS makes a dead shard withhold rather than PASS. Every
	# criterion is a ratio, so three shards produce a perfectly plausible
	# number, and on PASS that partial population would be saved as the
	# baseline the next run is measured against.
	matrix-verdict:
	needs: ecosystem-matrix
	# A single failed shard must degrade to a withheld verdict over the rows
	# that exist, not to a skipped check that reads as nothing happened. Not
	# always(): a cancelled run would find no rows, exit 2, and manufacture a
	# red out of someone pressing the stop button.
	if: '!cancelled()'
	runs-on: ubuntu-latest
	timeout-minutes: 10
	steps:
	- name: Checkout repository
	uses: actions/checkout@v7

	- name: Setup Python
	uses: actions/setup-python@v6
	with:
	python-version: ${{ env.PYTHON_VERSION }}

	# continue-on-error: when the corpus job fails, no shard ran and no
	# artifact exists - the verdict step must still run so the summarizer
	# reports the documented no-rows exit (2) instead of an opaque
	# download failure.
	- name: Merge shard rows
	continue-on-error: true
	uses: actions/download-artifact@v8
	with:
	pattern: ecosystem-matrix-shard-*
	path: matrix-rows
	merge-multiple: true

	# A stale registry means a different denominator, and that denominator
	# feeds the delta gate. refresh_registry.py records the fallback in
	# registry-stale.json; the summarizer surfaces it in the report and
	# stores it in metrics.json rather than leaving it in a job log.
	- name: Fetch corpus provenance
	continue-on-error: true
	uses: actions/download-artifact@v8
	with:
	name: corpus-lock
	path: corpus-lock

	# The baseline this PR is measured against: main's last green run. The
	# primary key never hits (saves carry a run_attempt suffix), so
	# restore-keys serves the newest save. A re-run can restore this same
	# run's earlier attempt - metrics carry runId and the summarizer skips
	# the delta rather than compare a run against itself.
	- name: Restore baseline metrics
	uses: actions/cache/restore@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
	with:
	path: matrix-metrics
	key: matrix-metrics-${{ github.run_id }}
	restore-keys: matrix-metrics-

	- name: Verdict
	id: verdict
	env:
	MATRIX_RUN_ID: ${{ github.run_id }}
	MATRIX_EXPECT_SHARDS: '4'
	run: \|
	MATRIX_OUT="$PWD/matrix-rows" \
	MATRIX_PREV="$PWD/matrix-metrics/metrics.json" \
	MATRIX_METRICS_OUT="$PWD/matrix-metrics/metrics.json" \
	MATRIX_STALE_MARKER="$PWD/corpus-lock/registry-stale.json" \
	python3 scripts/registry-census/summarize_matrix.py

	# Baseline-from-health, and only from main: a PR must not publish its
	# own numbers as the baseline every later PR is measured against, and a
	# FAILing run must not ratchet the baseline down to its own eroded
	# numbers.
	- name: Save baseline metrics
	if: \|
	github.event_name != 'pull_request'
	&& github.ref_name == github.event.repository.default_branch
	&& steps.verdict.outcome == 'success'
	&& hashFiles('matrix-metrics/metrics.json') != ''
	uses: actions/cache/save@caa296126883cff596d87d8935842f9db880ef25 # v5.1.0
	with:
	path: matrix-metrics
	key: matrix-metrics-${{ github.run_id }}-${{ github.run_attempt }}

	- name: Upload combined result
	if: always()
	uses: actions/upload-artifact@v6
	with:
	name: ecosystem-matrix-combined
	path: matrix-rows/
	retention-days: 30
	if-no-files-found: warn

	# Counter-evidence: the matrix runs over a synthetic corpus where every
	# pack breaks exactly one measurement channel (detection-proof/corpus/),
	# then this job requires BOTH that every channel fired with its poison
	# message AND that each poison pack breaches the criterion it targets
	# (verify_detection.py). A green matrix is only meaningful while this stays
	# green: a runner or frontend change that blinds a channel reds THIS job
	# instead of silently hollowing out the gate. No corpus/cache dependency -
	# the poison fixture is checked in, so this runs on every PR that touches
	# src/** even when the corpus is cold.
	matrix-detection-proof:
	runs-on: ubuntu-latest
	timeout-minutes: 15
	steps:
	# Runs the poison packs (and the same harness that runs real pack
	# code); same rule as the shard jobs - no persisted credential.
	- name: Checkout repository
	uses: actions/checkout@v7
	with:
	persist-credentials: false

	- name: Setup Python
	uses: actions/setup-python@v6
	with:
	python-version: ${{ env.PYTHON_VERSION }}

	- name: Setup frontend
	uses: ./.github/actions/setup-frontend
	with:
	include_build_step: false
	node_cache: 'false'

	- name: Gate unit tests
	run: python3 -m unittest discover -s scripts/registry-census -p 'test_*.py' -v

	- name: Build poison fixture
	run: CENSUS_ROOT=scripts/registry-census/detection-proof python3 scripts/registry-census/build_matrix.py

	- name: Run the matrix over the poison corpus
	run: \|
	MATRIX_OUT="$PWD/matrix-proof" pnpm exec vitest run \
	--config vitest.matrix.config.mts 2>&1 \| tail -15 \
	\|\| echo "vitest exited nonzero - tolerated, poison packs leak by design"

	- name: Every channel must fire and must gate
	run: MATRIX_OUT="$PWD/matrix-proof" python3 scripts/registry-census/verify_detection.py

	# Exit code must be exactly 1 (criteria FAIL): exit 2 would mean the
	# harness gate withheld the verdict, which on this corpus is itself a
	# detection failure.
	- name: Verdict must FAIL on poison
	run: \|
	set +e
	MATRIX_OUT="$PWD/matrix-proof" python3 scripts/registry-census/summarize_matrix.py > proof-verdict.txt 2>&1
	rc=$?
	set -e
	tail -8 proof-verdict.txt
	if [ "$rc" -ne 1 ]; then
	echo "::error::expected verdict FAIL (exit 1) on the poison corpus, got exit $rc"
	exit 1
	fi
	echo "verdict correctly FAILED on the poison corpus"

Navigation Menu

Uh oh!

ci: detection proof - poison corpus counter-evidence for the matrix #7

Workflow file

ci: detection proof - poison corpus counter-evidence for the matrix #7

Uh oh!

Workflow file for this run