fix: stage the symbol map CI scores against, which nothing ever did

The binding floors are the one quality number this repository gates on, and the gate
has never run on a runner. `score_corpus.py` verifies a staged symbol map before it
scores anything, that map lives in gitignored `.cache/specs/`, and no step in the
workflow produced it -- so every run that got as far as scoring was refused with
`no symbol map at .cache/specs/symbols.json`. The corpus fetch was failing earlier in
the job for longer, which is the only reason this went unnoticed.

`symbol_map.yaml` did record how to rebuild the map, in prose. Prose is not a step a
runner can execute, and that gap is the whole defect. `stage_symbol_map.py` is the
same instruction as a command: build from a pinned specification, compare against the
pinned digest, and refuse before writing rather than after. A refusal leaves whatever
was already staged untouched -- `.cache/` is per-worktree space several workers share,
and a rebuild that truncated the file on its way to failing would break a run that was
scoring correctly.

Measured across v2200, v2300, v2330 and v2345: one digest, 272 symbols. The SDK
document is deliberately not read, matching how the pin was taken; both forms produce
the same digest, and passing the argument anyway would make the output depend on
whether a large optional file happened to be staged.

The test asserts position within the scoring job rather than the workflow containing
the right words somewhere, because a step order that looks right and stages nothing
reads as a passing job on any run where the gate is never reached. Proven able to fail
by deleting the step and watching it go red.

Verified locally end to end: five tree digests ok, then binding precision 1.0000 and
recall 1.0000 at n=26, 7 falsifiable negatives, 17 pairs scored. Every floor cleared,
which is also the measurement the line-ending fix needed and had not yet had.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: stroland02

.github/workflows/ci.yml

@@ -132,6 +132,21 @@ jobs:
       - name: Fetch the frozen corpus
         run: uv run python scripts/fetch_corpus_repositories.py
 
+      # The corpus has two frozen inputs and this job used to stage one of them. `repositories.yaml`
+      # pins the trees; `symbol_map.yaml` pins the map that decides which call sites resolve at
+      # all, and it lives in gitignored `.cache/specs/` because it is generated and large. With
+      # nothing staging it, `score_corpus.py` refused every time it was reached -- `no symbol map
+      # at .cache/specs/symbols.json` -- so the binding gate below has never once run on a runner.
+      # The step before this one was failing for longer, which is what hid it.
+      #
+      # Both scripts verify what they write before writing it: a git blob hash per specification,
+      # and the pinned digest for the map. Neither can quietly substitute an input.
+      - name: Stage the pinned Stripe specifications
+        run: uv run python scripts/fetch_measurement_inputs.py
+
+      - name: Stage the pinned symbol map
+        run: uv run python scripts/stage_symbol_map.py
+
       # A database of its own. `score_corpus.py` truncates the graph once per pair, so pointing
       # it at the one the suite used would delete what a later step might read.
       - name: Score the frozen corpus

.gitignore

@@ -31,3 +31,9 @@ tools/
 # worktree when this line was added.
 .coverage.*
 coverage.json
+
+# The corpus score, written by `scripts/score_corpus.py --json` and read by
+# `scripts/gate_corpus.py`. A measurement of the tree it was taken on, so a committed copy is
+# a number that goes stale silently while looking authoritative. The floors are what gets
+# committed, in `gate_corpus.py`, beside the argument for each.
+corpus-score.json

scripts/stage_symbol_map.py

@@ -0,0 +1,86 @@
+"""Rebuild the symbol map the frozen corpus is scored against, and stage it where the scorer looks.
+
+`benchmark/corpus/symbol_map.yaml` pins that map by a digest and records in prose how to rebuild
+it: `build_symbol_map` over a Stripe specification staged in `.cache/specs/`, with no SDK
+document. Prose is not a step a runner can execute, and the cost was exact -- CI fetched the
+corpus, reached `scripts/score_corpus.py`, and was refused with `no symbol map at
+.cache/specs/symbols.json`. The binding gate has never run on a runner, and the fetch failure
+that came before it hid that for as long as it lasted.
+
+The SDK document is deliberately not read
+-----------------------------------------
+`_prepare_stripe` passes one when the tag publishes it, and the pin was taken without one. Both
+produce 272 symbols and the same digest, which the pin's own note records as measured. Passing
+the argument anyway would make this script's output depend on whether a large optional file
+happened to be staged, which is the class of thing the pin exists to stop.
+
+Any pinned tag will do, and that is a property rather than a convenience
+-----------------------------------------------------------------------
+Measured across `v2200`, `v2300`, `v2330` and `v2345`: one digest, 272 symbols. The map is keyed
+by SDK symbol and derived from path segments and `x-stableId`, both of which are stable across
+these versions. `--spec` defaults to the head the corpus pairs are cut against; a caller naming
+another is checked by the same pin, so a version that ever did move the map fails here rather
+than scoring quietly.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+from sync.signals.stripe.symbols import build_symbol_map
+
+# Run as `uv run python scripts/stage_symbol_map.py`, Python puts `scripts/` on the path and not
+# the repository root, so the sibling module below is unimportable by the name the tests use.
+# The same line for the same reason as in `scripts/score_corpus.py`.
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from scripts.symbol_map_pin import PIN, SymbolMapMismatch, read_pin, symbol_map_digest
+
+SPEC = Path(".cache/specs/v2330.json")
+
+
+def stage_symbol_map(spec: Path, into: Path, pin: Mapping[str, Any]) -> str:
+    """Build the map from `spec` and write it to `into`, or refuse and write nothing.
+
+    The check is before the write, and a refusal leaves whatever was already staged untouched.
+    `.cache/` is per-worktree space several workers share, so a rebuild that truncated the file
+    on its way to failing would break a run that was scoring correctly.
+    """
+    mapping = build_symbol_map(json.loads(spec.read_text(encoding="utf-8")), None)
+    digest = symbol_map_digest(mapping)
+    if digest != pin["digest"]:
+        raise SymbolMapMismatch(
+            f"{spec} builds a map at digest {digest[:12]} with {len(mapping)} symbols; the "
+            f"corpus records {pin['digest'][:12]} with {pin['symbols']}. Re-pinning is a "
+            f"deliberate act with a measurement attached -- see {PIN}."
+        )
+
+    into.parent.mkdir(parents=True, exist_ok=True)
+    into.write_text(json.dumps(mapping), encoding="utf-8")
+    return digest
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
+    parser.add_argument("--spec", default=str(SPEC), help="the specification to build from")
+    parser.add_argument("--pin", default=str(PIN), help="the pin the built map must match")
+    args = parser.parse_args()
+
+    pin = read_pin(Path(args.pin))
+    try:
+        digest = stage_symbol_map(Path(args.spec), Path(pin["staged_at"]), pin)
+    except (OSError, SymbolMapMismatch) as exc:
+        print(f"refused: {exc}")
+        return 2
+
+    print(f"{pin['staged_at']}  {digest}  {pin['symbols']} symbols")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tests/test_ci_stages_the_corpus_inputs.py

@@ -0,0 +1,72 @@
+"""A gate that cannot reach its inputs is not a gate, and this one could not for its whole life.
+
+`scripts/score_corpus.py` reads two frozen inputs. The trees come from
+`scripts/fetch_corpus_repositories.py`, which CI has always run. The symbol map comes from
+`.cache/specs/`, which is gitignored and which no step staged -- so every time the workflow got
+as far as scoring it was refused with `no symbol map at .cache/specs/symbols.json`, and the
+binding floors below it have never been measured on a runner. The fetch step failing for longer
+is what kept that invisible.
+
+The failure mode this pins is not "a step was deleted". It is a step order that looks right and
+stages nothing, which reads as a passing job on any run where the gate is never reached. So the
+assertions are on position within the job that scores, not on the workflow containing the words
+somewhere.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import yaml
+
+WORKFLOW = Path(__file__).resolve().parents[1] / ".github" / "workflows" / "ci.yml"
+
+SCORER = "scripts/score_corpus.py"
+
+# Everything the scorer reads that a checkout does not carry, and the script that produces it.
+# `fetch_corpus_repositories.py` is here too: it has never been the missing one, and a test that
+# only names the input that broke would let the next deletion through.
+STAGERS = (
+    "scripts/fetch_corpus_repositories.py",
+    "scripts/fetch_measurement_inputs.py",
+    "scripts/stage_symbol_map.py",
+)
+
+
+def _scoring_jobs() -> list[tuple[str, list[dict]]]:
+    workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
+    return [
+        (name, job.get("steps", []))
+        for name, job in workflow["jobs"].items()
+        if any(SCORER in (step.get("run") or "") for step in job.get("steps", []))
+    ]
+
+
+def _index_of(steps: list[dict], script: str) -> int | None:
+    for position, step in enumerate(steps):
+        if script in (step.get("run") or ""):
+            return position
+    return None
+
+
+def test_ci_scores_the_frozen_corpus_at_all():
+    assert _scoring_jobs(), (
+        f"no job in ci.yml runs {SCORER}. The binding floors are the only quality number this "
+        "repository gates on, and with this step gone nothing measures them"
+    )
+
+
+def test_every_input_the_scorer_reads_is_staged_before_it_runs():
+    for job_name, steps in _scoring_jobs():
+        scores_at = _index_of(steps, SCORER)
+        for script in STAGERS:
+            stages_at = _index_of(steps, script)
+            assert stages_at is not None, (
+                f"job {job_name!r} scores the corpus and never runs {script}. The scorer reads "
+                f"what it produces out of gitignored space, so on a runner that input is absent "
+                f"and the gate refuses instead of measuring"
+            )
+            assert stages_at < scores_at, (
+                f"job {job_name!r} runs {script} after {SCORER}, so the scorer reads whatever "
+                f"was there before it -- on a fresh checkout, nothing"
+            )

tests/test_stage_symbol_map.py

@@ -0,0 +1,96 @@
+"""The symbol map the corpus is scored against has to be reproducible by a command.
+
+`benchmark/corpus/symbol_map.yaml` pins the map and says in prose how to rebuild it. Prose is
+not executable, and the consequence was measurable: CI fetched the corpus, reached
+`score_corpus.py`, and was refused with `no symbol map at .cache/specs/symbols.json` -- the
+gate has never once run on a runner, because no step could stage what it verifies.
+
+The refusal is the part worth testing. A map that does not match the pin must not reach the
+path the scorer reads, or the next run scores a resolution nobody recorded.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+
+from scripts.stage_symbol_map import stage_symbol_map
+from scripts.symbol_map_pin import SymbolMapMismatch, symbol_map_digest, verify_staged_map
+from sync.signals.stripe.symbols import build_symbol_map
+
+# One path with one operation, which is all the builder needs to produce a map. The specification
+# the corpus really uses is 7.8 MB and gitignored; what is under test here is the pin check
+# around the builder, not the builder.
+SPEC = {
+    "paths": {
+        "/v1/payment_intents": {
+            "post": {"operationId": "PostPaymentIntents"},
+        },
+    },
+}
+
+
+@pytest.fixture()
+def spec(tmp_path: Path) -> Path:
+    path = tmp_path / "v2330.json"
+    path.write_text(json.dumps(SPEC), encoding="utf-8")
+    return path
+
+
+def _pin(mapping: dict) -> dict:
+    return {
+        "digest": symbol_map_digest(mapping),
+        "symbols": len(mapping),
+        "staged_at": ".cache/specs/symbols.json",
+        "built_from": "this test",
+    }
+
+
+def test_a_matching_map_is_staged_where_the_scorer_reads_it(spec, tmp_path):
+    """The whole point: after this runs, `verify_staged_map` accepts what is on disk."""
+    pin = _pin(build_symbol_map(json.loads(spec.read_text(encoding="utf-8")), None))
+    into = tmp_path / "specs" / "symbols.json"
+
+    digest = stage_symbol_map(spec, into, pin)
+
+    assert digest == pin["digest"]
+    assert verify_staged_map(into, pin) == pin["digest"]
+
+
+def test_a_map_that_does_not_match_the_pin_never_reaches_the_disk(spec, tmp_path):
+    """Refused before the write, not after it.
+
+    Writing first and checking second leaves the wrong map staged for whatever runs next, and
+    the scorer's own refusal is the only thing standing between that and a scored number over a
+    resolution nobody recorded.
+    """
+    pin = _pin(build_symbol_map(json.loads(spec.read_text(encoding="utf-8")), None))
+    pin["digest"] = "0" * 64
+    into = tmp_path / "specs" / "symbols.json"
+
+    with pytest.raises(SymbolMapMismatch):
+        stage_symbol_map(spec, into, pin)
+
+    assert not into.exists()
+
+
+def test_a_stale_map_already_on_disk_is_left_alone_when_the_new_one_is_refused(spec, tmp_path):
+    """A refusal must not be a deletion either.
+
+    The staged path is shared -- `.cache/` is per-worktree and several workers run here at once
+    -- so a refused rebuild that truncated the file would break a run that was scoring fine.
+    """
+    pin = _pin(build_symbol_map(json.loads(spec.read_text(encoding="utf-8")), None))
+    into = tmp_path / "specs" / "symbols.json"
+    into.parent.mkdir(parents=True)
+    into.write_text('{"already": {"here": "1"}}', encoding="utf-8")
+
+    with pytest.raises(SymbolMapMismatch):
+        stage_symbol_map(spec, into, {**pin, "digest": "0" * 64})
+
+    assert json.loads(into.read_text(encoding="utf-8")) == {"already": {"here": "1"}}