docs(spec): APR-DARK-TARGETS-001 — what the unrun test surface costs, holds, and when it should run - #3249
docs(spec): APR-DARK-TARGETS-001 — what the unrun test surface costs, holds, and when it should run#3249noahgift wants to merge 5 commits into
Conversation
… holds, and when it should run
654 test binaries have never been LINKED in CI, let alone run. Measured twice on
2026-09-14, one hour apart:
merge-queue full tier (#2838) 82,217 tests 73 binaries 37 min intel-clean-room-14
whole --lib --tests 98,621 tests 727 binaries 19 min lambda-vector (48c)
A binary that never links cannot fail, flake, or appear in a report. That is why
this was invisible rather than noisy, and it is the sentence the rest follows
from. `make coverage` is `--lib` too; nightly.yml runs no tests at all.
WHAT IT HELD: 37 failures, 5 binaries, and every one a GLOBAL-INVARIANT defect --
a golden oracle, a path to a sibling crate's assets, a contract copy that must
equal another file. Not one is a compute regression. That is structural, not
luck: BSE-17 selects by CHANGED CRATE and none of these lives in a diff. The
quick tier is NOT the gap and this spec does not touch it -- a PR touching crate
X does build and run X's integration targets.
The argument is golden_v2, not the count. A golden exists ONLY to catch silent
format drift and it missed a real format change for two months. #2254 happened to
be deliberate and correct; had it been wrong, nothing here would have said so. A
dark golden is worse than no golden, because its presence reads as coverage.
DECISION: nightly primary; pre-release (T-1) as a HARD gate, since "green on the
cut" over 73 of 727 binaries is a weaker claim than it reads; NOT the merge queue;
plus one addition to "nightly and pre-release only" -- PRs that touch the SELECTOR
(ci_test_tier.sh, gate_touched_crates.sh, tree_reader_tests.txt, ci.yml's --test
chain), because that is exactly when the dark set moves.
It starts ratcheted or it is born red at 37 and becomes `present` within a week.
But Phase B is reachable: with #3248/#3249/#3238 landed the count is ONE, so the
ratchet is scaffolding with a named exit condition, not a resting place.
I WAS WRONG ABOUT THE COST AND THE MEASUREMENT CAUGHT ME. The first draft asserted
the --tests delta was ">> 20 min" of linking, on the strength of a progress check
reading a stale PID. Measured, same box, cold target dir:
cargo build --workspace --lib 76 s 715 crates
+ --tests incrementally 145 s 228 crates -> 808 binaries, 39 GB
Two and a half minutes, not twenty. The cost is the RUN, not the link. That makes
the merge-queue question closer than the first draft implied -- ~+14 min (+35%) on
a 37-min critical path by throughput-calibrated extrapolation -- so the verdict is
unchanged but the REASON is now a number, and §3 names the condition under which
it flips. The number was already being measured when the claim was written; the
claim should have waited. §7.1 records that.
Ships with the failing set as measured, so the first ratchet baseline is not
re-derived from memory.
Refs #3239
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Written before the PR was opened, so it cited the number this document then got. The three PRs that take the failing set from 37 to 1 are #3248 (golden_v2), #3250 (present-yaml paths) and #3238 (mcp contract copy). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
… it was right on three counts All four agy lanes (3 plan + 1 teamwork) refused the draft. Three findings stand and the spec is revised rather than defended. 1. THE RATCHET HAD NO WRITER. The draft proposed shrink-only, tightening to hard-fail. Three lanes refused it independently and the defect is mechanical, not stylistic: `--update` writes a file and needs a commit, and a scheduled workflow has no PR and no author. Every other ratchet here (unwired_guards, pipe_grep_q, shell_lint) is updated by a human inside a PR BECAUSE it runs in one. A nightly's baseline would only ever be edited by someone who already noticed the job was red -- the exact population the ratchet exists to stop relying on. So: land #3250/#3248/#3238, gate doctor_speed_real, ship hard-failing on day one. The exit condition was already ONE test when the draft was written; there was never an interval in which a ratchet carried information. Fallback if the gating stalls: ONE named exclusion with its issue number, auditable in a line. 2. THE 39 GB TARGET DIR WAS UNBUDGETED. Unanimous, and the draft did not mention it once -- on the same day I spent an hour taking gx10 from 0 bytes free after it dequeued a PR from merge-queue position 1 (#3247). New §4bis makes it a constraint: not on gx10 (916 GB, the box that hit the wall), a per-run target dir reclaimed in-job rather than left to a reaper that already cannot reclaim under keep=3, and a preflight free-space assertion that fails ENV: instead of letting cargo die half-way and read as 650 new defects. 3. THE REVISIT CONDITION WAS WRONG. I keyed it on queue depth vs max_entries_to_build; lane 3 pointed out that is a CAP, not a measure of slack -- a queue can sit at depth 3 with idle capacity. Keyed on arrival rate vs drain rate instead. Also sharpened, from a measurement all lanes converged on: T-1 does not "add the full surface", it must BUILD `ci / deep`. That job has 0 hits in every workflow including the SHA-pinned upstream sovereign-ci.yml, and its other named parts run nowhere either -- `cargo test --doc` 0, `cargo run --example` 0, feature matrix 0, `--no-default-features` 1 hit that is a Windows BUILD flag. And the fast-pool pin leaves this spec entirely (§8bis). All four lanes rejected it with a computable break-even rather than a caveat: `runs-on` is a hard label filter with no fallback, so a pin is non-work-conserving and is a NET LOSS whenever fast-pool wait exceeds the ~18-22 min runtime delta -- which is the condition that holds exactly when the latency mattered. The asymmetry is real (#3251); the actuator is wrong. Lane 4 was briefed as /teamwork but agy 1.2.2 ran ZERO invoke_subagent calls and created no teamwork_projects dir (PMAT-126 reproduced). It is one model answering once and is not counted as a team review anywhere in this document. Refs #3239, #3251 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4-lane agy quorum:
|
Design for the findings in #3239, grounded in two measurements taken an hour apart today.
The number that matters
34809681888)intel-clean-room-14--workspace --lib --testsThe gap is not "20% more tests" — it is 10× the binaries. 654 test binaries have never been
linked in CI. A binary that never links cannot fail, cannot flake, and cannot appear in a
report; that is why this was invisible rather than noisy.
make coverageis--libtoo, andnightly.ymlruns no tests at all.What it held
37 failures in 5 binaries, and every one is a global invariant — a golden oracle, a path to a
sibling crate's assets, a contract copy that must equal another file. Not one is a compute
regression.
That is structural, not luck: BSE-17 selects by changed crate, and none of these lives in
anybody's diff. The quick tier is not the gap and this spec does not touch it — a PR touching
crate X does build and run X's integration targets.
The argument is
golden_v2, not the count. A golden exists only to catch silent format drift,and it missed a real format change for two months. #2254 happened to be deliberate and correct;
had it been wrong, nothing here would have said so. A dark golden is worse than no golden,
because its presence reads as coverage.
Decision
ci_test_tier.sh,gate_touched_crates.sh,tree_reader_tests.txt, ci.yml's--testchain. That is exactly when the dark set movesIt starts ratcheted or it is born red at 37 and becomes
presentwithin a week. But Phase Bis reachable: with #3248 / #3250 / #3238 landed the count is one
(
aprender-cgp::falsify falsify_cgp_061_doctor_speed_real). The ratchet is scaffolding with anamed exit condition, not a resting place.
I was wrong about the cost, and the measurement caught it
The first draft asserted the
--testsdelta was "≫ 20 min" of linking, on the strength of aprogress check that was reading a stale PID. Measured, same box, cold target dir:
Two and a half minutes, not twenty. The cost is the run, not the link. The verdict on the
merge queue is unchanged but the reason is now a number instead of a guess, and §7.1 records that
the figure was already being measured when the claim was written.
Ships with the measured failing set (
APR-DARK-TARGETS-001-failing-set-2026-09-14.txt) so thefirst ratchet baseline is not re-derived from memory.
Refs #3239
no-close: #3239 stays OPEN — this is the design for its close-out, not the close-out. The
workflow, the ratchet script and its case table are the implementation, and the capability-gating
precondition in §5 has to land first.
🤖 Generated with Claude Code