Skip to content

docs(spec): APR-DARK-TARGETS-001 — what the unrun test surface costs, holds, and when it should run - #3249

Open
noahgift wants to merge 5 commits into
mainfrom
PMAT-1098-dark-targets-design
Open

noahgift wants to merge 5 commits into
mainfrom
PMAT-1098-dark-targets-design

Conversation

@noahgift

@noahgift noahgift commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Design for the findings in #3239, grounded in two measurements taken an hour apart today.

The number that matters

tests binaries run where
merge-queue full tier (#2838, run 34809681888) 82,217 73 37 min intel-clean-room-14
whole --workspace --lib --tests 98,621 727 19 min lambda-vector, 48 cores

The gap is not "20% more tests" — it is 10× the binaries. 654 test binaries have never been
linked in CI. A binary that never links cannot fail, cannot flake, and cannot appear in a
report; that is why this was invisible rather than noisy. make coverage is --lib too, and
nightly.yml runs no tests at all.

What it held

37 failures in 5 binaries, and every one is a global invariant — a golden oracle, a path to a
sibling crate's assets, a contract copy that must equal another file. Not one is a compute
regression.

That is structural, not luck: BSE-17 selects by changed crate, and none of these lives in
anybody's diff. The quick tier is not the gap and this spec does not touch it — a PR touching
crate X does build and run X's integration targets.

The argument is golden_v2, not the count. A golden exists only to catch silent format drift,
and it missed a real format change for two months. #2254 happened to be deliberate and correct;
had it been wrong, nothing here would have said so. A dark golden is worse than no golden,
because its presence reads as coverage.

Decision

surface verdict
Nightly yes — primary
Pre-release (T-1) yes — hard gate. "Green on the cut" over 73 of 727 binaries is a weaker claim than it reads
Merge queue no — but closer than I first thought; ≈ +14 min (+35%) on a 37-min critical path. §3 names the condition under which it flips
Every PR no — §2 is the argument
PRs touching the SELECTOR yesci_test_tier.sh, gate_touched_crates.sh, tree_reader_tests.txt, ci.yml's --test chain. That is exactly when the dark set moves

It starts ratcheted or it is born red at 37 and becomes present within a week. But Phase B
is reachable: with #3248 / #3250 / #3238 landed the count is one
(aprender-cgp::falsify falsify_cgp_061_doctor_speed_real). The ratchet is scaffolding with a
named exit condition, not a resting place.

I was wrong about the cost, and the measurement caught it

The first draft asserted the --tests delta was "≫ 20 min" of linking, on the strength of a
progress check that was reading a stale PID. Measured, same box, cold target dir:

cargo build --workspace --lib          76 s   715 crates
+ --tests incrementally               145 s   228 crates  →  808 binaries, 39 GB

Two and a half minutes, not twenty. The cost is the run, not the link. The verdict on the
merge queue is unchanged but the reason is now a number instead of a guess, and §7.1 records that
the figure was already being measured when the claim was written.

Ships with the measured failing set (APR-DARK-TARGETS-001-failing-set-2026-09-14.txt) so the
first ratchet baseline is not re-derived from memory.

Refs #3239

no-close: #3239 stays OPEN — this is the design for its close-out, not the close-out. The
workflow, the ratchet script and its case table are the implementation, and the capability-gating
precondition in §5 has to land first.

🤖 Generated with Claude Code

… holds, and when it should run

654 test binaries have never been LINKED in CI, let alone run. Measured twice on
2026-09-14, one hour apart:

    merge-queue full tier (#2838)   82,217 tests   73 binaries   37 min  intel-clean-room-14
    whole --lib --tests             98,621 tests  727 binaries   19 min  lambda-vector (48c)

A binary that never links cannot fail, flake, or appear in a report. That is why
this was invisible rather than noisy, and it is the sentence the rest follows
from. `make coverage` is `--lib` too; nightly.yml runs no tests at all.

WHAT IT HELD: 37 failures, 5 binaries, and every one a GLOBAL-INVARIANT defect --
a golden oracle, a path to a sibling crate's assets, a contract copy that must
equal another file. Not one is a compute regression. That is structural, not
luck: BSE-17 selects by CHANGED CRATE and none of these lives in a diff. The
quick tier is NOT the gap and this spec does not touch it -- a PR touching crate
X does build and run X's integration targets.

The argument is golden_v2, not the count. A golden exists ONLY to catch silent
format drift and it missed a real format change for two months. #2254 happened to
be deliberate and correct; had it been wrong, nothing here would have said so. A
dark golden is worse than no golden, because its presence reads as coverage.

DECISION: nightly primary; pre-release (T-1) as a HARD gate, since "green on the
cut" over 73 of 727 binaries is a weaker claim than it reads; NOT the merge queue;
plus one addition to "nightly and pre-release only" -- PRs that touch the SELECTOR
(ci_test_tier.sh, gate_touched_crates.sh, tree_reader_tests.txt, ci.yml's --test
chain), because that is exactly when the dark set moves.

It starts ratcheted or it is born red at 37 and becomes `present` within a week.
But Phase B is reachable: with #3248/#3249/#3238 landed the count is ONE, so the
ratchet is scaffolding with a named exit condition, not a resting place.

I WAS WRONG ABOUT THE COST AND THE MEASUREMENT CAUGHT ME. The first draft asserted
the --tests delta was ">> 20 min" of linking, on the strength of a progress check
reading a stale PID. Measured, same box, cold target dir:

    cargo build --workspace --lib          76 s   715 crates
    + --tests incrementally               145 s   228 crates  -> 808 binaries, 39 GB

Two and a half minutes, not twenty. The cost is the RUN, not the link. That makes
the merge-queue question closer than the first draft implied -- ~+14 min (+35%) on
a 37-min critical path by throughput-calibrated extrapolation -- so the verdict is
unchanged but the REASON is now a number, and §3 names the condition under which
it flips. The number was already being measured when the claim was written; the
claim should have waited. §7.1 records that.

Ships with the failing set as measured, so the first ratchet baseline is not
re-derived from memory.

Refs #3239

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
noahgift and others added 2 commits September 14, 2026 09:22
Written before the PR was opened, so it cited the number this document then got.
The three PRs that take the failing set from 37 to 1 are #3248 (golden_v2),
#3250 (present-yaml paths) and #3238 (mcp contract copy).

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift
noahgift enabled auto-merge September 14, 2026 07:23
@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3249 head=5d99a544eef38c90813fa99efa083ee28d6e9912 verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

… it was right on three counts

All four agy lanes (3 plan + 1 teamwork) refused the draft. Three findings stand
and the spec is revised rather than defended.

1. THE RATCHET HAD NO WRITER. The draft proposed shrink-only, tightening to
   hard-fail. Three lanes refused it independently and the defect is mechanical,
   not stylistic: `--update` writes a file and needs a commit, and a scheduled
   workflow has no PR and no author. Every other ratchet here
   (unwired_guards, pipe_grep_q, shell_lint) is updated by a human inside a PR
   BECAUSE it runs in one. A nightly's baseline would only ever be edited by
   someone who already noticed the job was red -- the exact population the
   ratchet exists to stop relying on. So: land #3250/#3248/#3238, gate
   doctor_speed_real, ship hard-failing on day one. The exit condition was
   already ONE test when the draft was written; there was never an interval in
   which a ratchet carried information. Fallback if the gating stalls: ONE named
   exclusion with its issue number, auditable in a line.

2. THE 39 GB TARGET DIR WAS UNBUDGETED. Unanimous, and the draft did not mention
   it once -- on the same day I spent an hour taking gx10 from 0 bytes free after
   it dequeued a PR from merge-queue position 1 (#3247). New §4bis makes it a
   constraint: not on gx10 (916 GB, the box that hit the wall), a per-run target
   dir reclaimed in-job rather than left to a reaper that already cannot reclaim
   under keep=3, and a preflight free-space assertion that fails ENV: instead of
   letting cargo die half-way and read as 650 new defects.

3. THE REVISIT CONDITION WAS WRONG. I keyed it on queue depth vs
   max_entries_to_build; lane 3 pointed out that is a CAP, not a measure of
   slack -- a queue can sit at depth 3 with idle capacity. Keyed on arrival rate
   vs drain rate instead.

Also sharpened, from a measurement all lanes converged on: T-1 does not "add the
full surface", it must BUILD `ci / deep`. That job has 0 hits in every workflow
including the SHA-pinned upstream sovereign-ci.yml, and its other named parts run
nowhere either -- `cargo test --doc` 0, `cargo run --example` 0, feature matrix 0,
`--no-default-features` 1 hit that is a Windows BUILD flag.

And the fast-pool pin leaves this spec entirely (§8bis). All four lanes rejected
it with a computable break-even rather than a caveat: `runs-on` is a hard label
filter with no fallback, so a pin is non-work-conserving and is a NET LOSS
whenever fast-pool wait exceeds the ~18-22 min runtime delta -- which is the
condition that holds exactly when the latency mattered. The asymmetry is real
(#3251); the actuator is wrong.

Lane 4 was briefed as /teamwork but agy 1.2.2 ran ZERO invoke_subagent calls and
created no teamwork_projects dir (PMAT-126 reproduced). It is one model answering
once and is not counted as a team review anywhere in this document.

Refs #3239, #3251

Pmat-Ticket: PMAT-1098

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@noahgift

Copy link
Copy Markdown
Contributor Author

4-lane agy quorum: do-not-implement-as-written ×4 — spec revised, not defended

Lanes: 3 × --mode plan + 1 × --mode teamwork, writes=false. Conversations
ae4d0c12, bc9be3c9, 0a827402, bac6569a.

First, what the review was not. Lane 4 was briefed as /teamwork-preview. agy 1.2.2 ran
zero invoke_subagent calls and created no teamwork_projects directory — PMAT-126
reproduced. It is one model answering once, not a team review, and it is not counted toward
consensus anywhere below.

Three findings stand

1. The ratchet had no writer. The draft proposed shrink-only → hard-fail. Three lanes refused
it independently, and the defect is mechanical: --update writes a file and needs a commit, and a
scheduled workflow has no PR and no author. Every other ratchet here (unwired_guards,
pipe_grep_q, shell_lint) is updated by a human inside a PR because it runs in one. A
nightly's baseline would only ever be edited by someone who already noticed the job was red — the
exact population the ratchet exists to stop relying on.

Revised: land #3250 / #3248 / #3238, gate doctor_speed_real, ship hard-failing on day one.
The exit condition was already one test when the draft was written, so no interval existed in
which a ratchet carried information. Fallback if the gating stalls: one named exclusion with its
issue number, auditable in a line.

2. The 39 GB target dir was unbudgeted. Unanimous, and the draft did not mention it once — on
the same day I took gx10 from 0 bytes free after it dequeued a PR from merge-queue position 1
(#3247). New §4bis makes it a constraint: not on gx10 (916 GB, the box that hit the wall); a
per-run target dir reclaimed in-job rather than left to a reaper that already cannot reclaim under
keep=3; and a preflight free-space assertion that fails ENV: instead of letting cargo die
half-way and read as 650 new defects.

3. The revisit condition was wrong. I keyed it on queue depth vs max_entries_to_build; lane 3
pointed out that is a cap, not a measure of slack — a queue can sit at depth 3 with idle
capacity. Now keyed on arrival rate vs drain rate.

Also sharpened

T-1 does not "add the full surface" — it must build ci / deep. That job has 0 hits in
every workflow including the SHA-pinned upstream sovereign-ci.yml, and its other named parts run
nowhere either: cargo test --doc 0, cargo run --example 0, feature matrix 0,
--no-default-features 1 hit that is a Windows build flag.

And the fast-pool pin leaves this spec (§8bis)

All four lanes rejected it with a computable break-even rather than a caveat. runs-on is a hard
label filter with no fallback, so a pin is non-work-conserving and is a net loss whenever
fast-pool wait exceeds the ~18–22 min runtime delta — which is the condition that holds exactly
when the latency mattered. The asymmetry is real and measured (#3251); the actuator is wrong.

Revised in 0c6839ca7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1,424 of 1,469 integration targets run only when a PR touches their own crate — four of tonight's defects trace to this

1 participant