ci(mini): macos-arm64 leg on mini-m4 (NEON tests + core check) and the darwin nightly comes back - #3205
Open
noahgift wants to merge 30 commits into
Open
ci(mini): macos-arm64 leg on mini-m4 (NEON tests + core check) and the darwin nightly comes back#3205noahgift wants to merge 30 commits into
noahgift wants to merge 30 commits into
Conversation
…acos-arm64 leg on mini-m4, and the darwin nightly comes back Measured 2026-09-13T10:05Z: mini-m4's runner registration had been auto-deleted by GitHub on 2026-09-12 (idle >14 d), and even when online no aprender job could land on it — every selector names `Linux` or a `clean-room` docker pool, and macOS runs neither. Re-registered from the infra recipe with labels self-hosted,macOS,ARM64,apple-silicon,m4,mini (opt-in box label `mini`, one-box rule). This adds the work that only mini can do: `cargo check -p aprender-core` and the aprender-compute unit tests on the NEON path (measured rc=0 natively on mini, main @ 3786870), as a non-required ci job — one 16 GB runner must never hold a merge — and restores the aarch64-apple-darwin nightly build that nightly.yml said "comes back when a box for them does". actionlint learns the four labels. Operator 2026-09-13: "mini (you need this to be fulltime aprender build slave as well)". Closes #3204 Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
noahgift
enabled auto-merge
September 13, 2026 10:14
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
This was referenced Sep 13, 2026
Merged
check_roadmap_diff_additive.sh reported base id(s) missing at head: this branch forked before they landed on main, and a plain update takes one side of the file whole. Resolved as main's roadmap plus this branch's own new entries, each at its sorted slot among same-prefix peers. The other three roadmap guards pass on the result. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… four different versions, nothing looking
tools.toml names the fleet's pmat/bashrs and check_tool_versions.sh asserts
them — but only on whichever runner took the job. guard-cargo lands on
intel nearly every time, so for every other box the assertion had never
run once. Measured across the fleet 2026-09-13, against a declared pin of
pmat 3.39.0 / bashrs 7.0.4:
intel pmat 3.39.0 bashrs 7.0.1 one patch behind its OWN pin
gx10 pmat 3.39.0 bashrs 7.0.4 pmat declared nowhere, maintained by nothing
yoga pmat 3.31.0 bashrs ABSENT eight minor behind; no linter at all
mini pmat 3.20.0 bashrs 7.0.4 NINETEEN minor behind
yoga carries three `clean-room` runners and takes aprender PR work like
any other box. Every bashrs guard that landed there could only refuse:
check_shell_lint_ratchet.sh exits 2 — "without the linter this guard
cannot decide". Failing closed is correct, and a runner that can only
fail closed is not capacity.
All four hosts are now converged (paiml/infra: yoga gains a bashrs
declaration and pmat 3.31.0 -> 3.39.0; mini gains pmat; gx10 gains a pmat
declaration; intel's bashrs was replaced by rename(2), NOT by forjar,
because its rustup-installer resource carries a documented grenade —
re-running it rewrites the ~/.cargo/bin all sixteen runners share, which
that file calls "this fleet's most expensive recurring outage", and the
box was 16/17 busy).
7.0.1 -> 7.0.4 IS NOT A BEHAVIOUR CHANGE, and that was measured before it
was chosen, not after: both versions were built and run over all 195
scripts in scripts/ and returned the same 6 error lines, identical after
stripping positions. Skew would have been worse than absence — a
shrink-only ratchet comparing counts from two linters compares nothing.
FOUR EXPLICIT JOBS, NOT A MATRIX. infra#352: a guard must be able to SEE
the labels a job selects, so `runs-on: ${{ matrix.runs-on }}` would dodge
both actionlint.yaml and check_runner_labels.sh. Four jobs is the price of
being checkable. The mini leg uses `apple-silicon` for the same reason —
`mini` alone is not in check_runner_labels.sh's DISCRIM set.
Verified: check_runner_labels, check_no_hosted_runners (18 workflows),
check_workflow_env_defined (18), check_workflow_path_filters all green.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…cent statements — mini found it on its first run
`test_brick_profiler_reset_v2` did:
let timer = profiler.start_brick(BrickId::RmsNorm);
profiler.stop_brick(timer, 1);
assert!(profiler.total_ns() > 0);
with no work between start and stop. On x86 Linux the clock always ticked,
so it always passed. On Apple Silicon `Instant` is backed by
mach_absolute_time, whose tick is ~41.7 ns: two reads inside one tick
differ by exactly 0.
3325 passed, this one failed, on the fleet's FIRST mini-m4 run — which is
what a new silicon axis is for. Nothing about the profiler was wrong. The
test was measuring the host's clock granularity and calling it a profiler
property, which is the wall-clock-in-a-required-check class this repo has
been bitten by four times.
The precondition's job is to prove there IS state for reset() to clear. It
asserts that directly now, on the COUNT and the token total, both
deterministic on every platform. The reset assertions after it are
unchanged, so nothing the test used to prove is dropped.
Mutation-verified: removing the stop_brick call turns it RED at the
precondition, so the line is load-bearing and not decoration.
NOT A SWEEP. The two other `total_ns() > 0` assertions in this crate are
correct and stay — tile_and_profiler sleeps 100 µs first, checksum_coverage
feeds record_elapsed an explicit Duration. Neither reads the clock twice in
a row. This is one site, not a class.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… and mini was not in the matrix at all
Replaces the fleet-tools.yml I added an hour ago. That was a second
workflow asking a question this one already exists to ask, which is the
duplication this repo keeps paying for; the gap was never a missing
workflow.
TWO GAPS IN THE ONE THAT EXISTS.
1. IT DID NOT ASK. fleet-toolset.yml opens with "what does each box
actually have on it?" and ci_self_hosted_preflight.sh's DEFAULT_TOOLS
are `jq curl git python3 tar sha256sum rustup cargo`. pmat and bashrs
are not in that list and never were — the two tools every aprender
guard depends on, and the two that had drifted:
intel pmat 3.39.0 bashrs 7.0.1
gx10 pmat 3.39.0 bashrs 7.0.4
yoga pmat 3.31.0 bashrs ABSENT
mini pmat 3.20.0 bashrs 7.0.4
Four hosts, four combinations, one "pin". The daily fleet probe had no
opinion because it had no question. `--need pmat bashrs` puts them in
the record, and the new step turns the record into a PREDICATE:
tools.toml names the pin and check_tool_versions.sh fails closed when
the box it runs on disagrees. A recording is evidence; a comparison is
a gate; this workflow had the first and not the second.
2. MINI WAS NOT PROBED. The matrix is clean-room / gx10 / yoga. mini —
the box carrying pmat NINETEEN minor versions behind, and the one the
operator's 2026-09-13 ruling makes a full-time aprender build host —
was the one box outside the matrix, so its toolset decided builds and
nothing looked at it. Added, `cuda: false` because it is an M4.
The assertion runs BEFORE the upload; the upload's `if: always()` is what
keeps a red box's record, which this file's own header calls the record
most worth having.
I also corrected myself here: the first version of this comment said the
versions had been "recorded and never compared". They were not recorded at
all. Checked DEFAULT_TOOLS before shipping the sentence.
Verified: check_runner_labels, check_no_hosted_runners,
check_workflow_env_defined, check_workflow_path_filters,
check_perf_concurrency_groups all green; the preflight with
`--need pmat bashrs` records both with versions.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…build on a box, and nothing named it
Measured on mini 2026-09-13. A `sudo htop` left
/Users/<user>/.config drwx------ root staff
and from that moment cargo could not fingerprint ANY package whose build
script lacks `rerun-if-changed`:
failed to determine package fingerprint for build script for apr-cli
Caused by: failed to determine list of files in crates/apr-cli
Caused by: Could not read repository exclude
Caused by: Permission denied (os error 13)
gix probes $HOME/.config/git/ignore for the XDG excludes file. An ABSENT
path answers ENOENT and is fine; an UNREADABLE PARENT answers EACCES, and
gix treats that as a hard error. The message names a crate and a build
script and says nothing about $HOME, which is why it cost an afternoon.
`cargo check --workspace --lib` on mini went from IMPOSSIBLE to 1m01s over
79 packages with one chown. mini's own CI leg never noticed, because it
checks aprender-core and tests aprender-compute — neither is apr-cli. The
box looked healthy and could not build the workspace.
THREE ROWS, and two of the three polarities are what make it useful:
unreadable -> RED readable -> GREEN ABSENT -> GREEN
The third matters most. A check that flagged an absent ~/.config would fire
on every clean box and be switched off within a day.
Mutation-verified: deleting the probe turns row 11 RED and leaves 13 and 15
green, so the row is load-bearing and specific. Falsifier 16/16.
bashrs 0 errors on both files.
Same family as #3219 (root-owned target/ on the gx10 pools): a root-owned
artifact in a user tree, created by one sudo, breaking builds that never
mention it.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ce suite for the first time and it was the only failure `test_read_system_memory_total_mb` asserts `read_system_memory_total_mb()` returns `Some`, and its own doc line says why that holds: "On Linux `/proc/meminfo` always reports a positive MemTotal". There was no gate, so on darwin the reader returns `None` and the assertion fires. FOUND BY PUTTING WORK ON mini. Its CI leg checks aprender-core and tests aprender-compute; neither reaches this crate. Running the whole workspace `--lib` suite on darwin/arm64 for the first time — 3m43s, 79 packages — produced exactly one failure, and this is it. GATED, NOT SKIPPED, and darwin keeps a row. The Linux assertion is a claim about /proc and every platform the unified-memory VRAM fallback exists for (GB10, GH200, Jetson) is Linux. Off Linux the function must degrade to `None` rather than panic or invent a number, which is what callers depend on — so that is asserted instead. `#[cfg]`, never `#[ignore]`. Linux: 10 passed, 0 failed in profilers::system. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…, which is the only darwin coverage the repo has
Measured, job 103736930145: 5s checking aprender-core, 20s on the SIMD
tests, 32s total, while intel spent an hour on the same commit. mini is 10
cores / 16 GB and the operator's ruling makes it a full-time aprender build
host. Two crates is neither coverage nor work.
MEASURED ON THE BOX BEFORE BEING WRITTEN INTO THE WORKFLOW:
cargo check --workspace --lib 1m01s 79 packages
cargo test --workspace --lib 3m43s warm
7x the work, and the only darwin/arm64 coverage that exists here.
IT PAID FOR ITSELF ON THE FIRST RUN. That suite produced exactly one
failure — `test_read_system_memory_total_mb`, asserting /proc/meminfo
exists, in a test whose own doc line begins "On Linux". No Linux runner can
find that class, by construction. Fixed in the previous commit.
THE THREE EXCLUSIONS are workspace-test's, for its reason: aprender-gpu,
aprender-cuda-edge and aprender-compute need a CUDA toolchain.
aprender-compute is still tested on its own above, natively — that is the
point of this runner.
--test-threads=4 rather than the default 10: a 16 GB box running 10 test
threads across 79 packages is how a runner starts swapping and reports a
timeout instead of a result.
TIMEOUT 40 -> 60, with the basis stated AND its gap stated. BSE-05 derives T
from a job's own p99 and this step has no history; 3m43s was WARM, and the
first run after a cache wipe builds 79 packages cold. That number is
unmeasured, so the timeout is wide enough for the first run to report a
RESULT instead of a timeout, and is to be tightened from evidence once the
leg has a p99 like every other job.
Verified: check_workflow_env_defined, check_runner_labels,
check_no_hosted_runners, check_no_timing_in_required all green.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…in one process, so eval could run the wrong program
Found by the expanded mini leg on its first real run: 7237 passed, 1 failed
on darwin/arm64.
---- assertion_failure_reports_nonzero_and_traceback ----
assertion failed: !r.success
The test writes `assert 1 == 2` and requires a non-zero exit. On mini
`python3 -c "assert 1 == 2"` exits 1 (3.9.6, sys.flags.optimize=0, checked
on the box), so the program that ran was NOT the one written.
let tmp = temp_dir().join(format!("apr_eval_{}_{}.py",
std::process::id(), SystemTime::now()...as_nanos()));
Ten tests in that module call this, and every eval run calls it once per
test program. Under threads the PID IS CONSTANT, so uniqueness rested
entirely on `SystemTime::now()` resolving differently for each caller —
which it is not required to, and on darwin does not. Two callers landed on
one path and one executed the other's file.
THIS IS NOT A TEST PROBLEM. `apr eval` runs USER test programs through this
function; a collision silently executes the wrong program and reports its
result — a wrong answer, not an error, which is the worse of the two.
tempfile::Builder creates with O_EXCL and a random suffix, so uniqueness
comes from the kernel rather than from a clock this code cannot control.
tempfile is already a [dependencies] entry, not just dev.
DECOMPOSED IN THE SAME COMMIT, because the pre-commit gate refused the edit:
inference.rs carries debt and the added match took
execute_python_test_with_diagnostics to Cognitive 28 against a ceiling of
25. Extracting new_eval_program_file() brings it to 25 and gives the
uniqueness rule one place to live.
WHAT IS AND IS NOT PROVEN: on Linux the module passes 5/5 across three runs
at --test-threads=8 — that shows no regression and is NOT proof, because
Linux's finer clock resolution is why the collision never surfaced there.
darwin is where it failed and where it has to pass; mini's leg is the
verification.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…de asserts its own kernel The ETXTBSY falsifier asserted that a held write fd makes a direct spawn fail with errno 26. That is a Linux guarantee. On mini-m4 the same held fd lets the spawn straight through, so the row reddened the whole macos-arm64 leg on a premise that is false there (left: None, right: Some(26)). Neither a #[cfg] skip nor a relaxed assertion: darwin now asserts the ABSENCE of the interlock — the direct spawn must succeed AND must have run the shim — so the row still measures something on both platforms, and the day darwin grows the interlock it turns RED instead of passing quietly. Linux branch re-mutated: Some(26) -> Some(99) turns it RED with left: Some(26), proving the kernel really is refusing the exec and the retry loop is load bearing. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… filter test assumed the clock ticks between two statements mini ran the whole workspace natively for the first time and the darwin leg found two more Linux-only assumptions. 6563 passed, 2 failed. 1. CacheEntry::is_expired -- A PRODUCTION BUG, not a test defect. `Instant::now() > self.expires_at`, with `expires_at = now + ttl`. For `Duration::from_secs(0)` that is `now > created`, which needs the clock to have STRICTLY advanced since the constructor. On darwin/arm64 mach_absolute_time advances in ~41.7 ns steps, both reads land in one tick, and a zero-TTL entry answers "fresh". Zero TTL means expired on arrival; that is the whole meaning of Duration::ZERO. The sibling is the proof: the on-disk CachedResponse::is_expired twelve lines below has always used `now >= self.expires_at`. Two spellings of one predicate disagreeing is the defect, and `>=` is the half that is right. 2. test_filter_by_since -- the test, not the filter. It stored one fragment, read Utc::now() on the next line, stored a second, and asked for everything at-or-after the cut. chrono is microsecond resolution on darwin, both writes landed in one tick, the filter (`f.created_at < since` skips) kept both, and 2 != 1. Linux nanoseconds are why no Linux runner can find this class. One 5 ms sleep, BEFORE the cut and not after: the second fragment only needs created_at >= after_first, which an equal timestamp already satisfies. The filter semantics are unchanged. aprender-orchestrate --lib: 6535 passed, 0 failed on linux; clippy clean. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…arser behind five levels of nesting workspace-test went red on this branch with thread falsify_cgp_043_profile_binary panicked at falsify.rs:209:5: assertion failed: output.status.success() twice (nextest TRY 1 and TRY 2, gx10-pool1) and that is the entire report: not the exit code, not stderr, not whether nvidia-smi is on the box. The row runs `cgp profile binary nvidia-smi`; an absent nvidia-smi or nsys is an ENV death and a present one that still failed is a CODE defect, and those need opposite responses. The assertion now prints the status and both streams and names the fork. Nothing on this branch touches aprender-cgp/tests/ -- the only cgp change here is 25 cfg-gated lines in src/profilers/system.rs that compile to nothing on linux. Landing that one-line edit meant paying the file down: the pre-commit gate refuses any change to a file carrying debt, and falsify_cgp_090_trueno_gemm_ at_peak was Cognitive 37 against a ceiling of 30 -- five nested levels inside a #[test], which also made its parsing untestable without a GPU and a benchmark run. Extracted UNCHANGED into first_measured_avx512_gflops + measured_value_on_row, both pure over a string. Max Cognitive in the file: 37 -> 20; CGP-090 itself is now Cognitive 0. WHAT THE EXTRACTION EXPOSED. "first parseable field after index 1" is the TIME column on a `backend label time tflops` row, not TFLOP/s. FALSIFY-CGP-090 has been asserting `time * 1000 > 100` -- microseconds wearing the name GFLOPS. The two new rows pin the behaviour as it IS, with the quirk named in the doc comment: moving to the right column changes what that gate asserts and is not something to smuggle in under a complexity decomposition. Filed separately. Also observed and NOT fixed here: falsify_cgp_quant_all_001_summary fails on this branch AND with the branch stashed, so it is pre-existing. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… better I wrote a diagnostic assertion for FALSIFY-CGP-043 without first checking whether an open PR already owned that file. #3210 does, it landed first, and its fix is the better one: it PROBES for nvidia-smi and nsys and declines to answer where they are absent, saying so out loud -- FALSIFY-CGP-043 NOT RUN: {tool} is absent, so the nsys profiling claim cannot be falsified on this host. This is an unmeasured gate, not a passing one. -- where mine only made the eventual failure legible. It also names the root cause my version guessed at: clean-room runners have no nsys. The CGP-043 region of this file is now byte-identical to origin/main, so the two PRs no longer collide. What stays is the CGP-090 decomposition, which is in a different region, is what the pre-commit complexity gate requires of any edit to this file, and is what surfaced #3231 (CGP-090 asserts on the time column and calls it GFLOPS). Max Cognitive in the file: 20, from 37. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eness rested on the clock ticking
mini found the third instance of one defect today. `uuid_v4()` is neither a
UUID nor v4 nor random -- it is `format!("{:x}{:x}", nanos, pid)` -- and the
pid does not change within a process, so two calls collide whenever the clock
does not advance between them. On darwin SystemTime is microsecond
resolution: 100 calls in a loop produced fewer than 90 distinct ids and
test_uuid_v4_generates_unique_ids went red on mini-m4 (job 103767…).
A monotonic AtomicU64 makes it unique within the process by construction at
any clock resolution; nanos and pid still separate processes. Format is now
`<nanos>-<pid>-<seq>`; call sites only assert `starts_with("run-")`.
TWO THINGS I CHECKED RATHER THAN ASSUMED:
* The existing row is NOT load bearing on Linux. Deleting the counter leaves
test_uuid_v4_generates_unique_ids GREEN here, because nanosecond timestamps
carry the uniqueness on their own. So the counter is asserted DIRECTLY, in
its own separated field, by a new row that reddens everywhere:
the counter must advance by one per call, got [1954378, 1954378, 1954378, ...]
(that repeated number is the pid, which is what the last field becomes once
the counter is gone).
* `>= 90` became `== 100`. The slack existed to absorb clock collisions, and
absorbing them is precisely how this survived -- a microsecond clock simply
pushed the rate past the slack. With the counter the guarantee is
unconditional, so it is asserted as one.
aprender-test-lib --lib: 6745 passed, 0 failed; clippy clean.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…reservation read as DEAD and was silently released
mini ran aprender-train natively and 21 gpu::guard tests failed with
`left: 0` against `right: 5000` -- every reservation gone the instant it
was made.
/// Whether the holding process is still alive (Linux /proc check).
pub fn is_alive(&self) -> bool {
Path::new(&format!("/proc/{}/stat", self.pid)).exists()
}
There is no /proc on darwin, so that answered FALSE for every reservation,
should_prune() answered true for every reservation, and total_reserved() was
permanently 0. The doc comment names the Linux assumption and the code ships
it unconditionally. This is production logic, not a test.
THE DIRECTION OF THE DEFAULT IS THE DEFECT. "Cannot determine, so assume
DEAD" releases live reservations, and two trainers then both believe the VRAM
is free -- an OOM. "Cannot determine, so assume ALIVE" holds a dead process
reservation until its lease expires, which costs a wait and is bounded by
lease_hours. A resource guard fails towards the conservative answer, so the
non-Linux arm returns true.
sysinfo is already a workspace dependency and would answer portably and
exactly; it is not used because it refreshes a process table per call and
this runs once per reservation per ledger read. Named in the doc comment as
the upgrade path.
BOTH ARMS ARE ASSERTED, so neither is vacuous. Linux: our own pid alive AND
an impossible pid (0x7FFF_FFFF, above any pid_max) dead. Non-Linux: both
alive, and an unexpired lease held by an unknown-liveness pid not pruned --
the impossible-pid half is the load-bearing one, since that is exactly what
the old code got wrong. Mutating the Linux arm to `true`:
linux: an impossible pid must read as dead, or the check is vacuous
I CHECKED THE OBVIOUS ALTERNATIVE FIRST and it was wrong: mini runs
`cargo test -- --test-threads=4` where CI runs nextest (process per test), so
I expected shared-state coupling. The identical invocation on linux is
7624 passed, 0 failed. It really is the platform.
aprender-train --lib: 7625 passed, 0 failed; clippy clean.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… what their own platform does My is_alive fix dropped mini from 21 failures to 2, and the 2 are tests that encoded the behaviour I changed: test_reservation_is_alive_dead_process assert!(!reservation.is_alive()) test_ledger_data_prune_dead assert_eq!(reservations.len(), 1) Both use pid u32::MAX as "a dead process", which is only dead where liveness is knowable. I should have looked for callers of the predicate when I changed it rather than finding them on the next darwin run. Split rather than skipped, both times. A bare #[cfg] would leave darwin proving nothing about the very predicate that broke it: linux a u32::MAX pid is dead, and should_prune agrees not-linux it reads ALIVE, and an unexpired lease is NOT pruned prune_dead keeps a platform-independent half: the EXPIRED row is pruned everywhere, because lease expiry needs no /proc. Only the dead-pid row is conditional -- linux expects 1 survivor, darwin 2, and darwin names both. aprender-train gpu::ledger: 38 passed, 0 failed; clippy clean. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wall clock with the invariant The fifth instance of one family on darwin today. mini: assertion failed: stats.generation_time_ms > 0 || stats.total_generated < 10 Ten tiny programs, `elapsed.as_millis()` truncating, an M4: 10 generated in 0 ms, so both disjuncts are false. The row asserts that the machine is slow enough to be measured, which is not a property of the code under test. The production code already knows zero is reachable -- PipelineStats:: throughput() returns 0.0 for generation_time_ms == 0 rather than dividing, and predictions_per_sec guards total_time_ms > 0.0 the same way. The tests were the only things pretending otherwise. Replaced with what each row was reaching for: pipeline.rs total_generated == programs.len() (it IS all_programs.len()) evaluator.rs total_time_ms and predictions_per_sec are >= 0 and FINITE evaluator.rs:637 had not failed yet -- a closure returning a constant over 1000 predictions is the same sub-resolution shape and would have cost another darwin round trip, so it is fixed here rather than found later. This repo already has the rule: NO wall-clock assertion in a required check (four have failed; a "clever" ratio rewrite once blocked all nine open PRs). aprender-verify-ml --lib: 1145 passed, 0 failed; clippy clean. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t cost six round trips
Plain `cargo test` stops after the first failing test BINARY. Without
--no-fail-fast this leg reports one crate and stops, so each darwin fix buys
only the right to discover the next one ~25 minutes later. That is literally
what it cost today:
apr-cli -> aprender-mcp -> aprender-orchestrate -> aprender-test-lib
-> aprender-train -> aprender-verify-ml
Six runs for six INDEPENDENT defects that one run could have listed. Four of
them are production bugs, so the cost was not hypothetical.
workspace-test avoids this by running nextest, whose measurement note in this
file was taken with --no-fail-fast.
I nearly shipped "nextest is not installed on mini" as the reason this leg
uses plain cargo test. It IS installed (0.9.137, checked on the box). The
real reason to keep cargo test is that nextest gives every test its own
PROCESS while `-- --test-threads=4` shares one -- a different isolation model
and therefore different coverage, which is worth having on the one platform
nothing else covers. The flag simply has to be written out rather than
inherited from a nextest profile.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
The leg is green, and here is everything it cost and provedFirst green. Seven independent defects had to be fixed to get here, four of them production bugs rather than test defects:
Five of the eight share one root cause: code that assumes the clock ticks between adjacent statements. Linux's nanosecond clock hides it; darwin's microsecond clock does not. No Linux runner can find this class, by construction — which is the entire argument for this leg. The VRAM one is the serious one. "Cannot determine liveness, so assume dead" releases live reservations and two trainers then both believe the memory is free. It now assumes alive, bounded by the lease. What cost the most, and is now fixedThe leg had no I also nearly shipped "nextest is not installed on mini" as the justification for using plain What I checked and found CLEANNegative results, so nobody re-does them:
What remains UNMEASURED on darwin40 files under 🤖 Generated with Claude Code |
This was referenced Sep 14, 2026
…ped the other
workspace-test failed again on this PR, run 34806202374, same class and a
different file:
aprender-test-lib::falsify_playbook_drives_a_browser
a_playbook_click_changes_the_real_page
evaluate_is_decided_by_the_page
screenshots_are_written_as_real_files
could not launch a browser-backed executor
30,481 passed, 1 failed. Same mechanism as the previous commit:
aprender-orchestrate's dev-dependency carries features = ["browser"], the BSE-17
quick tier builds the selected crates' --lib --tests in ONE graph, and this
crate's own browser-gated target compiled and ran on a browser-less clean-room
runner.
The survey that found the first file printed `2 #![cfg(feature = "browser")]`.
I read the count and acted on one file. That is the same incomplete-fix shape
this branch's book.yml work is about -- "the fix stopped at the step that hurt"
-- committed one commit after writing that sentence.
Both targets now gate on `browser-falsify`. Verified in both directions:
--features browser --test falsify_playbook_drives_a_browser -> 0 tests
--features browser-falsify --test falsify_playbook_drives_a_browser -> 6 tests
and `grep -rln '^#!\[cfg(feature = "browser")\]' crates/*/tests/` is now 0, which
is the check I should have run the first time rather than reading a count.
#3246's ratchet is the mechanical version: `aprender-orchestrate [dev] ->
aprender-test-lib/browser` is in its baseline precisely because `browser` still
gated a target, so this change makes that baseline SHRINK. A count I read is not
a check; a baseline that moves is.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…as none
workspace-test, run 34809941798: 39,613 passed, 2 failed --
aprender-verify-ml::book verification::oracles::test_io_oracle_example
aprender-verify-ml::book verification::oracles::test_python_executor_example
assertion failed: verdict.is_ok()
Third instance tonight of one class: a test that asserts a RUNNER CAPABILITY,
gated by nothing, compiled and run because the BSE-17 quick tier reached its
crate. The browser falsifiers were the first two.
MEASURED, not inferred, on both sides of the seam:
docker run --rm localhost:5000/sovereign-ci:stable command -v python3
-> NO_PYTHON3 (and NO_PYTHON; rustc IS present)
and the SAME built test binary, run twice:
with python3 on PATH 3 passed, 1 ignored
with an empty PATH 3 FAILED, 1 ignored
So it is the interpreter, not the code, and not the platform.
CI reported 2 failures, not 3: `test_verification_with_input_example` did not run
in that selection. It fails identically without an interpreter -- the run above is
how I know -- so the gate covers the module, not the two rows that happened to be
reported. Fixing only what the log named is what left the second browser falsifier
armed two commits ago.
`python-oracle`, on `mod oracles;`, and nothing enables it. A capability needs a
name no sibling reaches by accident; `browser-falsify` in aprender-test-lib is the
same shape, and #3246's ratchet is the mechanical version of not letting one leak.
NO SKIP. Where `python-oracle` is on, the rows still fail hard without python3 --
they are examples from the book and their whole point is that the executor really
executes. Verified both directions:
(default) --test book verification:: -> 0 tests, module compiled out
--features python-oracle --test book ...::oracles -> 3 passed, 1 ignored
Also dark, and not fixed here: `grep -c aprender-verify-ml .github/workflows/ci.yml`
is 0, so this target has never run in CI. It surfaced only because this PR's
selection reached the crate. And `test_timeout_handling_example` carries a bare
`#[ignore]` ("Timeout handling currently hangs - needs executor fix") which the
repo bans; it predates this change and needs its own ticket.
The close-out is python3 in the clean-room image, then arm the feature. Measure
first, assert second.
Pmat-Ticket: PMAT-1098
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
noahgift
added a commit
that referenced
this pull request
Sep 14, 2026
…ext in seven places Read start to finish. Every finding is a place where a dated amendment (§3.4 merge-queue parallelism, §3.5 SSH, §6.1 continuous triage, #3205 mini, the one-subagent rule) was landed beside original text that still said the opposite, so a reader could cite either. - §1 measured the fleet with `gh api …/actions/runners` — the call the operator rejected, a `busy` snapshot that cannot see ephemeral runners, and the hourly average that read 9.5 % while 15/16 workers were busy. Now the fleet-pack ledger record, instantaneous busy/online, both traps named. - §1 said "one PR in CI at a time means gate latency IS throughput"; §3.4 was amended to 3-parallel on 2026-09-12. Bound is now 3 × 72 h / p95. - `mini` is a declared full-time build host (#3205) and appeared only in a §8 bullet. Added to §0 row 0, §1, §5 P0·Pack fields and Done, §7 pack:, §8. - §0 row 3 still scheduled triage once per train; §6.1 made it per-wakeup. - §2 named the required check `ci / gate`; the rules API says `gate` and `workspace-test`, and `present` is not required. - §3.4 allowed "≤ 3 read-only subagents" against the one-at-a-time rule and §10's fan-out-through-agy. - §8's last bullet stopped on "a second concurrent aprender PR in CI" and on "SSH into a host" — both allowed by the amended §3.4/§3.5. Now stops on a host CONFIG change over SSH instead of forjar. - §9 asked for the 0.67 cascade wall to be measured; it was: 70 min, attended 0. That is 3.5× the [A] line, so by §9's own rule the cascade is the next kaizen target; where the minutes go is [U]. - §7 train: line said T-0..T-4; T-5 exists. §6's tail paragraph gets a §6.4 heading. §11.3 cited the one-PR rule §3.4 no longer has; fixed. - `make build-report` does not exist on main (P0·Instrument not done) — said so where p95 is marked [U]. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 14, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 14, 2026
noahgift
added a commit
to guyernest/aprender
that referenced
this pull request
Sep 15, 2026
…ment that outlives a session (paiml#3244) * ci(unwedge): the zombie aggregator that cost 8.5 hours has no containment that outlives a session paiml#3229's proposed fix -- give the aggregator `if: always()` -- is ALREADY IN PLACE (paiml/.github@main sovereign-ci.yml:1474) and the wedge happens anyway. Measured, not read: the line has been there the whole time. It is nearer the cause than the cure. `if: always()` is what gets the job SCHEDULED during a cancellation instead of skipped; it enters `queued`, the run is cancelling so no runner is ever assigned, and `timeout-minutes: 5` cannot fire because a job timeout starts when the job STARTS. That is why nothing times out. The control is in this repo: ci.yml's own `gate` carries no `if:` at all, is skipped on cancellation, and reaches a terminal state. The zombie is always `ci / gate`, never bare `gate`. The candidate fix is `if: ${{ !cancelled() }}` -- untested, one line, in an org-wide reusable workflow every repo consumes and that aprender pins by SHA. Asserting it from a reading is the move that made paiml#3229 wrong the first time. So: containment, with an expiry condition stated in the workflow header. Cost this is containing. Main merged NOTHING for 8.5 hours on 2026-09-13 with three merge groups wedged. It recurred on paiml#3205 at 04:28Z: old run queued with one pending job (`ci / gate`), new run created with ZERO jobs; force-cancel gave the new run 14 jobs within ~45 s. It is silent -- the branch shows a run "in progress", the required checks are never created, and the PR sits BLOCKED with no red anywhere to look at. THE PREDICATE NEEDS BOTH SIGNALS. `jobs == 0` alone is not one: run 34803859272 read jobs=0 for a moment while fourteen real jobs were coming, and I called it a zombie on that basis once already. Wedged means all three of: (a) >=1 job reached failure or cancelled, (b) >=1 job is still pending, and (c) EVERY pending job is an aggregator (^(ci / )?gate$). (c) is a pattern, not a count. "<= 1 pending job" both MISSES a group wedged on two aggregators (`ci / gate` AND `gate`) and FIRES on a run genuinely down to its last real job. Both are rows. Nine rows over committed fixtures, no network: three wedged shapes, five healthy, one ENV. Every guard is mutation-proved load-bearing -- aggregator pattern matches everything H1, H5 the discrimination rows drop the "nothing stopped" guard H2 drop the "all terminal" guard H3 drop the "real work pending" guard H1, H5 all three removed H1..H5, including jobs == 0 A BARE invocation runs the case table, never the scan. guard_tree.sh runs every scripts/check_*.sh with no arguments, and a guard that needs network and actions:write to answer must not be what a bare run reaches. The predicate's soundness is the assertion here; the scan is an ACTION, wired by a workflow that holds the permission -- and that workflow runs the falsifier as step 1, so the instrument proves itself before it is allowed to cancel anything. Verified: --self-test 9/9 green; live `--scan --dry-run --limit 12` against this repo looked at 5 non-completed runs and freed 0 (correct -- the one live zombie had already been cleared by hand); bashrs 0 errors; check_guards_are_wired baseline did not grow; check_no_hosted_runners, check_ci_reusable_workflow_pinned, check_no_timing_in_required, check_runner_labels, check_perf_concurrency_groups, check_cargo_install_private_root, check_hardcoded_paths and check_tool_versions --audit-workflow all pass on the new file. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci(unwedge): 40% of the merge-group load was computing verdicts for refs that no longer exist A second class the same sweep can see, orthogonal to the parked aggregator. Every merge rebases the groups behind it, so GitHub deletes those `gh-readonly-queue/...` refs and creates new ones -- and leaves the old CI runs building. They keep drawing runners to compute a verdict on a ref that returns 404. Nothing can ever read it. Measured 2026-09-14 04:43Z with a 9-deep queue and the packer reporting gx10 60% / yoga 49%: five non-completed merge_group CI runs, and TWO of them had dead refs -- live 34805626883 jobs=15 pr-3006-b360a4 live 34805625229 jobs=15 pr-3056-d3c2f4 live 34805623711 jobs=15 pr-3060-73adf6 DEAD-REF 34805561949 jobs=16 pr-3006-0c740b gx10-pool3, intel-clean-room-8 DEAD-REF 34804495711 jobs=16 pr-3056-e2ef10 intel-clean-room-14 Three runners held, more jobs queued behind them, 40% of the merge-group load. Both were also already red, so even a live ref would have dequeued the group. NOT redundant with the wedge predicate, and the proof is in the data: run 34805561949 still had `workspace-test` pending, so the wedge predicate correctly answered HEALTHY. It was dead for an entirely different reason. Two predicates, one sweep. `gh run cancel` does not clear these either -- cancelling leaves the aggregator parked and the run `queued`, which IS the wedge above. Measured on both runs tonight: cancel requested, both still `queued` 65 s later, both cleared by force-cancel. VACUITY IS THE WHOLE RISK. If the branch -> ref derivation breaks, every run reads 404 and a per-row janitor cancels the entire queue. So collect first, decide second: at least one merge_group candidate must resolve LIVE before any DEAD verdict is acted on. Zero live with one or more dead means the LOOKUP is broken, not the fleet -- refuse, exit 2, cancel nothing. Six new rows, 15 total, and the two new guards are mutation-proved: drop the corroboration gate (always ACT) D3 cancels the queue on a broken lookup any branch is a queue ref D2 a PR run becomes a candidate Verified: --self-test 15/15; live `--scan --dry-run --limit 25` names exactly the dead-ref run and no live one; bashrs 0 errors. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(unwedge): the predicate would have force-cancelled the head of the merge queue Caught by running --scan --dry-run against this repo before trusting the table, which is the only reason it is a near-miss and not an outage. At 04:49Z the sweep proposed freeing run 34805623711 -- paiml#3060's merge group at QUEUE POSITION 1. Its real job list: completed/failure guard-tree completed/success workspace-test, guard-cargo, vendored-schemas, ci / provenance, ci / lint, ci / coverage, ci / test, ci / security completed/skipped 6 more queued/- ci / gate Perfectly alive. `guard-tree` FAILED, everything else finished, and `ci / gate` was queued waiting for a runner on a saturated fleet -- it would have got one and reported the failure. Force-cancelling it would have destroyed the verdict at the head of the queue. Condition (a) said "at least one job reached failure OR cancelled". That is two different worlds: A FAILURE is an ANSWER. The gate still runs, reads needs.*.result, and reports it. The run terminates on its own. A CANCELLATION is what a supersede leaves behind, and it is the only state in which `if: always()` schedules the gate into a run that can never assign it a runner. That is the whole of paiml#3229. So (a) is now "at least one job was CANCELLED". Everything else is unchanged. The contrast is committed rather than described: h6_failed_not_cancelled.json is that run's actual job list, and reverting the condition turns exactly H6 red -- M8 select(.conclusion == "cancelled") -> select(.conclusion == "failure" or "cancelled") H6 The same live dry-run now names only the genuine zombie (this PR's own superseded run: 8 cancelled, `ci / gate` parked) and leaves paiml#3060 alone. 16 rows. H1 and H5 were never enough on their own: both carry a failure and a pending REAL job, so they discriminate condition (c) and say nothing about (a). H6 is the row that discriminates (a), and there was no such row until a live run produced one. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(unwedge): DET002 on the operational timestamp — and my local bashrs could not see it guard-cargo step 19, `bashrs SEC/DET/IDEM: the release gate, on the PR (paiml#3196)`: scripts/check_ci_unwedge.sh:303 DET002 bashrs-gate: FAIL 1 SEC/DET/IDEM error(s) over 300 file(s) `$(date -u +%FT%TZ)` in the sweep's summary line. A wall-clock stamp on an operational log line is the POINT of the line -- it says when the sweep ran -- so the fix is the suppression bashrs itself names, with the reason written down, which is the idiom this repo already uses at check_llama_pin.sh:248 and ci_target_watch.sh:158. Applied at BOTH sites, not just the flagged one: the second `date -u` is the same construct and only accidentally unreported. WHY I DID NOT CATCH THIS LOCALLY, stated rather than glossed: this box has bashrs 7.4.1 and the fleet pin is 7.0.1 (tools.toml). 7.4.1 does not raise DET002 here; the pinned 7.0.1 in the clean-room image does. `bashrs lint scripts/check_ci_unwedge.sh` printed `0 error(s)` and was measuring a different instrument -- exactly what check_baseline_ratchets.sh's tool_version rows exist to make impossible for baselines, and what tools.toml's header calls "the defect this fixes". So the local re-run is NOT the proof. `scripts/check_bashrs_gate.sh` passes here (300 files, 0 SEC/DET/IDEM) but it passed before the change too, under 7.4.1. The discriminating run is CI's own gate under the pin. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
noahgift
added a commit
to guyernest/aprender
that referenced
this pull request
Sep 15, 2026
…tinuous triage, decision procedure, ontology kaizen (§11), and the chain of reasoning (§12) (paiml#3268) * docs(spec): APR-RELEASE-001 §4.1/§4.2/§5.1/§10 — the work the train did not name A day spent on T-1 surfaced five things the spec did not cover. Four are now clauses; the fifth turned out to be covered already and is only made precise. §4.1 FEATURE MATRIX — T-1 named it from the start and it was never built. Its first run (paiml#3262) measured 100 of 430 (crate, feature) pairs RED, every one unreachable from any default set — which is exactly why it could go dark: `cargo check --workspace` stays green over all of it because feature unification hands each crate whatever its siblings enabled. The section defines the universe (per-pair from `cargo metadata`, never a powerset — aprender- orchestrate alone declares 78 features), the five shapes that accounted for all 100, the known-red list and why a listed pair that PASSES must be fatal, and the `compile_error!` + private `__x-linked` form for a feature that cannot be built at all. Struct drift is listed as shape 5 because nothing else in the train watches for an upstream type growing a field. §4.2 EXAMPLES — T-1 says every `cargo run --example`, which is a different clause from building them (83 s vs ~2 h). Measured: three of five random examples ran past a 60 s cap. They are compute demos, not CLIs, so TIMEOUT IS A PASS and the assertion owed is "starts and does not crash". Asserting a duration would be a wall-clock assertion in a required check. Both clauses carry a vacuity floor: a discovery that finds nothing reports zero failures, which reads exactly like a pass. §5.1 THE DEBT TAX — the pre-commit gate refuses any commit touching a file with a function over cyclomatic 30 / cognitive 25, and `--no-verify` is banned, so a one-line fix costs the decomposition of every offender in that file. Measured in one day: 11 pre-existing violations paid down, worst cognitive 91, 73, 61, none in code that day's changes wrote. It is not a build row because it is not schedulable — it is a toll on whatever you touch. Now it is at least measured: `debt:` in §7. §10 DECISION PROCEDURE — §6 is deliberately "no judgement calls", so design forks had nowhere to go. Codifies what worked: fan out through agy not Claude subagents; the brief carries the measurements so lanes do not each measure the premise differently; plant one trap question; a verdict is a claim until the orchestrator re-runs the acceptance command; **a premise error voids the vote and the fix is another round, not the orchestrator's judgement** (paiml#3179 round 2 overturned round 1 unanimously once three facts were read out of the tree); overriding the majority is allowed once, only on a fact no lane had, and must be recorded with the losing argument quoted; prefer the reversible option when the vote is close. §6 — `untriaged` must be counted PER SURFACE. Measured 2026-09-14: issues were 319/320 triaged while PRs were 20 of 34 with no milestone at all, and one number reported the clean surface while hiding the breached one. Also states plainly that triage is not disposal: the ledger grew net +149 over ten days while ~100 % triaged, with 312 of 320 open issues opened by the agent itself. This is the same finding §6 already recorded for the 0.67 train ("filing was the work product and closing was nobody's"); it now has a stop rule. §8 — three stop conditions: a declared full-time build host at 0 % occupancy while a queue has pressure (mini, measured all of 2026-09-14) is a routing defect, not spare capacity; a stale known-red list; a design fork goes to §10. readme_contract 15/15. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(spec): §6.1/§6.2/§6.3 — triage is continuous, covers branches, and a milestone must fit its date Three gaps in §6, each measured 2026-09-14. §6.1 CADENCE. "Once per train" is what let 20 of 34 open PRs carry no milestone at all, every one opened in the preceding two days. A train is 48-72 h; a PR opened an hour after the pass is invisible for the rest of it. Triage now runs on the P0 · Pack wakeup, beside the fleet sample — same cadence, same receipt, and equally P0 per the operator ("ticket, pull requests, branches that are not triaged are P0"). The per-wakeup pass is mechanical and bounded; the once-per-train pass keeps only what needs the whole window: the §6.3 capacity check and the T-5 reconcile. §6.2 BRANCHES were the unwatched surface. 107 remote branches, 35 with an open PR, 72 without: 32 younger than 7 d, 27 in a 7-14 d band NO RULE LOOKS AT, 13 already R-3-eligible. R-3 archives a branch with no PR and a tip older than 14 d, so work that stalls on day 8 is invisible for six more days and is then deleted without ever having been seen. New test: no open PR and a tip older than 7 d must get a PR (draft is fine) or be archived now. A branch with no PR is not work in progress, it is work nobody can see. §6.3 PRIORITISATION. §4 says scope is assigned after the fact — right for what a train CONTAINS, wrong as a plan for what it PROMISES. With no capacity rule a milestone is a dumping ground with a date on it. Measured at closure = 6.1 issues/day over the trailing 7 days: 0.68.0 280 open due in 1 d needs ~46 d OVER BY 45 DAYS 0.69.0 53 open due in 4 d needs ~9 d over by 5 d 0.70.0 15 open due in 7 d needs ~2 d fits A date 45 days of arithmetic away from its content is not a commitment; it is a label, and every number computed from it is fiction. The rule is arithmetic, so it stays inside §6's no-judgement-calls design: capacity = days_remaining x measured closure_rate_p50. Over capacity is reported every wakeup, not treated as an error. At T-0 an overcommitted next milestone SPILLS lowest-priority-first until it fits — P0 never spills, then P1, then unlabelled, then oldest kept. The operator sets priority by labelling; the arithmetic sets the cut line, so no train needs a judgement call about scope. A P0 set that alone exceeds capacity is a STOP (§8): that is over-promising at the one level the operator controls, and only the operator can cut it. closure_rate is MEASURED; under 7 days of data it reports [U] and spills nothing. Arrival is the other half: 6.1/day closure against a ledger that grew net +149 in ten days means the cut line moves further out every train however it is drawn. R-5 is the control on that; capacity only decides what a date may claim. readme_contract 15/15. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(spec): §11 — a surface sweep ends in `pv`, or it did not end ONT-001 v4.3 §6 assigns aprender every ontology row but three. Measured at fa6e35f: 1 of 17 merged, 0 of 1818 contracts carry `entity:`, `shape:` or `evidence:`, no `pv census`, no `pv extract`, no `ontology/` module. Two of those measurements are the reason for this section. `pv kaizen` IS the kaizen loop and it is code-only — bindings, call sites, E0/E1/E2 assertions. The train sweeps features, examples, README, CLAUDE.md, workflows, model files and CSVs, and the loop that is supposed to improve on each sweep cannot see one of them. The upstream spec is UNTRACKED in infra. No commit, no history, unfetchable from gx10, yoga or mini — so a quorum lane cannot read the premise at all and every ontology verdict it returns is unverifiable by construction. Stop condition, fixed in infra (ONT-P), not here. §11.1 makes the rule mechanical: a surface sweep closes with one of four deltas — an entity type + extractor, a shape whose violation is the defect class just found, a new Unknown{} reason, or a `resolves:` target — or with a named `ont-delta: none <reason>`. Unnamed is P0. This spec's own §4.1 is the counter-example: 100 red pairs as an awk matcher inside night.yml, no contract, no shape, re-derived by hand before every undraft. §11.4 is why this makes the quorum more effective, which is the point. Premises cite ids, verdicts are ONT-6 lattice elements, reduce is meet=min rather than a vote count, and the planted trap becomes Unknown{PositiveControlFailed} by rule instead of by the orchestrator noticing. paiml#3179 round 1 was a 2/3 majority over verdicts that had no lattice meaning; under §11.4 it does not reduce to Pass. §11.2 ratchets five counters, §11.3 puts one row per train (16 rows, ~40 days [A]) and lands every gate unarmed, §11.5 adds the `ontology:` report line and `lattice` to `quorum:`, §11.7 gives five falsifiers. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(spec): §11 review — ids disambiguated, paiml#3179 claim corrected, stops and report lines in one place Review of the §11 draft against ONT-001 and against this spec's own conventions: - ONT-001's R-n/F-n/§n ids collided with this spec's T-5 predicates R-1..R-5. Upstream ids are now written `ONT R-n` / `ONT F-n` / `ONT §n` throughout §11. - The draft said paiml#3179 round 1 was "a 2/3 majority over lattice-invalid verdicts". It was not: the verdicts were well-formed, the PREMISE was false (a launched kernel entry point that does not exist in the tree). Corrected to what the ontology actually does about it — `resolves: symbol` on the premise returns Unknown{…} at extraction, before any lane votes. - `contracts/lint-baseline.json` and `make ont-ratchet` do not exist in aprender yet; §11.0 and §11.2 now say so instead of naming them as if present. - A "sweep PR" is now a file predicate (night.yml, docs/specifications/**, the CLI registry, README.md, CLAUDE.md, any known-red list) so FR-1 can be a `check_pr_closes_issue.sh`-class PR-body check rather than a reading. paiml#3268 itself is one and carries `ont-delta: none`. - The four stop conditions live in §8 and the two report lines in §7, once; §11.5/§11.6 point there instead of duplicating them. `ontology:` gains `deltas <n>/<sweep PRs>` so §11.1 is measurable at T-5. - ONT §0.2's push constraint (never while a release-titled run is in progress) is named as §3's one-PR rule seen from the other repo. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(spec): final review — the amendments contradicted the original text in seven places Read start to finish. Every finding is a place where a dated amendment (§3.4 merge-queue parallelism, §3.5 SSH, §6.1 continuous triage, paiml#3205 mini, the one-subagent rule) was landed beside original text that still said the opposite, so a reader could cite either. - §1 measured the fleet with `gh api …/actions/runners` — the call the operator rejected, a `busy` snapshot that cannot see ephemeral runners, and the hourly average that read 9.5 % while 15/16 workers were busy. Now the fleet-pack ledger record, instantaneous busy/online, both traps named. - §1 said "one PR in CI at a time means gate latency IS throughput"; §3.4 was amended to 3-parallel on 2026-09-12. Bound is now 3 × 72 h / p95. - `mini` is a declared full-time build host (paiml#3205) and appeared only in a §8 bullet. Added to §0 row 0, §1, §5 P0·Pack fields and Done, §7 pack:, §8. - §0 row 3 still scheduled triage once per train; §6.1 made it per-wakeup. - §2 named the required check `ci / gate`; the rules API says `gate` and `workspace-test`, and `present` is not required. - §3.4 allowed "≤ 3 read-only subagents" against the one-at-a-time rule and §10's fan-out-through-agy. - §8's last bullet stopped on "a second concurrent aprender PR in CI" and on "SSH into a host" — both allowed by the amended §3.4/§3.5. Now stops on a host CONFIG change over SSH instead of forjar. - §9 asked for the 0.67 cascade wall to be measured; it was: 70 min, attended 0. That is 3.5× the [A] line, so by §9's own rule the cascade is the next kaizen target; where the minutes go is [U]. - §7 train: line said T-0..T-4; T-5 exists. §6's tail paragraph gets a §6.4 heading. §11.3 cited the one-PR rule §3.4 no longer has; fixed. - `make build-report` does not exist on main (P0·Instrument not done) — said so where p95 is marked [U]. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(spec): §12 — the chain of reasoning: why the loop never terminates and the four things it moves The spec had eleven sections of mechanism and no argument. §12 is the argument, step by step, each with its mechanism and its falsifier: §12.1 why it runs forever — the selector is total (row 4 always matches), a stop stops the session never the loop (every §8 line names the mechanism that prevents its recurrence), every counter is a ratchet, no number is a guess and a guess that becomes measurable is replaced (§9's 70-min cascade is the worked example), the ledger is the memory, the clock cuts the train. §12.2 the four axes — the repo, the released binaries, the CRUX competitors, the fleet — on the pv/ontology substrate. Each with its dated position, its mechanism, its ratchet and its §7 line. The spec had NO competitor axis before this: the train shipped binaries and nothing in it said where they stand. CRUX monitors 9 competitors through 275 stories (✅39 🔨80 ❌156 at v2.2 intake [C]; FALSIFY-CRUX-010 declared, not found under crates/ or scripts/ on main — [U] until landed). BEATS has 16 contracts; Ollama GPU decode is PARITY with a 0.90 floor, llama.cpp c=1 a narrow loss, fail-closed WON. Approaching = ❌→🔨→✅ by demand tier, which §6.3 already schedules; surpassing = a beat threshold that is a floor first and moves above 1.0 only on three agreeing medians on the PUBLISHED binary — which is why the post-publish dogfood (paiml#3202) precedes any ratio. Hooks: `beats:` line in §7, a §8 stop on a beat RED on the published binary or a `measured-on published` claim from a dev build, a §0 pointer. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(spec): §11 — infra main tracks ONT-001 v3.1; it is v4.3 that is untracked The previous wording said the upstream spec was untracked with "no commit, no history". Measured against origin/main in a fresh worktree: main has v3.1 (414 lines, f1269d0, infra#570). The untracked file is v4.3 (972 lines, sha256 512a16d5…), the version §11 is measured against. Substance unchanged — no other host can fetch v4.3 — detail corrected in §11.0 and §8. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(spec): §4.3 — the check is `deep`, and T-1 cannot be satisfied by the lane merely EXISTING This spec named `ci / deep` in five places. No such check can exist in this repo, so T-1's already-done test was reading for a string that would never appear. GitHub prefixes a check with the job that CALLS it. `ci.yml` calls the org-wide `sovereign-ci.yml` as a job named `ci`, which is why this repo reports `ci / lint`, `ci / test`, `ci / gate` — and why `workspace-test`, `guard-tree`, `guard-cargo` and `gate`, which are top-level jobs in a workflow file, appear bare. Measured on this PR's own check list, both halves. `ci / deep` would therefore require a `deep` job inside the ORG-WIDE reusable workflow, with blast radius across every consuming repo. paiml#3260's lane is `.github/workflows/deep.yml` with a job named `deep`, emitting `deep`. Amending this document is the cheap half of that trade; amending an org-wide workflow to match a string this document happened to write is the expensive half. §4.3 also records the sequencing hazard, which is the part that would have cost a train: $ gh workflow run deep.yml --ref PMAT-1098-ci-deep-lane HTTP 404: workflow deep.yml not found on the default branch `workflow_dispatch` is honoured only on the default branch, and a deep lane deliberately has no `pull_request` trigger. So the lane cannot be exercised AT ALL before it merges — its first execution would be the cut it gates — and §4 turns a red step into SKIPPED, so a lane born red costs the train silently instead of failing loudly. T-1 is consequently not satisfied by `deep` existing. The already-done test is a green `deep` run recorded against a sha ON MAIN, and the first such run must be a deliberate `gh workflow run deep.yml --ref main` after the lane lands and before a cut is attempted. A gate whose first run is the thing it certifies is the defect class this document exists to remove. spec_conformance.sh: exit 0. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(ont): §11.1 gets a caller — "absent is a PR-body lint failure" was itself only a review comment §11.1 states: "Every sweep PR body carries one line ... Absent is a PR-body lint failure, not a review comment." $ grep -rl ont-delta scripts/ .github/ Makefile (nothing) The rule shipped as prose in paiml#3268 and nothing read it. A rule whose enforcement is "not a review comment", enforced only by review comment, is this repo's anti-theater class one level up from the guard it now sits beside in ci.yml. WHAT A SWEEP PR IS — and only half of it may be a list. * the prose sinks §11.1 NAMES are constants here, quoted, and printed on every run so drift between spec and guard is visible instead of silent. * "a known-red list anywhere" is DERIVED: the union of a working-tree `find` and the index, the same rule check_baseline_ratchets.sh uses and for the same two reasons — a new baseline arriving unclassified is how the class survives, and a tracked-only universe is a free pass for a file present but not yet added. The derived half is what makes it non-trivial, measured on real PRs: paiml#3268 sweep via prose sink docs/specifications/... PASS (carries none+reason) paiml#3277 sweep via KNOWN-RED LIST scripts/cb200_baseline.txt FAIL -> now fixed paiml#3278 not a sweep PASS paiml#3245 not a sweep PASS paiml#3277 touches no prose sink at all. A hand-typed sink list would have passed it, and it is a true positive: that PR withdraws a wrong FAIL and adds an ONT-6 Unknown reason, which is precisely §11.1 form 3. Its body now carries `ont-delta: reason ont6-unread-window`. Case table, 15 rows, and it DISCRIMINATES: deleting the vocabulary check turns the table red (verified by mutation, not by reading). Rows cover kind-outside-the- vocabulary, none-without-a-reason, id-absent, case, leading space, and empty body. Vacuity floor: an empty changed-file list exits 2, because "not a sweep PR" is a verdict this guard could not have reached. Two defects found writing it, both kept as comments: * a RETURN trap runs after bash destroys the function's locals, so `rm -rf "$tmp"` died on an unbound variable AFTER fifteen green rows — a self-test that passed and exited 1. * bashrs SEC011: an unvalidated `rm -rf "$var"` is a delete-anything primitive. Now shape-checked before the sweep. bashrs 7.4.1: 0 errors. WORKFLOW CHANGE, stated rather than buried: this adds one step to ci.yml. §11.1 cannot exist without a caller, and guard_tree.sh runs check_*.sh BARE — which would run only the self-test, fifteen green rows judging no PR body, the exact failure the neighbouring step's comment documents. ont-delta: resolves scripts/check_pr_ont_delta.sh — §11.1 was a prose claim; this turns it into a checkable one (form 4). Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(ci): the §6 R-2 PR-body gate is dark on every PR — Actions steps are fail-fast and one red guard skipped 48 of 59 Measured on run 34875945193 (PR paiml#3268), job `guard-tree`: total steps 59 ran 10 skipped after the step-8 failure 48 step 8 failure Every cargo-free guard runs, and every failure is reported step 55 skipped A PR body must close every issue it cites (§6 R-2) step 56 skipped A sweep PR closes with an ontology delta (§11.1) GitHub Actions steps are fail-fast: one red step darkens every step after it. Why this is worse than a missed run. check_pr_closes_issue.sh exists because the 0.67.0 T-5 reconcile found 48 merged PRs since the previous tag of which only NINE closed anything. Its own wiring comment, four lines above this change, records that running it bare executes only its self-test — "Nine green rows about a regex, on every PR, judging NO PR body" — and that a real caller was the remedy. It got a real caller. The caller is masked. On any PR where one cargo-free guard is red the guard is dark exactly as it was before it was wired, and step 8 is currently red on EVERY PR (the pin/advisory deadlock, paiml#3277), so §6 R-2 has been dark fleet-wide for the duration. `!cancelled()` rather than a bare event check: a step whose `if:` contains no status function is still skipped on a prior failure. These two read `github.event.pull_request.body` and nothing else, so no guard result can be their precondition — which is what makes this the narrow, defensible half of the fix. The other 46 skipped steps are mostly CASE TABLES — the mutation-verification proving the neighbouring guards can still go red. A case table that does not run is the theater this repo keeps deleting. They are NOT swept here: some (`target-watch:` markers) plausibly do depend on ordering, and a blanket always() over 48 steps would be its own kind of wrong. paiml#3282 carries the classification. Stated rather than buried: unmasking means these steps now report on PRs that are already red for another reason, so the first sweep will surface findings that have been invisible for as long as the masking has. Same class as nextest --fail-fast hiding four dark failures across seven rounds; that lesson said "sweep the CI selection" and nothing had swept the STEP surface. ont-delta: none — a CI wiring fix; this PR's delta is already recorded against scripts/check_pr_ont_delta.sh. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
noahgift
added a commit
to guyernest/aprender
that referenced
this pull request
Sep 15, 2026
…has run (paiml#3299) main is RED at 768e740 on guard-tree, and this is why: PROMOTE x86_64-cuda-sm89 marked pending:361, but a job CARRIED it 0d ago — CUDA Nightly (GPU QLoRA falsifiers) / ada-yoga (x86_64, sm_89) .github/silicon-coverage.txt's own comment prescribes exactly this: "it promotes to `required` when the `ada-yoga` leg has actually run — not before (N-1)". It ran at 2026-09-15T06:36:05Z. The blocker is gone and the only thing left was the line. This is the guard working, not the guard being wrong. A `pending:` marker CANNOT ROT by design — the preflight fails the moment a job carrying that axis concludes, because at that point the deferral is the only thing standing between us and the coverage. After: axes declared: 5 (required 3: covered 3, STALE 0, UNCOVERED 0, MISSING 0; deferred 2: PROMOTABLE 0, ready 1) check_silicon_coverage.sh rc=0 aarch64-macos stays deferred and reads `ready`, not PROMOTE: mini-m4 can serve it but nothing has run on it yet. Promoting an axis no job has carried is the runner-existence mistake this guard was rewritten to stop making — paiml#3204 and paiml#3205 are the leg that will make it run. Refs paiml#3057, paiml#3204. no-close: paiml#3057 also covers the CUDA nightly being RED on the PP-26 witness, which is a real code defect and is untouched here. paiml#3204 is the mini macOS leg. Pmat-Ticket: PMAT-3057
This was referenced Sep 15, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Sep 16, 2026
This was referenced Sep 16, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The operator ruled 2026-09-13 that mini (Apple M4) is a full-time aprender build host. Measured state: the
mini-m4runner registration had been auto-deleted by GitHub on 2026-09-12 for inactivity, and no aprender workflow could select a macOS runner anyway (every job namesLinuxor theclean-roomdocker pool). Re-registered at 10:07Z with labelsself-hosted,macOS,ARM64,apple-silicon,m4,mini— online.What
ci.yml: newmacos-arm64job on[self-hosted, macOS, ARM64, apple-silicon, mini]—cargo check -p aprender-core --libandcargo test -p aprender-compute --lib(the NEON SIMD path no Linux runner exercises). Not ingate's needs: one 16 GB runner must never hold a merge. Timeout 40[U].nightly.yml:build-darwinrestores theaarch64-apple-darwinapr archive natively on mini (no docker on macOS);releasewaits for both lanes and requires only the Linuxbuild..github/actionlint.yaml: labelsmacOS,apple-silicon,m4,mini.Evidence
Native on mini, main @ 3786870, 2026-09-13T10:11Z:
cargo test -p aprender-compute --lib -j6rc=0;cargo check -p aprender-core --librc=0. Guards on the result: check_runner_labels, check_no_hosted_runners, check_workflow_env_defined, check_workflow_path_filters, check_hardcoded_paths, check_no_timing_in_required, check_perf_concurrency_groups all PASS; actionlint finding count unchanged from main (2 pre-existing SC2016 in the docker lanes).Sibling: paiml/infra PR flips
machines/mini/README.mdfrom "cowork box, CI is a reluctant guest" to the operator's ruling and addsminito the declared label set.Closes #3204
🤖 Generated with Claude Code