Skip to content

Commit 7f94d8b

Browse files
noahgiftclaude
andcommitted
docs(spec): final review — the amendments contradicted the original text in seven places
Read start to finish. Every finding is a place where a dated amendment (§3.4 merge-queue parallelism, §3.5 SSH, §6.1 continuous triage, #3205 mini, the one-subagent rule) was landed beside original text that still said the opposite, so a reader could cite either. - §1 measured the fleet with `gh api …/actions/runners` — the call the operator rejected, a `busy` snapshot that cannot see ephemeral runners, and the hourly average that read 9.5 % while 15/16 workers were busy. Now the fleet-pack ledger record, instantaneous busy/online, both traps named. - §1 said "one PR in CI at a time means gate latency IS throughput"; §3.4 was amended to 3-parallel on 2026-09-12. Bound is now 3 × 72 h / p95. - `mini` is a declared full-time build host (#3205) and appeared only in a §8 bullet. Added to §0 row 0, §1, §5 P0·Pack fields and Done, §7 pack:, §8. - §0 row 3 still scheduled triage once per train; §6.1 made it per-wakeup. - §2 named the required check `ci / gate`; the rules API says `gate` and `workspace-test`, and `present` is not required. - §3.4 allowed "≤ 3 read-only subagents" against the one-at-a-time rule and §10's fan-out-through-agy. - §8's last bullet stopped on "a second concurrent aprender PR in CI" and on "SSH into a host" — both allowed by the amended §3.4/§3.5. Now stops on a host CONFIG change over SSH instead of forjar. - §9 asked for the 0.67 cascade wall to be measured; it was: 70 min, attended 0. That is 3.5× the [A] line, so by §9's own rule the cascade is the next kaizen target; where the minutes go is [U]. - §7 train: line said T-0..T-4; T-5 exists. §6's tail paragraph gets a §6.4 heading. §11.3 cited the one-PR rule §3.4 no longer has; fixed. - `make build-report` does not exist on main (P0·Instrument not done) — said so where p95 is marked [U]. Pmat-Ticket: PMAT-1098 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 6b14708 commit 7f94d8b

1 file changed

Lines changed: 37 additions & 25 deletions

File tree

docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md

Lines changed: 37 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,10 @@ Implement docs/specifications/APR-RELEASE-001-train-and-build-kaizen.md autonomo
1616

1717
| # | If | Do |
1818
|---|---|---|
19-
| **0** | `yoga` or `gx10` is under-utilised (§1 packing rule) while intel has queue pressure | **P0, minutes not a session:** arm every green PR, route what can leave intel, reap disk (§5 P0·Pack, P0·Reap); record the `pack:` line; then continue to the first matching row below |
19+
| **0** | `yoga`, `gx10` or `mini` is under-utilised (§1 packing rule) while intel has queue pressure — and, every wakeup regardless, anything arrived since the last sample is untriaged (§6.1) | **P0, minutes not a session:** arm every green PR, route what can leave intel, reap disk (§5 P0·Pack, P0·Reap); triage what arrived (§6.1); record the `pack:` and `triage:` lines; then continue to the first matching row below |
2020
| 1 | ≥ 48 h since the last tag on `main` **and** no SKIPPED record for the current HEAD | run the train (§4) |
2121
| 2 | else a §5 row whose *Done* test fails at HEAD | do the first such row, one PR |
22-
| 3 | else the last train (shipped or skipped) has no triage record | do the triage pass (§6) |
22+
| 3 | else the last train (shipped or skipped) has no once-per-train triage record | do the once-per-train pass: §6.3 capacity check and the T-5 reconcile receipt |
2323
| 4 | else | emit the §7 report, exit 0 |
2424

2525
Nothing in this spec asks a question. Running it ten times a day is safe.
@@ -28,22 +28,28 @@ Nothing in this spec asks a question. Running it ten times a day is safe.
2828

2929
Ship a tag every 48–72 h (`0.67 → 0.68 → …`) **on a clock, not on scope**, and keep
3030
shrinking the wall-clock from *PR opened* to *tag published* so the clock stays cheap.
31-
The objective is elapsed time and green trains. **Packing rule (operator, 2026-09-12, verbatim):** "these two boxes: yoga and gx10 should be always 80% full of PRs from aprender if ANY queue pressure on intel … not acceptable to have slow releases when boxes are idel". Utilisation is not the goal for its own sake; an idle GPU box next to an intel queue is lost release time and is a **P0 defect**, not a state to tolerate. Measure it every wakeup:
31+
The objective is elapsed time and green trains. **Packing rule (operator, 2026-09-12, verbatim):** "these two boxes: yoga and gx10 should be always 80% full of PRs from aprender if ANY queue pressure on intel … not acceptable to have slow releases when boxes are idel". Utilisation is not the goal for its own sake; an idle GPU box next to an intel queue is lost release time and is a **P0 defect**, not a state to tolerate. `mini` (Apple M4) was declared a full-time aprender build host on 2026-09-13 (#3205) and is under the same rule; its ceiling is the macOS-capable job classes, not capacity. Measure it every wakeup, **from the ledger, never from a runner-list snapshot**:
3232

3333
```
34-
gh api --paginate orgs/paiml/actions/runners --jq '.runners[] | "\(.name) \(.status) \(.busy)"' # busy/online per host prefix
35-
intel pressure = any aprender job queued, or a workspace-test running, on intel
36-
under-utilised = intel pressure AND (busy/online < 0.8 on yoga OR on gx10)
34+
record = docs/build-ledger/<date>/<sha>-<host>-fleet-pack-*.json # written by §5 P0·Pack
35+
intel pressure = aprender_runs_queued > 0, or a workspace-test running, on intel
36+
under-utilised = intel pressure AND (busy/online < 0.8 on yoga, on gx10, OR on mini)
3737
```
3838

39+
Two measurement traps, both already paid for: a `busy` snapshot from the runners API cannot see
40+
ephemeral runners, and an hourly **average** hides saturation — `occ_1h` read 9.5 % on 2026-09-14
41+
while 15 of 16 intel workers were busy at load 162. The `pack:` line carries the instantaneous
42+
`busy/online` at sample time; an average is a trend, not a verdict.
43+
3944
Coupling, one line:
4045

4146
```
42-
max PRs per train ≈ 72 h / p95 `ci / gate` wall-clock (upper bound)
47+
max PRs per train ≈ 3 × 72 h / p95 `ci / gate` wall-clock (upper bound; 3 = merge-queue parallelism, §3.4)
4348
```
4449

45-
One PR in CI at a time means gate latency *is* release throughput. Compute this on every
46-
train. p95 is `[U]` until §5 P0 lands.
50+
The merge queue builds 3 entries in parallel (§3.4); gate latency still bounds release throughput.
51+
Compute this on every train. p95 is `[U]` until §5 P0 lands — `make build-report` does not exist on
52+
`main` as of 2026-09-14.
4753

4854
## §2 Ground truth — verify at HEAD before writing anything
4955

@@ -52,7 +58,7 @@ train. p95 is `[U]` until §5 P0 lands.
5258
| `intel` — clean-room runner, 8 concurrent, memory-bound, 3.6 TB NVMe | `infra/machines/intel/forjar.yaml` | `[V]` snapshot |
5359
| `yoga` — CI runner, RTX 4060 8 GB, 32 GB RAM, 10G NIC needs `bolt.service` | `infra/machines/yoga/forjar.yaml` | `[V]` snapshot |
5460
| `gx10` — aarch64 GB10, sm_121, 120 GB unified; **not** a documented general runner | `infra/machines/gx10/forjar.yaml` | `[V]` snapshot |
55-
| `main` protected; required check literally named `ci / gate` | org ruleset | `[V]` |
61+
| `main` protected; required checks are exactly `gate` and `workspace-test``present` is NOT required, it is the review-receipt backlog | `gh api repos/paiml/aprender/rules/branches/main` | `[V]` 2026-09-14 |
5662
| Last tag = `git describe --tags --abbrev=0` on `main`; next minor = that + 1 | git | `[V]` live |
5763
| p95 `ci / gate`, tag→publish, cascade wall time (automated) || `[U]` unmeasured |
5864
| `workspace-test` is pinned `runs-on: [self-hosted, X64, Linux, clean-room]` (#3104) — the long pole never lands on gx10; #3139 lifts the pin (795 s on gx10-pool3, 34693750990) | `.github/workflows/ci.yml` | `[V]` 2026-09-12 |
@@ -74,7 +80,8 @@ Live `forjar.yaml` beats this table. Record the diff in the receipt and continue
7480
(roadmap-additive guard, a red required check) and fix or trim it rather than park it. Under
7581
intel pressure, arm every green PR and prefer the ones whose jobs can land on `yoga`/`gx10`.
7682
Splitting *one* gate across hosts (§5 P2) is the other half of the same rule. "Parallel"
77-
inside a session still means ≤ 3 read-only subagents, never two sessions merging.
83+
inside a session means ONE Claude subagent at a time and fan-out through agy (§10) — never two
84+
sessions merging.
7885
*(Amended 2026-09-12 by operator ruling; the previous text said "one aprender PR in CI at a
7986
time" and this session dequeued four PRs on it — #3175's group was running its workspace-test
8087
on yoga at the time.)*
@@ -168,9 +175,10 @@ busy/online per host (§1), decide `intel pressure`, and act in minutes — arm
168175
withdrawn CI, trim non-additive roadmap diffs, route arch-neutral work off intel, dispatch the
169176
nightlies that produce T-1 evidence on the idle GPU boxes. Record the sample as a ledger record
170177
`docs/build-ledger/<date>/<sha>-fleet-pack.json` `{at, intel_busy, intel_online, gx10_busy,
171-
gx10_online, yoga_busy, yoga_online, intel_pressure, verdict}`.
178+
gx10_online, yoga_busy, yoga_online, mini_busy, mini_online, intel_pressure, aprender_runs_queued,
179+
aprender_runs_live, idle_gpu_runners, verdict}` (5 records on `main` as of 2026-09-14, none yet with the `mini` fields).
172180
*Done:* over the trailing 10 trains, in every sample with `intel_pressure=true`, median busy/online
173-
≥ 0.8 on both `yoga` and `gx10`; the §7 `pack:` line is never `P0-UNDERUTILIZED` two wakeups in a row.
181+
≥ 0.8 on `yoga`, `gx10` and `mini`; the §7 `pack:` line is never `P0-UNDERUTILIZED` two wakeups in a row.
174182
The structural lever is `workspace-test` leaving its `X64` pin (#3139: 82,085 tests in 795 s on gx10
175183
against 1,000–6,000 s on intel); until it lands, `pack` is bounded by the short jobs.
176184

@@ -335,6 +343,7 @@ and is committed to the ledger; the autopilot writes `DONE` only after `check_re
335343
every predicate 0. Falsifier: seed one fixed-but-open issue (a closed test issue reopened) — the check
336344
must go RED.
337345

346+
### §6.4 Lifecycle — stale, the closure ratchet, and what is recorded
338347

339348
Queues stabilise when closure ≥ arrival; age falls when WIP is capped. That is the whole
340349
mechanism.
@@ -361,10 +370,10 @@ mechanism.
361370

362371
```
363372
APR-RELEASE-001 | did=<TRAIN|BUILD|TRIAGE|NOOP> | train=v0.<N>.0 | verdict=<SHIPPED|SKIPPED|MERGED|NOOP>
364-
train: step reached <T-0..T-4> | skip reason <none|…> | cascade wall min <n|[U]> | attended min 0
373+
train: step reached <T-0..T-5> | skip reason <none|…> | cascade wall min <n|[U]> | attended min 0
365374
build: row <P0..P3|none> | PR <url|none> | records added <n>
366375
gate: p95 ci/gate <min|[U]> | max PRs/train <n|[U]> | queue p95 intel <s> yoga <s> gx10 <s>
367-
pack: intel <busy>/<online> | gx10 <busy>/<online> | yoga <busy>/<online> | intel-pressure <yes|no> | verdict <OK|P0-UNDERUTILIZED>
376+
pack: intel <busy>/<online> | gx10 <busy>/<online> | yoga <busy>/<online> | mini <busy>/<online> | intel-pressure <yes|no> | verdict <OK|P0-UNDERUTILIZED>
368377
triage: arrival <n> | closure <n> | open PRs <n> (age p95 <d>) | untriaged issues <n> prs <n>
369378
branches: total <n> | no-PR <n> | no-PR >7d <n> | archived this pass <n>
370379
capacity: milestone <M> open <n> | closure/day <x|[U]> | capacity <n> | verdict <FITS|OVERCOMMITTED by <n>d> | spilled <n>
@@ -385,7 +394,7 @@ next: train eligible at <timestamp>
385394
- Measured max PRs/train < 10 `[A]` → stop cutting trains; finish P1/P2 first.
386395
- `yoga` or `gx10` has no runner unit in live `forjar.yaml` and the infra PR is unmerged → stop at P2.
387396
- Fewer than 20 ledger records → stop at P0.
388-
- `yoga` or `gx10` under-utilised (§1) while intel has queue pressure, two wakeups in a row → P0:
397+
- `yoga`, `gx10` or `mini` under-utilised (§1) while intel has queue pressure, two wakeups in a row → P0:
389398
stop the current build row, pack first (arm, route, reap), report the `pack:` line.
390399
- A runner box at or below `REAPER_CRITICAL_GB` free, or any job dead on ENOSPC → P0: reclaim
391400
over SSH now, then land the forjar change that makes it automatic, before anything else.
@@ -399,9 +408,8 @@ next: train eligible at <timestamp>
399408
question to the operator.
400409
- The `P0`-labelled set of the next milestone alone exceeds its capacity (§6.3) → stop: the release
401410
is over-promised at the one priority level the operator controls, and only the operator can cut it.
402-
- Any step would need an invented threshold, a second concurrent aprender PR in CI,
403-
`--allow-dirty`, or SSH into a host → stop.
404-
411+
- Any step would need an invented threshold, `--allow-dirty`, or a host CONFIG change made over SSH
412+
instead of through forjar (§3.5 — SSH for measurement and for unclogging is expected) → stop.
405413
- The upstream ontology spec is not reachable at a committed sha (today: untracked in `infra`,
406414
sha256 `512a16d5e09c…`) → stop: no lane on another host can read the premise, and every ontology
407415
verdict is unverifiable by construction. Fixed in `infra` (ONT-P), not here (§11).
@@ -416,9 +424,12 @@ next: train eligible at <timestamp>
416424
K̂ = 14 sessions (10 trains to `0.76` + P0, P1, infra, P2) · K = 18 · andon at 16 sessions
417425
or 2 consecutive skips `[A]`.
418426

419-
The 82-crate cascade is automated (ruling 2026-09-13) and its wall time is unmeasured.
420-
Record cascade wall minutes at T-4 on the `0.67` train. Above ~20 min `[A]` it delays the
421-
post-publish receipts and the cascade — not the build — becomes the next kaizen target.
427+
The 82-crate cascade is automated (ruling 2026-09-13). **Measured on the `0.67` train**
428+
(`docs/build-ledger/2026-09-13/e45eaab47-lambda-vector-train.json`): `t4_wall_minutes: 70`,
429+
`attended_minutes: 0`. That is 3.5× the ~20 min `[A]` line this paragraph drew before any
430+
measurement existed, so by its own rule the cascade — not the build — is the next kaizen target.
431+
Where the 70 minutes go is `[U]`: record per-crate wall on the `0.68` train before naming a fix.
432+
The ratchet takes §5 P3's shape (≤ 0.70 × baseline, `[A]`) once three trains have measured it.
422433

423434
## §10 Decision procedure — for the forks §6 refuses to make
424435

@@ -467,7 +478,8 @@ Upstream: `ONT-001 v4.3` (`infra/docs/specifications/paiml-ontology.md`, sha256
467478
Its §6 assigns **aprender** every row but three. This section is how those rows get worked
468479
continuously instead of in one heroic push, and what the train owes the ontology every time it
469480
sweeps a surface. Upstream ids are written `ONT R-n` / `ONT F-n` / `ONT §n` here; a bare `§n` is
470-
this spec, and `R-1``R-5` without the prefix are the T-5 predicates (§6.3).
481+
this spec; a bare `R-1``R-5` is a T-5 predicate (§6.3) and an unhyphenated `R1–R6` is the publish
482+
preflight (§3.7).
471483

472484
### §11.0 Measured baseline — `main` @ `fa6e35f23`, 2026-09-14
473485

@@ -544,8 +556,8 @@ is RED. No bulk rewrite: ≤5 corpus files per PR except a named ratchet touch (
544556
silently reported as 0.
545557
- **One ONT row per train.** 16 rows outstanding; trains run every 2–3 days ⇒ ≈40 days to ONT-10
546558
`[A]`. That is a derived horizon, not a promise — the rows spill by §6.3 like any other work.
547-
- **An ONT row PR obeys ONT §0.2**: never pushed while a release-titled run is in progress. It is
548-
§3's one-aprender-PR-in-CI rule seen from the other repo.
559+
- **An ONT row PR obeys ONT §0.2**: never pushed while a release-titled run is in progress, so a
560+
moving branch never races T-0..T-3 of a train.
549561
- **New gates arrive unarmed** (ONT R-8, ONT §3.9). An ONT gate lands computing everywhere; arming
550562
is a later, separate PR whose body shows the counter it moved.
551563
- **T-5 carries the `ontology:` line but does not gate on it** until

0 commit comments

Comments
 (0)