Skip to content

fix(tools): pin pmat 3.40.1 and CB-200 back to 602 — the fleet converged 2026-09-14; re-measuring under 3.40.1 read 603 (PMAT-3300) - #3301

Merged
noahgift merged 2 commits into
mainfrom
PMAT-3300-tools-pin-pmat-3.40.1
Sep 15, 2026
Merged

noahgift merged 2 commits into
mainfrom
PMAT-3300-tools-pin-pmat-3.40.1

Conversation

@noahgift

Copy link
Copy Markdown
Contributor

fix(tools): pin pmat 3.40.1 — the fleet converged on 2026-09-14 and the aprender half of the contract was never done; CB-200 back to 602

tools.toml's own header: "Bump this file in the SAME commit as the infra
forjar.yaml version bump." infra#591 (ba369f9, 2026-09-14 17:53Z) moved
intel's stack-tool-pmat to 3.40.1; this file stayed at 3.40.0. Measured on
one PR ten minutes apart: main on yoga-build (not yet converged) green,
PR 3093 on intel-clean-room-5 FAIL pmat pinned 3.40.0 found 3.40.1. The
verdict of an aprender PR was a function of which box picked it up.

CHANGING THE INSTRUMENT RE-RECORDS THE BASELINES, so both pmat-measured
baselines were re-measured under 3.40.1 rather than relabelled:

complexity_baseline.txt 679 rows under 3.40.0 -> 679 under 3.40.1
(check_complexity_ratchet.sh --update)
cb200_baseline.txt 602 -> 602, but only after the fix below

CB-200 was 603 on main. pmat comply check under 3.40.1 reads 602 at
a0634f7 (the commit that banked 602) and 603 at origin/main — same
instrument, so the tree moved. Bisected to ad8f987 (#3004). Roster diff
of sub-B definitions between the two trees is one line:

crates/aprender-compute/src/registry/wgpu_probe.rs entry B- (72.7)

It merged because no required job runs pmat comply. entry inlined
three lookup tables (vendor id, wgpu backend -> transport, device type ->
status) plus the unified-memory rule into one constructor. Each is now a
named pure function with its own unit test; entry is assembly only.
Fresh index: entry A+ 98.5, classify A 91.0, transport_name A- 89.5,
vendor_name / mem_kind A+; CB-200 Warn 602 … at the recorded baseline.

Two measurement traps, on the record because both cost a wrong reading:

  • pmat comply reads ~/.cache/paiml-mcp-agent-toolkit/comply/index//
    context.db and did NOT refresh it after the edit; pmat query "x"
    (dogfood.sh's "index first" step) refreshes .pmat/context.db, a
    different file. comply reported the OLD entry (B-, cx 19) on a tree
    that no longer contained it. Only moving the comply index aside made it
    re-measure. Filed against pmat.
  • min_grade "B" counts B- as below B. 603 reproduces from the index as
    grade in (B-,C+,C,C-,D+,D,D-,F) minus the [tdg] excludes.

Local bashrs on this workstation is 7.3.0, so check_tool_versions.sh and
the shell_lint header row FAIL here on bashrs; CI runners carry 7.4.1 and
main is green on both. Not this PR's fact.

Closes #3300.

ont-delta: none — a fleet pin bump and one complexity refactor; no entity type, shape, resolves rule or reason id.

🤖 Generated with Claude Code

…he aprender half of the contract was never done; CB-200 back to 602

tools.toml's own header: "Bump this file in the SAME commit as the infra
forjar.yaml version bump." infra#591 (ba369f9, 2026-09-14 17:53Z) moved
intel's stack-tool-pmat to 3.40.1; this file stayed at 3.40.0. Measured on
one PR ten minutes apart: main on yoga-build (not yet converged) green,
PR 3093 on intel-clean-room-5 `FAIL pmat pinned 3.40.0 found 3.40.1`. The
verdict of an aprender PR was a function of which box picked it up.

CHANGING THE INSTRUMENT RE-RECORDS THE BASELINES, so both pmat-measured
baselines were re-measured under 3.40.1 rather than relabelled:

  complexity_baseline.txt   679 rows under 3.40.0 -> 679 under 3.40.1
                            (check_complexity_ratchet.sh --update)
  cb200_baseline.txt        602 -> 602, but only after the fix below

CB-200 was 603 on main. `pmat comply check` under 3.40.1 reads 602 at
a0634f7 (the commit that banked 602) and 603 at origin/main — same
instrument, so the tree moved. Bisected to ad8f987 (#3004). Roster diff
of sub-B definitions between the two trees is one line:

  crates/aprender-compute/src/registry/wgpu_probe.rs  entry  B- (72.7)

It merged because no required job runs `pmat comply`. `entry` inlined
three lookup tables (vendor id, wgpu backend -> transport, device type ->
status) plus the unified-memory rule into one constructor. Each is now a
named pure function with its own unit test; `entry` is assembly only.
Fresh index: entry A+ 98.5, classify A 91.0, transport_name A- 89.5,
vendor_name / mem_kind A+; CB-200 `Warn 602 … at the recorded baseline`.

Two measurement traps, on the record because both cost a wrong reading:
  * `pmat comply` reads ~/.cache/paiml-mcp-agent-toolkit/comply/index/<wt>/
    context.db and did NOT refresh it after the edit; `pmat query "x"`
    (dogfood.sh's "index first" step) refreshes .pmat/context.db, a
    different file. comply reported the OLD `entry` (B-, cx 19) on a tree
    that no longer contained it. Only moving the comply index aside made it
    re-measure. Filed against pmat.
  * min_grade "B" counts B- as below B. 603 reproduces from the index as
    grade in (B-,C+,C,C-,D+,D,D-,F) minus the [tdg] excludes.

Local bashrs on this workstation is 7.3.0, so check_tool_versions.sh and
the shell_lint header row FAIL here on bashrs; CI runners carry 7.4.1 and
main is green on both. Not this PR's fact.

Closes #3300.

ont-delta: none — a fleet pin bump and one complexity refactor; no entity type, shape, resolves rule or reason id.

Pmat-Ticket: PMAT-3300
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 15, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3301 head=8d3a0bf9a6454be0b79ffbb3bce4b226a4b39fbc verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

@noahgift

Copy link
Copy Markdown
Contributor Author

Holding this rather than fixing it — it asserts a state that does not exist, and
converging the fleet twice for one release is the wrong trade.

Measured

machines/fleet-hosts/pmat-pin-check.sh, run 2026-09-15:

lambda-labs   3.39.0       3.40.1       DRIFT  declared pin is not what is installed
intel         3.39.0       3.40.1       DRIFT  declared pin is not what is installed
gx10          -            -            SHAPE  manifest parses as unpinned, pmat-pins.txt says pinned
mini          3.39.0       3.40.1       DRIFT  declared pin is not what is installed
yoga          3.39.0       3.40.1       DRIFT  declared pin is not what is installed

tool=pmat machines=5 reachable=4 | matched=0 drifted=4 shape_errors=1

So the split is three-way, not two:

layer version how measured
container image 70f2cdc9e537 3.40.0 this PR's own guard-cargo failure
hosts 3.40.1 pmat-pin-check above
infra declarations 3.39.0 pmat-pin-check above
tools.toml on main 3.40.0 git show origin/main:tools.toml

guard-cargo and guard-tree run inside the container, so the version that decides
every verdict is 3.40.0 — which is exactly what tools.toml pins on main today. aprender
CI is internally consistent right now; this PR is the only thing that is broken, and it is
broken because it asserts 3.40.1, a version the deciding layer does not carry.

That is also why the failure appears in both directions: this PR fails
recorded under pmat 3.40.1, runner has pmat 3.40.0, while its own body cites
PR 3093 on intel-clean-room-5: FAIL pmat pinned 3.40.0 found 3.40.1. No repo pin can be
right while the container and the host disagree.

Why hold rather than repair

Making this PR pass needs the container image rebuilt to 3.40.1 and the infra
declarations moved to match. pmat 3.41.0 is days away (paiml/paiml-mcp-agent-toolkit#1364
plus #1365's declared gate, then clean-room). Converging hosts and image to 3.40.1 now and
again to 3.41.0 immediately after is two fleet convergences for one release.

The pin moves directly 3.40.0 → 3.41.0 after publish, in one PR with the baseline
restamps this PR already carries — they are good work and will be reused.

Order of truth, for the record

image  ≥  hosts  ≥  declarations  ≥  repo pin

The repo pin is asserted last because it is the only one of the four with no measurement
behind it. This PR asserted it first.

Landing independently (not blocked on the release)

  • gx10's manifest shape, so pmat-pin-check can read it at all — SHAPE today means gx10's
    pmat is unverified, which is worse than drift and invisible in a pass/fail count.
  • infra declarations 3.39.0 → 3.40.1 to match measured reality.
  • pmat-pin-check wired into infra's nightly, failing on drifted>0 || unchecked>0 — the
    declarations were wrong for a day and nothing noticed, which is the gate that was missing.

No forjar apply yet: the hosts are already on 3.40.1, so there is nothing to converge
until 3.41.0 exists.

@noahgift noahgift closed this Sep 15, 2026
@noahgift

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-3300",
 "head": "1a5da828be17b9e9f1f2cc0d0ce941ecd2580e78",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "NO-VERDICT",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

@noahgift noahgift reopened this Sep 15, 2026
@noahgift

Copy link
Copy Markdown
Contributor Author

Reopening. I closed this on a false premise and the measurement says the opposite.

What I got wrong

I closed it arguing "guard-cargo runs in the container, the container is 3.40.0,
tools.toml is 3.40.0, therefore aprender CI is internally consistent and only this PR
is broken."

guard-cargo and guard-tree are runs-on: [self-hosted, Linux, clean-room]
bare-host jobs, no container. The image is never consulted. The layer that decides them is
the runner user's PATH, and that pool is split.

The evidence

Run 34944505814 — this PR's
own run — carries both readings:

pinned 3.40.0  found 3.40.1
pinned 3.40.1  found 3.40.0

found is the runner's own pmat. Both cannot be true on one machine.

And it is not theoretical: #3295 was ejected from the merge queue on
34945313381,
guard-cargoFAIL pmat pinned 3.40.0 found 3.40.1 on actions-runner-7. The split is
costing queue entries now, not after the next release.

Why "hosts uniform" and "runners disagree" are both true

machines/fleet-hosts/pmat-pin-check.sh reports every reachable host at 3.40.1:

lambda-labs 3.39.0 -> 3.40.1 DRIFT   intel 3.39.0 -> 3.40.1 DRIFT
mini        3.39.0 -> 3.40.1 DRIFT   yoga  3.39.0 -> 3.40.1 DRIFT
gx10        SHAPE  manifest parses as unpinned, pmat-pins.txt says pinned
matched=0 drifted=4 shape_errors=1

It measures the login user over SSH. Jobs run as the runner service user under
override.conf plus the repo-deployed prejob hook — a different principal with a
different PATH. So the pin check cannot see a split pool by construction, and gx10 is
not merely drifted but unchecked, in the same clean-room label pool.

Direction: up, never down

Hosts are already 3.40.1. Downgrading runners to match a stale repo pin would invert the
order of truth — image ≥ hosts ≥ declarations ≥ repo pin — and leave residue. The
restamps in this PR are needed at 3.41.0 regardless; the only duplicated cost is one
runner-PATH fix, which has to happen anyway to find the split runner.

Sequence before this merges

  1. Emit command -v pmat and pmat --version from the prejob hook into the job log —
    repo-deployed, no host access — and read the pool off the next queue run. Count runners
    per version; gx10 is the first suspect precisely because it is the one that cannot be
    checked.
  2. Infra PR: declarations 3.39.0 → 3.40.1; the gx10 manifest shape so it is readable at
    all; pmat-pin-check re-pointed at the runner principal and promoted from audit to
    gate — a prejob preflight that refuses the job when the resolved pmat differs from
    pmat-pins.txt. A split pool then cannot eject a queue entry again: it fails preflight
    on that runner and re-dispatches.
  3. forjar apply -r stack-tool-pmat per host, plus the prejob-hook deploy and clean-room
    runner restarts.
  4. This PR lands once the log shows one version across the pool.

Rebasing on main now.

@noahgift

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-3300",
 "head": "1a5da828be17b9e9f1f2cc0d0ce941ecd2580e78",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "NO-VERDICT",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

2 similar comments
@noahgift

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-3300",
 "head": "1a5da828be17b9e9f1f2cc0d0ce941ecd2580e78",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "NO-VERDICT",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

@noahgift

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-3300",
 "head": "1a5da828be17b9e9f1f2cc0d0ce941ecd2580e78",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "NO-VERDICT",
   "findings": 0
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

@noahgift

Copy link
Copy Markdown
Contributor Author

quorum-review (AD-04): NOT agreed (auto_merge: checked=true was_armed=false disarmed=false)

{
 "ticket": "PMAT-3300",
 "head": "1a5da828be17b9e9f1f2cc0d0ce941ecd2580e78",
 "width": 3,
 "executor": "agy",
 "agreed": false,
 "auto_merge": {
  "checked": true,
  "was_armed": false,
  "disarmed": false,
  "note": "auto-merge not armed"
 },
 "lanes": [
  {
   "lane": 1,
   "verdict": "PASS",
   "findings": 0
  },
  {
   "lane": 2,
   "verdict": "FAIL",
   "findings": 2
  },
  {
   "lane": 3,
   "verdict": "PASS",
   "findings": 0
  }
 ]
}

@noahgift noahgift changed the title fix(tools): pin pmat 3.40.1 — the fleet converged 2026-09-14 and every intel-hosted PR is red; CB-200 back to 602 (PMAT-3300) fix(tools): pin pmat 3.40.1 and CB-200 back to 602 — the fleet converged 2026-09-14; re-measuring under 3.40.1 read 603 (PMAT-3300) Sep 15, 2026
@noahgift noahgift added this to the 0.68.0 milestone Sep 15, 2026
@noahgift
noahgift added this pull request to the merge queue Sep 15, 2026
Merged via the queue into main with commit 4bad830 Sep 15, 2026
25 of 37 checks passed
@noahgift
noahgift deleted the PMAT-3300-tools-pin-pmat-3.40.1 branch September 15, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant