plugins/ml: run 3 results - held-out promotion works; tangent relabel is the robust finding

Tried: run 3 on DeepInfra/TypeSafe/MothQuantum only (5 LLM advisors, 3-VLM
ensemble, held-out promotion, quantum-drawn held-out and test splits).
Found (6 untouched test feeds, 3-critic mean): mirror 0.176, syzygy as
shipped 0.217, tangent 0.336, motion bg=edges 0.332 with motion recall 0.27,
held-out-promoted winner 0.340 vs search-set-promoted 0.300. The ensemble
cut the search/held-out gap from ~0.2 to ~0.05. QPU draw still queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014B7KcHL9b1jwug3L7dmsV3

Author: claude

docs/marks/ledger.csv

@@ -23,3 +23,4 @@ mP02,2026-09-29,production,HEWN,"I1,I2,I3","all","sh tools/suite-total.sh -> 7 s
 mL01,2026-09-30,plugins/ml,HEWN,none (beside the kernel),"0005,docs/poc","node plugins/ml/selftest.mjs: 43 checks 0 failures (offline); registry + typed-params gate, projectors mirror/syzygy/motion keep the contract on 4 feeds, syzygy projector's port hashes to golden 0x6dbdd1a8, motion finds the moving object's direction 4/4 and equals mirror on a still feed, compare.mjs scores known cases exactly","a real camera clip; more feeds with repeated critic samples",n/a
 mL02,2026-09-30,plugins/ml,SHAPED,none,"mL01","node plugins/ml/playtest.mjs (Qwen3-VL-30B critic, image mode; replays from cache): ceiling 0.919; search set (4 feeds) mirror 0.322, syzygy 0.282, syzygy tangent 0.551, motion bg=edges 0.549 motion recall 1.00; held out (6 feeds) mirror 0.405, syzygy 0.245, tangent 0.494, motion bg=edges 0.512; search best 0.672 -> 0.462 held out","n>=3 critic samples per cell with intervals; CLIP-embedding JEPA target; qpu quantum draws",n/a
 mL03,2026-09-30,plugins/ml,SHAPED,none,"mL02","playtest --stage search --tag run2-edges (warm start motion bg=edges): search-set U 0.524 -> 0.615 (vlm 0.656; 112-col variant 0.750) but held out none beat the warm start (0.512): #9 0.480, #18 0.497, #19 0.461; Kimi advisor 429 exceeded_current_quota_error on retry","promotion on a held-out split with >=3 critic samples per cell",n/a
+mL04,2026-09-30,plugins/ml,SHAPED,none,"mL03","playtest --stage run3 (DeepInfra advisors Kimi-K2.6/Qwen3-235B/gpt-oss-120b/MiniMax-M3 + DeepSeek, 3-VLM ensemble, held-out promotion, MothQuantum emu splits): TEST feeds mirror 0.176, syzygy gradient 0.217, tangent 0.336, motion bg=edges 0.332 (motion recall 0.27), held-out winner 0.340 vs search-set winner 0.300; jevcheck Spearman(jev,vlm) 0.21; selftest 45/45","more feeds per split; credit assignment across advisor cells; a completed qpu draw",n/a

docs/ml-projection-landscape.md

@@ -257,7 +257,73 @@ This is the loop working as intended, and it is also why a held-out set must be
 
 **No run-2 config beat its own warm start on unseen feeds.** Every gain on the search set was the critic's noise on four feeds, and the search dutifully climbed it. The conclusion for the next iteration is structural, not a parameter: promotion must use a score the search cannot see, meaning a held-out split plus several critic samples per cell. Until then, the search's job is to *propose*, and the held-out stage decides.
 
-### 4.7 The typed gate
+### 4.7 Run 3: the fixes, built on DeepInfra, TypeSafe and MothQuantum only
+
+Runs 1 and 2 said the loop needed held-out promotion, less critic noise, and advisors that weren't down (the Moonshot endpoint returned `exceeded_current_quota_error`). Run 3 (`node plugins/ml/playtest.mjs --stage run3`) makes each change using only three providers.
+
+**Advisors, all hosted on DeepInfra** [built]: Kimi-K2.6, Qwen3-235B-A22B, gpt-oss-120b and MiniMax-M3, beside DeepSeek. Getting them to answer was three measured fixes:
+
+| advisor | first attempt | fix |
+|---|---|---|
+| Kimi-K2.6 | 3 × 120 s timeouts: it reasons at length by default | `reasoning_effort: "none"` → 21 s, both proposals valid |
+| MiniMax-M3 | 4000 tokens and 172 s of reasoning, **empty** content | `reasoning_effort: "none"` (probe: 10 tokens) |
+| Qwen3-235B | valid proposals inside broken JSON (one stray `}`) | `extractProposals` salvages each well-formed proposal object |
+| gpt-oss-120b | worked first time (30 s) | — |
+
+**A three-VLM critic ensemble, all on DeepInfra** [built]. Each candidate critic was first shown the real frames and scored against ground truth:
+
+| critic | ceiling | used |
+|---|---|---|
+| Qwen3-VL-30B-A3B | 0.919 | yes |
+| Gemma-3-27B | 0.903 | yes |
+| Mistral-Small-3.2-24B | 0.901 | yes |
+| Llama-4-Scout-17B | 0.656 | no |
+
+Each critic reads the real frames and the text independently, and the three fidelities are averaged (`ctx.critic_models`).
+
+**Quantum-drawn splits** [built, emu]. Two MothQuantum `comet-qrng-v1` draws (job ids and pre-outcome commitment hashes are in `logs/playtest-run3.json`) pick six **held-out** feeds, which steer promotion, and six **test** feeds, which are touched once at the end. The point is that no advisor, seed or person knows in advance which feeds will judge the search. The draws are emu (simulator). A real-QPU draw was attempted: the IBM queue hadn't returned after 13 minutes and the job was still `processing` 25 minutes later (job `37666e50`, still `processing` about 50 minutes after submission when this was written).
+
+**Promotion by held-out score** [built]. After each generation the top 3 new configs by search-set U are re-scored on the held-out feeds (`row.hold`). Parents and the final winner are ranked by `hold`, and the advisors are told in their prompt that `hold` alone decides promotion.
+
+**TypeSafe Jev as a system-one critic** [built, measured weak]. Before run 3, `node plugins/ml/jevcheck.mjs` asked Jev for a 0–4 legibility score on the text of every config the first two searches had scored: 43 configs × 4 feeds, with the VLM fidelities re-read from cache. Jev takes about 150 ms per call.
+
+```
+pair level   (172)   Spearman(jev, vlm) 0.146    Spearman(inverse, vlm) 0.077
+config level  (43)   Spearman(jev, vlm) 0.207    Spearman(inverse, vlm) -0.158    Pearson(jev, vlm) 0.297
+as a filter: send only Jev's top half (22) to the VLM -> 4 of the VLM's top 5 survive
+```
+
+Jev reading the text is a weak but *positive* signal, which is better than the free tonal scorer. It would halve the VLM calls at the cost of one top-5 config in five. At about $0.0002 per VLM call that trade isn't worth it here, so Jev is logged, not used as a gate. It would earn its place where the critic is expensive (a larger VLM, or a human).
+
+**Results.** Warm-started from motion `bg: edges` and syzygy tangent; 3 generations; one proposal per cell per generation; 556 API calls, about $0.08 on DeepInfra. Log: `plugins/ml/logs/playtest-run3.txt`.
+
+```
+   hold #3  0.364  (search-set vlm 0.370)    motion bg=edges, warm start
+   hold #5  0.353  (search-set vlm 0.426)    Kimi-K2.6's g1 proposal: best search-set score, not promoted
+   hold #16 0.395  (search-set vlm 0.417)    Qwen3-235B's g3 proposal: motion, edges, 72 cols, head marks, unicode arrows
+   hold #17 0.325  (search-set vlm 0.452)    gpt-oss-120b's g3 proposal: highest search-set score of the run
+```
+
+Then all finalists on the six **test** feeds, which nothing in the run had seen:
+
+| config | test (3-critic mean) | per critic Qwen / Gemma / Mistral | motion recall | chars |
+|---|---|---|---|---|
+| mirror | 0.176 | 0.197 / 0.142 / 0.188 | 0.10 | 1216 |
+| syzygy as shipped (gradient) | 0.217 | 0.260 / 0.197 / 0.194 | 0.13 | 1216 |
+| syzygy `orient: tangent` | 0.336 | 0.400 / 0.237 / 0.371 | 0.10 | 1216 |
+| motion `bg: edges` (runs 1–2 best) | 0.332 | 0.387 / 0.235 / 0.374 | **0.27** | 1216 |
+| **run-3 winner by held-out** (#16, Qwen3-235B) | **0.340** | 0.453 / 0.218 / 0.351 | 0.10 | 1584 |
+| run-3 winner by search-set U (#5, Kimi-K2.6) | 0.300 | 0.375 / 0.220 / 0.304 | 0.07 | 1216 |
+
+What run 3 establishes:
+
+- **Held-out promotion picks better configs.** On data neither ranking had seen, the config promoted by held-out score beat the one the search set would have promoted (0.340 vs 0.300). With one critic the search/held-out gap was about 0.2; with three critics it is about 0.05 for the seeds (e.g. 0.370 → 0.364). The ensemble takes out most of the noise the first two searches were climbing.
+- **The tangent relabel is the robust finding.** Across three critics and three independent feed sets, relabelling the kernel's `─`/`│` scores about 1.5× the kernel as shipped (0.217 → 0.336 here) and about 1.9× the mirror. Every critic agrees on the direction.
+- **Among tangent-edge projections, the search can't separate the top three.** The held-out winner (0.340), plain tangent (0.336) and motion-on-edges (0.332) are within 0.01. The one clear difference is motion: only `bg: edges` with arrows gets motion recall above noise (0.27, vs 0.07–0.13 for the rest).
+- **The critics disagree on level, not direction.** Gemma scores every config lowest and Qwen scores the search's own winner highest, which is a trace of optimizing against a Qwen-heavy history. All three rank tangent above gradient above mirror.
+- **Every advisor contributed, none dominated.** Best held-out score per cell: Qwen3-235B 0.395, local 0.364, quantum 0.363, DeepSeek 0.361, Kimi 0.353, gpt-oss 0.339. MiniMax's one scored proposal wasn't in its generation's top 3, so it never reached held-out; its other two proposals were duplicates. gpt-oss produced the highest search-set score of the run (0.452) and one of the lowest held-out scores (0.325), the clearest single case of the overfitting that held-out promotion is there to catch.
+
+### 4.8 The typed gate
 
 [built] Every proposal passes `registry.validateParams` first. It is deterministic and free, and it names every fault (unknown key, wrong type, out of range, not in enum). Only a legal config reaches TypeSafe's Jev (`/v1/systemone`, a `noul` question: *"is this config legal and not self-defeating?"*). Jev was measured against 8 hand-labelled configs (`node plugins/ml/gatecheck.mjs`):
 
@@ -362,11 +428,13 @@ plugins/ml selftest: 43 checks, 0 failures
 | frame-diff / tracked-blob motion glyphs | **[built]** `projectors/motion.mjs` | median background, blob tracking, arrows, trails, legend; add real optical flow (block matching per cell) for non-rigid motion |
 | VLM reconstruction as GAN-check | **[built, thin]** `scorers/vlm.mjs` | text + image critic modes, two references; next: 3 samples per cell with intervals, more feeds, a real camera clip |
 | VLM wireframe / 3-D sketch as auxiliary target | **[built, 2-D only]** | boxes are the 2-D wireframe; a depth-ordered layer list or a VLM-drawn SVG is **[proposed]** |
-| LLM advisor proposing params | **[built]** `search.mjs` (DeepSeek, Kimi, ZAI) | the advisors found the braille jump; mutation cells refined it (§4.6). Next: give the advisor the held-out score too, so it is rewarded for what generalizes |
+| LLM advisor proposing params | **[built]** `search.mjs`: DeepSeek, ZAI, and on DeepInfra Kimi-K2.6, Qwen3-235B, gpt-oss-120b, MiniMax-M3 (§4.7) | the advisors found the braille jump; mutation cells refined it (§4.6). Next: give the advisor the held-out score too, so it is rewarded for what generalizes |
 | typed system-one gate (TypeSafe Jev) | **[built]**, measured 7/8 | advisory; next: ask it semantic questions the schema can't |
+| TypeSafe Jev as a cheap critic | **[built, weak]** `scorers/jev.mjs`, `jevcheck.mjs` | Spearman 0.21 with the VLM (§4.7); worth it only when the paid critic is expensive |
+| multi-critic ensemble | **[built]** `ctx.critic_models` | 3 VLMs on DeepInfra; cut the search/held-out gap from ~0.2 to ~0.05 |
 | JEPA-style next-frame predictor | **[built, thin]** linear probe, 16×8 \|Δ\| target | next: frozen image encoder target (CLIP via DeepInfra `clip-ViT-B-32`), a small MLP |
-| quantum draws for exploration | **[built, emu]** `apis.quantumBytes` | a `qpu` run; compare with the xorshift control over many seeds |
-| evolving advisor cells, backtest, promote | **[built, thin]** | 3 + 2 generations here; next: make the held-out stage part of promotion, and assign credit (more proposals to cells whose winners hold out) |
+| quantum draws for exploration | **[built, emu]** `apis.quantumBytes` | mutation cell and the held-out/test split draws; a `qpu` draw was submitted and was still queued ~50 min later |
+| evolving advisor cells, backtest, promote | **[built]** | runs 1–3; run 3 promotes on held-out and checks the result on a quantum-drawn test set. Next: credit assignment (more proposals to cells whose winners hold out), more feeds per split |
 | chiaroscuro shape-match election (4×6 Hamming) | **[proposed]** | a projector plugin; the contract needs nothing new |
 | half-block colour projection | **[proposed]** | colour is out of the ASCII brief; `▀` with ANSI colour doubles vertical resolution |
 | critic-side learning (a reader tuned to the projector) | **[proposed]** | a projector/reader pair is a code; the gap to 0.92 says the reader matters as much as the writer |

plugins/ml/MARK.md

@@ -13,10 +13,11 @@
   quantum cell): the code runs and its real outputs are cached and logged, but a
   model's answer is evidence, not a proof, and n = 4 scenes is small.
 - **RUNS** —
-  `node plugins/ml/selftest.mjs` → `plugins/ml selftest: 40 checks, 0 failures`;
+  `node plugins/ml/selftest.mjs` → `plugins/ml selftest: 45 checks, 0 failures`;
   `node plugins/ml/playtest.mjs --stage all` → ceiling, baseline grid, search
   (real API calls; replays from `cache/` for free; `SYZ_ML_OFFLINE=1` forbids
-  the network).
+  the network); `--stage run3` = DeepInfra advisors + 3-VLM ensemble +
+  held-out promotion + quantum-drawn held-out/test splits.
 - **WHERE IT ATTACHES** — the kernel's JS port `docs/poc/syzygy.js` (byte-exact
   with the C kernel, V02). The `syzygy` projector loads it unmodified and the
   selftest re-derives the golden `0x6dbdd1a8` from the very module it loads. No

plugins/ml/cache/deepinfra/000eaacb2c503d4b55c7a580.json

@@ -0,0 +1 @@
+{"text":"{\"setting\": \"outdoor-day\", \"scene\": \"a landscape with a road, trees, and a vehicle\", \"objects\": [{\"label\": \"tree\", \"category\": \"plant\", \"region\": \"center\", \"box\": [0.38, 0.3, 0.5, 0.7], \"moving\": \"none\"}, {\"label\": \"tree\", \"category\": \"plant\", \"region\": \"top-right\", \"box\": [0.75, 0.2, 0.95, 0.6], \"moving\": \"none\"}, {\"label\": \"car\", \"category\": \"vehicle\", \"region\": \"bottom-left\", \"box\": [0.05, 0.65, 0.25, 0.8], \"moving\": \"right\"}, {\"label\": \"car\", \"category\": \"vehicle\", \"region\": \"top-left\", \"box\": [0.1, 0.35, 0.25, 0.5], \"moving\": \"right\"}]}","usage":{"prompt_tokens":501,"total_tokens":729,"completion_tokens":228,"estimated_cost":0.00021195000000000003,"prompt_tokens_details":null},"model":"Qwen/Qwen3-VL-30B-A3B-Instruct","request":[{"role":"user","content":[{"type":"text","text":"The image shows a character-art rendering of ONE view from a fixed camera. Each character stands for a small patch of the image: denser characters are brighter or darker patches depending on the renderer, lines may mark edges, and some renderings mark motion with arrow characters (> < ^ v or → ← ↑ ↓) and '~' for where something just was. It may contain several frames of the same view separated by blank lines, oldest first.\nReconstruct the scene in the latest frame: what objects are there, where, and which way each is moving (\"none\" if you cannot tell or it is still).\nReturn ONLY a JSON object, no prose:\n{\"setting\": \"outdoor-day\" | \"outdoor-night\" | \"indoor\" | \"unknown\",\n \"scene\": \"<one short sentence>\",\n \"objects\": [{\"label\": \"<noun>\", \"category\": \"person\" | \"animal\" | \"vehicle\" | \"building\" | \"plant\" | \"celestial\" | \"furniture\" | \"shape\" | \"weather\" | \"other\",\n   \"region\": \"top-left\" | \"top\" | \"top-right\" | \"left\" | \"center\" | \"right\" | \"bottom-left\" | \"bottom\" | \"bottom-right\",\n   \"box\": [x0, y0, x1, y1],\n   \"moving\": \"left\" | \"right\" | \"up\" | \"down\" | \"none\"}]}\nbox is the object's bounding box as fractions of the image width/height (0 = left/top, 1 = right/bottom).\nList each distinct object once (at most 8). Do not list ground, sky, walls, floor or other background surfaces as objects."},{"type":"image_url","image_url":{"url":"<image 19126B sha=025a8134a5cb18a47b16554f>"}}]}],"ms":9944}

plugins/ml/cache/deepinfra/00fd27ec74511767b97cc7bf.json

@@ -0,0 +1 @@
+{"text":"```json\n{\n  \"setting\": \"indoor\",\n  \"scene\": \"A room with a building-like structure and a hexagonal shape.\",\n  \"objects\": [\n    {\n      \"label\": \"building-like structure\",\n      \"category\": \"building\",\n      \"region\": \"center\",\n      \"box\": [0.2, 0.3, 0.8, 0.5],\n      \"moving\": \"none\"\n    },\n    {\n      \"label\": \"hexagonal shape\",\n      \"category\": \"shape\",\n      \"region\": \"bottom\",\n      \"box\": [0.4, 0.7, 0.6, 0.9],\n      \"moving\": \"none\"\n    }\n  ]\n}\n```","usage":{"prompt_tokens":564,"total_tokens":721,"completion_tokens":157,"estimated_cost":0.0000737,"prompt_tokens_details":null},"model":"mistralai/Mistral-Small-3.2-24B-Instruct-2506","request":[{"role":"user","content":[{"type":"text","text":"The image shows a character-art rendering of ONE view from a fixed camera. Each character stands for a small patch of the image: denser characters are brighter or darker patches depending on the renderer, lines may mark edges, and some renderings mark motion with arrow characters (> < ^ v or → ← ↑ ↓) and '~' for where something just was. It may contain several frames of the same view separated by blank lines, oldest first.\nReconstruct the scene in the latest frame: what objects are there, where, and which way each is moving (\"none\" if you cannot tell or it is still).\nReturn ONLY a JSON object, no prose:\n{\"setting\": \"outdoor-day\" | \"outdoor-night\" | \"indoor\" | \"unknown\",\n \"scene\": \"<one short sentence>\",\n \"objects\": [{\"label\": \"<noun>\", \"category\": \"person\" | \"animal\" | \"vehicle\" | \"building\" | \"plant\" | \"celestial\" | \"furniture\" | \"shape\" | \"weather\" | \"other\",\n   \"region\": \"top-left\" | \"top\" | \"top-right\" | \"left\" | \"center\" | \"right\" | \"bottom-left\" | \"bottom\" | \"bottom-right\",\n   \"box\": [x0, y0, x1, y1],\n   \"moving\": \"left\" | \"right\" | \"up\" | \"down\" | \"none\"}]}\nbox is the object's bounding box as fractions of the image width/height (0 = left/top, 1 = right/bottom).\nList each distinct object once (at most 8). Do not list ground, sky, walls, floor or other background surfaces as objects."},{"type":"image_url","image_url":{"url":"<image 13350B sha=a3208a859195fb4d1de8a0d5>"}}]}],"ms":7994}

plugins/ml/cache/deepinfra/01176b0e1259cfb64d550266.json

@@ -0,0 +1 @@
+{"text":"```json\n{\"setting\": \"outdoor-day\",\n \"scene\": \"A vehicle is moving to the right across a field.\",\n \"objects\": [{\"label\": \"vehicle\", \"category\": \"vehicle\", \"region\": \"bottom\", \"box\": [0.1, 0.6, 0.5, 0.9], \"moving\": \"right\"}, {\"label\": \"object\", \"category\": \"other\", \"region\": \"top-right\", \"box\": [0.6, 0.1, 0.9, 0.4], \"moving\": \"none\"}]}\n```","usage":{"prompt_tokens":638,"total_tokens":769,"completion_tokens":131,"estimated_cost":0.000072,"prompt_tokens_details":null},"model":"google/gemma-3-27b-it","request":[{"role":"user","content":[{"type":"text","text":"The image shows a character-art rendering of ONE view from a fixed camera. Each character stands for a small patch of the image: denser characters are brighter or darker patches depending on the renderer, lines may mark edges, and some renderings mark motion with arrow characters (> < ^ v or → ← ↑ ↓) and '~' for where something just was. It may contain several frames of the same view separated by blank lines, oldest first.\nReconstruct the scene in the latest frame: what objects are there, where, and which way each is moving (\"none\" if you cannot tell or it is still).\nReturn ONLY a JSON object, no prose:\n{\"setting\": \"outdoor-day\" | \"outdoor-night\" | \"indoor\" | \"unknown\",\n \"scene\": \"<one short sentence>\",\n \"objects\": [{\"label\": \"<noun>\", \"category\": \"person\" | \"animal\" | \"vehicle\" | \"building\" | \"plant\" | \"celestial\" | \"furniture\" | \"shape\" | \"weather\" | \"other\",\n   \"region\": \"top-left\" | \"top\" | \"top-right\" | \"left\" | \"center\" | \"right\" | \"bottom-left\" | \"bottom\" | \"bottom-right\",\n   \"box\": [x0, y0, x1, y1],\n   \"moving\": \"left\" | \"right\" | \"up\" | \"down\" | \"none\"}]}\nbox is the object's bounding box as fractions of the image width/height (0 = left/top, 1 = right/bottom).\nList each distinct object once (at most 8). Do not list ground, sky, walls, floor or other background surfaces as objects."},{"type":"image_url","image_url":{"url":"<image 19126B sha=025a8134a5cb18a47b16554f>"}}]}],"ms":7412}