PK2: campaign contract + KEYSTONE/scoreboard rows (T1 prefill LIVE GREEN, T2/T3/T4 offline GREEN, live task-loop honest boundary)

Author: nihilistau

papers/CONTRACT-PRODUCT-KEYSTONE-2.md

@@ -0,0 +1,108 @@
+---
+type: contract
+title: "CONTRACT — PRODUCT KEYSTONE-2: from organism to product"
+description: "The post-KEYSTONE product campaign (2026-07-07): unblock the prefill stall, expand the harness into a real coding/agentic system, tighten the tiered memory into MEM-OKF v2, and grow the personality + served UI into a product surface. Four tranches, each gated, each default-off = null floor."
+tags: [contract, campaign, product, harness, agency, memory, mem-okf, persona, ui, prefill]
+timestamp: 2026-07-07T00:00:00Z
+resource: shannon-prime-lattice
+sp_status: ACTIVE
+sp_gate: "G-PK2-* (per-tranche, §T1–§T4)"
+sp_commit: TBD
+sp_repro: "per-tranche DONE-WHEN commands, §T1–§T4"
+---
+
+# CONTRACT — PRODUCT KEYSTONE-2
+
+> Foundation: [PPT-LAT-KEYSTONE.md](PPT-LAT-KEYSTONE.md) (the organism, ~90% built, live).
+> This contract is the campaign that turns the organism into a **product**: robust under real
+> load, able to *do work* (code, tools, agency), with memory it can trust and a face worth using.
+> Discipline unchanged: receipts-first, pre-registered gates, default-off = byte-identical null
+> floor, no number without a command + a row.
+
+## 0. Priorities (operator, 2026-07-07)
+
+1. Coding / agentic functionality
+2. Expand the harness system
+3. Expand the personality system
+4. Tighten + expand the memory system (MEM-OKF v2)
+5. Expand the UI
+
+Cross-cutting surfaces named: ADR-002..005, XBAR, NIGHTSHIFT, L5, decoder, execute, KAIROS,
+the Spine, the heads, Telepathy, SSE, the inject seam.
+
+## T1 — PREFILL STALL (the root blocker; do first)
+
+**Problem** (`project_daemon_prefill_stall`, engine `START-HERE-prefill-stall.md`): daemon
+prefill wedges forever at n≈1000–1700 tokens. max_tokens=1 also stalls ⇒ PREFILL. Blocks the
+agent gateway (persona+tools ≈ >1000-tok prompts) and long chats — i.e. blocks priorities 1–3.
+
+**Diagnosis targets** (recon 2026-07-07, `cuda_forward.cu`):
+- H1: kernel-launch error is *silent* until the `cudaStreamSynchronize` at the end of
+  `gemma4_kv_prefill` (~:4630); the launch-check at ~:4364 reads `cudaGetLastError` too late.
+- H2: per-token prefill loop queues n×~48-layer kernel launches (~85k at n=1765) with **zero
+  intermediate sync** — stream-queue exhaustion wedges the driver.
+- H3: `attn_shm = Pmax*4` shared-memory sizing / Pmax bound.
+
+**Fix shape:** periodic mini-sync (every 256 tokens) inside the prefill loop + pre-launch error
+check + Pmax/shm validation at open. All engine-local, no ABI change.
+
+**DONE-WHEN (G-PK2-PREFILL):** `SP_WORDS=1300 python tests\perf\_g_bigprompt_probe.py` →
+`got_DONE=True`; then n=299 regression still GREEN; then gateway `_g_memory_check.py` on :8800
+completes. Receipt `tests/perf/G-PK2-PREFILL.log`.
+
+## T2 — CODING / AGENTIC + HARNESS EXPANSION
+
+The harness already has: ephemeral `<tool>` calling (`run_with_tools`), memory/conversation
+tools, system tools (shell/python/web/file), sandboxed coding tools, the agency round +
+KAIROS scheduler, the :8800 gateway (persona live-read). Expansion = **depth + robustness**:
+
+- **E1 Agentic task loop:** a `run_task(goal)` multi-step loop (plan → act → observe → check),
+  bounded steps + budget, task state persisted to MEM-OKF (resumable), honest failure surface.
+- **E2 Coding campaign tools:** workspace-scoped edit (anchored find/replace, not whole-file),
+  patch-apply + diff receipt, test-runner tool (`run_pytest`), git status/diff (read-only first).
+- **E3 Robustness:** malformed-tool-call recovery (re-prompt with the parse error, N retries),
+  tool timeout + output truncation tiers (head+tail), loop/no-progress detector.
+- **E4 KAIROS agency growth:** the maintenance round gains a *work queue* (operator-posted tasks
+  the organism advances on ticks), receipted per tick.
+
+**DONE-WHEN:** G-PK2-TASKLOOP (a 3-step coding task — write module + test, run tests, fix a
+seeded failure — completes E2E on the live 12B through the gateway); G-PK2-TOOLROBUST (seeded
+malformed/timeout/looping tool scenarios all recovered, receipts); existing H1–H7 stay GREEN.
+
+## T3 — MEMORY: MEM-OKF v2
+
+- **M1 Provenance lane** (vision `emergent_provenance`): every registry fact + MEM-OKF record
+  carries `src` (turn/episode/operator/consolidator + timestamp); recall recites provenance on
+  demand ("where did I learn that?").
+- **M2 Extraction hardening:** consolidator pulls convo facts into MID reliably (dedupe via
+  token-overlap vs registry before append; the DECIDE/MERGE layer already handles conflicts).
+- **M3 Registry hygiene:** compaction pass (drop superseded/forgotten tombstones), integrity
+  check tool (`okf_mem verify`), size/age telemetry in `/v1/metrics`.
+- **M4 Tier plumbing:** `recall_conversations` quality pass + capabilities corpus refresh; keep
+  the Jaccard @0.6 verifier as the production gate (26B cascade stays retired).
+
+**DONE-WHEN:** G-PK2-PROVENANCE (fact stored → "where did you learn X?" → correct source
+recited); G-PK2-MEMHYGIENE (verify tool GREEN on live stores; compaction preserves recall on a
+before/after probe set); G-FORGET/G-DECIDE/G-MERGE regression GREEN.
+
+## T4 — PERSONALITY + UI
+
+- **P1 Persona:** structured `persona.md` v2 (identity / voice / values / self-knowledge
+  sections), gateway hot-reload (exists) + a persona *editor* in the console; persona changes
+  are memory events (provenance: operator-set).
+- **P2 Self-knowledge:** capabilities corpus refresh post-T2 (the organism can state its own
+  new tools), seeded via `_seed_capabilities.py`.
+- **U1 Console v2 (`index.html`):** tool-call display (render `<tool>` calls + results as
+  cards in the stream), memory browser pane (registry facts + MEM-OKF LUT, read-only first),
+  persona editor textarea (POST to gateway), SSE robustness (reconnect, abort button wired).
+- **U2 Telemetry pane:** /v1/metrics poll (VRAM, tok/s, memory counts).
+
+**DONE-WHEN:** G-PK2-UI (chat + tool cards + memory pane + persona edit round-trip live on
+:8800/:3000); G-PK2-PERSONA (persona edit → next turn reflects it → provenance recorded).
+
+## Order & rules
+
+T1 → T2 → (T3 ∥ T4). Git ops native PowerShell only (mount CRLF-churns). Large-file reads on
+the mount truncate — verify via native tools. One daemon at a time (12B fills the 2060).
+Pre-flight `okf_mem lookup` before building any new capability. Every mechanism lands behind a
+default-off flag or additive endpoint; the null floor is byte-identical.

papers/PPT-LAT-KEYSTONE.md

@@ -325,6 +325,23 @@ exact-fetch. It ships curated records, never raw latent (ADR-002). **Remaining =
 Design: [PPT-LAT-DESIGN-SWARM-MEMORY-MESH.md](PPT-LAT-DESIGN-SWARM-MEMORY-MESH.md); call surface:
 [PPT-LAT-MESH-API.md](PPT-LAT-MESH-API.md).
 
+**Post-KEYSTONE addition (2026-07-07) — PRODUCT KEYSTONE-2 [see CONTRACT-PRODUCT-KEYSTONE-2]:**
+the campaign that hardens the organism into a product. **T1 (LIVE GREEN):** the >~1000-token
+daemon **prefill "stall" is RESOLVED** — root cause was the SWA ring never arming in served
+daemons (Rust `std::env::set_var` is invisible to the CUDA lib's C `getenv` on Windows), so
+every served daemon silently ran ring-off full-cache at PMAX=20000 → VRAM oversubscription →
+WDDM paging thrash. Fix: `_putenv_s` CRT bridge in `daemon.rs` + chunked-sync/fail-fast/telemetry
+in `gemma4_kv_prefill`. Gate `SP_WORDS=1300 → got_DONE=True` 98.2s (was: forever). Receipt
+`engine/tests/perf/G-PK2-PREFILL.log`. **T2 (harness, offline GREEN + live finding):** a coding/
+agentic tool set (`edit_file`/`run_tests`/git), a bounded **resumable task loop** (`run_task`,
+work queue drained on the KAIROS tick), and tool-call robustness (malformed-recovery, no-progress
+break, **verify-before-accept** — a 12B will confabulate "DONE" without its edit landing; the
+harness now checks). `G-PK2-TOOLROBUST` 10/10. **T3 (offline GREEN):** MEM-OKF v2 — a
+**provenance lane** (`remember(source=…)` + `provenance()`), near-dup extraction guard, registry
+`verify`/`compact` hygiene. `G-PK2-MEMOKF-V2` 6/6. **T4 (offline GREEN):** operator-panel gateway
+surfaces (`/v1/memory`, `/v1/tasks`, `/v1/persona` editor), `operator.html`, self-knowledge
+refresh. `G-PK2-UI-ENDPOINTS` 5/5. Receipts: `harness/tests/G-PK2-TRANCHE-SUMMARY.md`.
+
 **Open edges (next):** (1) **persistent O(1) conversation KV** — the daemon re-prefills the whole
 conversation each turn (correct but O(n)); the L1 stateful kvdecode verb can make "continue the
 cache" true O(1). (2) The external **two-physical-GPU** bit-identical check for byte-exact.

papers/VERIFIED-SCOREBOARD.md

@@ -45,6 +45,8 @@ Method: a read-only fleet checked each claim against (a) a commit that resolves
 | **LIVING MEMORY stack** (ADR-005: hot-reload · decision-telemetry · turn-telemetry · idle refine) | **GREEN, default-off** | engine `de8506e`→`6431e55`→`e272f4a`→`70e4992` | `chat_fullstack/G-LM-{RECONCILE,TELEMETRY,REFINE,TURNTELEM,COMPOSE}.log` | VERIFIED — 4 bricks, each isolated-gated, then COMPOSED (`G-LM-COMPOSE`): all flags on at once serve 3 policies (counterfact/private-secret/persona) while the idle thread BOTH reconciles a concept written mid-run (`+1 hot-loaded` → served "Marlowe City", no restart) AND model-refines a mis-classed secret (counterfact→private-secret on idle), every turn logs a decision + turn record (4+4), private-secret redacted (secret 0 hits). **B1** `SP_MEM_RECONCILE` idle reconciler; **B2** `SP_TELEMETRY` class-redacted decision log; **B2b** turn-outcome (output+tok_s+obeyed, secret output redacted = finetune data); **B3** `SP_MEM_CLASSIFY_REFINE` idle model-refine (safety-monotone). Faithfulness-neutral: none of these flags are set by `run_console_faithful.bat`, and none touches L5 selection scoring → the 54/61 one-config stack is byte-identical. Design: `PPT-LAT-ADR-005-LIVING-MEMORY.md` + `CONTRACT-LIVING-MEMORY.md` |
 | **SELF-IMPROVEMENT LOOP** (data-gen + finetune: telemetry → train → promote → deploy, autonomous) | **GREEN, default-off** | harness `b3fb755`→`bfaa4eb`; engine `6d418a4` | harness `tests/G-DF-{CONVERT,SEED,TRAIN-CLOUD,EVAL,DEPLOY,PARITY,AUTOTRAIN}.log` + engine `chat_fullstack/G-DF-LIVE.log` | VERIFIED — the LM-B2 telemetry flywheel closed into a live self-improvement loop (`CONTRACT-DATAGEN-FINETUNE`, every brick lifts a named CosySim module, anti-rebuild). **DF-B1** telemetry→Alpaca JSONL (privacy choke: redacted skipped); **DF-B2** synthetic mem_class seed (6 balanced, incl synthetic private-secret); **DF-B3** QLoRA on **Colab T4** (Qwen2.5-0.5B, HF-mediated, adapter→`KnackAU/sp-mem-class-adapter`); **DF-B4** distinct-held-out A/B (**20%→83.3%** vs base) + registry `gate_and_promote` (MUST_IMPROVE); **DF-B6** harness curator deploys the 0.5B **on CPU** (safety-monotone) replacing the engine's 12B `model_classify`; **DF-B5** auto-train trigger (accrued telemetry→fire, idempotent, DRY unless `SP_AUTOTRAIN_LIVE`). **G-DF-LIVE**: curator corrects the served store live → engine **reconcile-on-edit** serves the correction, no restart. **G-DF-PARITY**: the deployed 0.5B **BEATS** the 12B it replaces (**0.83 vs 0.33** vs ground truth — the 12B `model_classify` was the weak classifier). Whole loop default-off; the 12B refine stays as fallback. |
 | **PERSONALITY FRAMEWORK** (self-modifiable + system-curatable persona/self-model) | **GREEN, default-off (`SP_PERSONALITY`)** | harness `a1c59ea`→`e35cfdf` | harness `tests/G-PF-{OWNERSHIP,PERSONA,TAGS,DECORATORS,CURATE}.log` | VERIFIED — `CONTRACT-PERSONALITY` PF-B1..B5, every brick EXTENDS a named existing seam (anti-rebuild). **PF-B1** `mem_owner` axis (self\|user) orthogonal to `mem_class`, owner-tagged OKF concepts (`memory-okf-self/`), no classifier; **PF-B2** structured `persona.md` `## Personality state` block folded into `load_agent_system` + self-model injection; **PF-B3** `PersonalityStateInterceptor` persists the `[MOOD]/[VOICE]/[TRAIT±]` tags the model already emits; **PF-B4** `@personality` decorators (=`@skill` pack) — the model CALLS `adjust_mood/set_voice/set_trait/remember_self` via `run_with_tools` to durably self-modify; **PF-B5** `consolidate_personality` (mirrors `consolidate_conversation`) extracts transcript shifts + prunes stale traits + snapshots to content-addressed `memory-okf-personality/` tier, wired into `agency.py` gated `SP_PERSONALITY`. Personality self-modifiable (model) AND system-curatable (NIGHTSHIFT). PF-B6 (engine-native personality head) DEFERRED. ADR-002: personality = DECISION → clean tag/decorator/label → EXECUTE. |
+| **PRODUCT KEYSTONE-2 T1: prefill "stall" fix** (`gemma4_kv_prefill` + `daemon.rs` CRT bridge) | **GREEN-LIVE 2026-07-07** | engine (this session) | `tests/perf/G-PK2-PREFILL.log` | VERIFIED — the >~1000-tok "stall" was the SWA ring never arming in served daemons (Rust `set_var` invisible to the CUDA C `getenv` on Windows) → ring-off full-cache @PMAX=20000 → VRAM oversubscription/WDDM thrash. Fix: `_putenv_s` CRT bridge + chunked-sync/fail-fast/telemetry. `SP_WORDS=1300 → got_DONE=True` 98.2s (was forever); n=299 regression 12.0s; ring-ON chunk times flatten past dpos=1024 (O(1)-context). LESSON: Rust→C env needs `_putenv_s` |
+| **PRODUCT KEYSTONE-2 T2/T3/T4: harness product hardening** (coding tools · task loop · MEM-OKF v2 · UI surfaces) | **GREEN offline; T2 live = honest capability boundary** | harness (this session) | harness `tests/G-PK2-{TOOLROBUST,MEMOKF-V2,UI-ENDPOINTS}.log` + `G-PK2-TRANCHE-SUMMARY.md` | VERIFIED (offline) — **T2** `G-PK2-TOOLROBUST` 10/10: `edit_file`/`run_tests`/git tools, resumable `run_task` + work queue, malformed-recovery + no-progress break + **verify-before-accept** (closes the 12B "DONE"-confabulation gap found live); **T3** `G-PK2-MEMOKF-V2` 6/6: provenance lane (`remember(source)`+`provenance()`), near-dup guard, registry `verify`/`compact`; **T4** `G-PK2-UI-ENDPOINTS` 5/5: `/v1/memory`·`/v1/tasks`·`/v1/persona` gateway surfaces + `operator.html` panel + self-knowledge refresh. Live task-loop: the 12B autonomously editing code to green is at/beyond this model's reliable capability (honest negative); the harness now refuses to confabulate a pass. Contract: `CONTRACT-PRODUCT-KEYSTONE-2.md` |
 
 ## The open frontier (what we actually skipped)