Skip to content

EV-007 main run: 3-arm pre-registered A/B on the learning loop (bare / harness-no-learning / harness-learning) #202

Description

@shojikumaru

Why

EV-007 P0 (2026-08-21) proved the short-cycle distillation loop wires up — and simultaneously showed the content it moves is near-worthless progress chatter (REPORT-P0 §4.3), the intake gate is advisory (§4.2), and pending flushes could be silently clobbered (§4.5). The #144 adjudication (2026-08-24) ruled: the main-run pre-registration is written against the new learning-loop design (#147 raw layer / #148 nightly cross-model review / #145 Last session pointer / #170 VF-GR caps), all of which have since merged (main = v0.19.0). Separately, EV-008 completed the overflow-sentinel effect measurement and the owner declared default-on GO (v0.14.1) — so the original #159 D5 arm definition ("bare / always-harness / sentinel") would now re-measure what EV-008 already answered.

This issue is the home for the EV-007 main run: a 3-arm pre-registered A/B measuring the README core claim — the harness learns from raw experience and the next job gets better — on the sealed EV-006 rig.

Clarify writeback (owner decisions, 2026-08-27)

# Question Decision
C1 Arm definition Redefine on the learning-loop axis: bare / harness with learning loop disabled / harness with learning loop enabled (raw layer + nightly review + apply). Sentinel stays default-on in both harness arms per the EV-008 GO. This supersedes #159 clarify D5; a superseding comment is posted on #159.
C2 Experiment pin (HARNESS_DIR) Update the pinned worktree to v0.19.0 (current pin @3eab97a is a v0.12.1-era side branch without the new learning loop; incompatible with the #144 ruling). Pin SHA sealed in the prereg.
C3 apply-step ordering apply implementation lands first (own issue, size L). Prereg design proceeds in parallel, but before sealing: one light full-loop wiring re-check (P0-style smoke incl. apply, on the updated pin), then a second owner confirmation. Owner's words: "ライトに確実にいこう".
C4 Review (judge) model Kimi K3 — the only path validated by the 34-day real-corpus replay (5/6 windows, fabrication guard exercised); invocation via the decoration-stripping wrapper with measured timeouts. Sealed in the prereg.

Scope of this lane

  1. Write experiments/ev007/PREREGISTRATION-MAIN.md against the new design (arms per C1, pin per C2, judge per C4; metrics, pass/fail lines, budget, empty-run rules — P0 decisions carried forward where still valid).
  2. Observation targets carried in from open advisories: handoff files actually containing the 4 declared fields (PR last-session: session index with host-side fold and handoff files (#145) #172 Opus advisory — model-contract only, never verified); status of P0 findings §4.1–§4.5 under the new design (4.5 is claimed solved by learning loop: formalise the never-deleted raw layer and cut it weekly (splits 'where lessons accumulate' from 'what rides every prompt') #147/learning loop: weekly cross-model review over the raw layer — replace hash-equality k>=2 with theme-level judgement (and implement the 'reviewed by a different AI' claim for the first time) #148 — the prereg treats that as a guarantee to verify, not assume).
  3. Pin worktree update to v0.19.0 (C2) before the wiring re-check.
  4. Prereg passes heterogeneous 3-seat upstream review (L1-9; experiment definition = requirements-class work) before sealing.
  5. Seal gate (C3): apply issue merged → light wiring re-check → owner GO → seal → run.

Done when

  • PREREGISTRATION-MAIN.md drafted against v0.19.0 design (arms C1 / pin C2 / judge C4) with metrics, pass lines, budget cap, and deviation rules
  • 3-seat heterogeneous upstream review of the prereg: all blocking findings resolved
  • Pin worktree updated to v0.19.0 and recorded
  • Light full-loop wiring re-check executed after the apply issue merges (result reported to the owner)
  • Owner GO obtained → prereg sealed (immutable record, EV-006/EV-008 sealing conventions)
  • Main run executed per the sealed prereg; report published with the same correction-ledger discipline as EV-008

Files to touch (prediction)

  • experiments/ev007/PREREGISTRATION-MAIN.md, experiments/ev007/runner/*outside git (same posture as P0/EV-006)
  • pinned worktree caty-agent-harness-ev-pin (git worktree move to v0.19.0 — no source edits)
  • No caty-agent-harness source files are modified by this lane. Repo-side artifacts (report/docs) will be declared in a separate PR lane when the run completes.

Refs: #144 (adjudication comment 2026-08-24), #159 (D5 — superseded by C1), #148 #147 #145 #170, apply issue #201, PR #172 advisory, EV-008 record in #159 / docs/benchmark.md#ev-008.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions