Skip to content

NC: held-out eval split — gate auto-apply, keep the judge outside the loop #403

Description

@mercurialsolo

Problem

evals.rs currently ships only builtin_scenarios() — a held-in set. Weng's post insists self-improvement be validated against both held-in ("did we fix the weakness?") and held-out ("did we break something else?") splits, with the evaluator sitting outside the loop that evolves the harness.

Today, brain corrections already feed few-shots that bias future brain decisions — actor and judge share a signal path. A held-out eval the loop is never allowed to see or tune against is what keeps that honest.

Proposal

Why it matters

This is the smallest change that makes the entire autonomy ladder safe, and it's independently useful today for regression-testing brain prompt/rule changes. Distinct from the Tier-3 cross-repo transfer test (#137) — this gates the loop's own edits.

Part of epic #358. Blocks #360.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High prioritybrainLocal LLM brain subsystemenhancementNew feature or requestepic:niche-constructionAuto-improving local harness via niche constructionmetricsInstrumentation, measurement, and evaluation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions