Problem
evals.rs currently ships only builtin_scenarios() — a held-in set. Weng's post insists self-improvement be validated against both held-in ("did we fix the weakness?") and held-out ("did we break something else?") splits, with the evaluator sitting outside the loop that evolves the harness.
Today, brain corrections already feed few-shots that bias future brain decisions — actor and judge share a signal path. A held-out eval the loop is never allowed to see or tune against is what keeps that honest.
Proposal
Why it matters
This is the smallest change that makes the entire autonomy ladder safe, and it's independently useful today for regression-testing brain prompt/rule changes. Distinct from the Tier-3 cross-repo transfer test (#137) — this gates the loop's own edits.
Part of epic #358. Blocks #360.
Problem
evals.rscurrently ships onlybuiltin_scenarios()— a held-in set. Weng's post insists self-improvement be validated against both held-in ("did we fix the weakness?") and held-out ("did we break something else?") splits, with the evaluator sitting outside the loop that evolves the harness.Today, brain corrections already feed few-shots that bias future brain decisions — actor and judge share a signal path. A held-out eval the loop is never allowed to see or tune against is what keeps that honest.
Proposal
decisions.rsretrieval or prompt distillation.Why it matters
This is the smallest change that makes the entire autonomy ladder safe, and it's independently useful today for regression-testing brain prompt/rule changes. Distinct from the Tier-3 cross-repo transfer test (#137) — this gates the loop's own edits.
Part of epic #358. Blocks #360.