A standing benchmark base set: quality, performance, intelligence and achievement — coding, league, math and algorithms, with graded metrics that do not saturate

[OriNachum]

The gap this proposes to close

This repo can now say, precisely and at scale, how fast and how much. It
cannot say how well.

The asymmetry is stark once you line the results up:

what	best evidence	scale
throughput	bee-hive-width.md — SEPARATED-WIDTH, ~3.3× at width 8	7,200 calls, 48 cells, 3 paired repetitions
token cost	league-commander.md, worker-scoped-overhead.md	384 records; per-call accounting
latency	muse-latency.md, streaming-probe.md	derived bounds, CI-enforced
protocol / interface compliance	league-h2h.md, arena-budget.md	196 model calls, zero truncation
correctness	designed-problem.md (n=4 per arm), muse-arms.md (8/8 — a ceiling), devague-legs.md (11/12, pools separable by input length — #28)	n ≤ 12, and saturated

Six result documents carry the word ceiling. Four experiments tied at one
(#35). And the one metric that decided a three-way model ranking turned out to
be interface compliance, not play — league-h2h.md separated full-qwen > mixed > full-gemma while its outcome metric tied 0–0 in all six matches.

So the honest statement of where we are: every quality claim this repo has
made rests on a saturated metric, a bespoke task, or n≤12 — and no two
experiments share a task, so nothing composes across cycles. Meanwhile the
throughput lane is measured to three decimal places on 7,200 calls.

This issue proposes the fix: a standing base set — fixed tasks, fixed
graded metrics, re-runnable every cycle, comparable across cycles.

Why now, specifically

Three things changed that make this affordable and honest for the first time.

1. Width separated. ~3.3× sustained at width 8 means a graded suite of a few
   thousand items is ~1/3 the wall clock it would have cost before. Grading was
   previously expensive enough to skip; it is not any more.
2. The clocks are derived and CI-enforced. test_timeout_bounds.py walks 8
   constants against a dated rate config. A long-running suite can no longer be
   silently truncated by a clock — which is exactly what censored a completion
   distribution while recording truncated: false.
3. Streaming is the default transport. Generation length stopped being the
   binding quantity at every hop, so a suite with long reasoning traces is
   dialable at all.

Design rules — inherited, not re-derived

These come from #35, #40 and this cycle's corrections. Quoting them so the
suite is built against them rather than rediscovering them a fifth time.

• H0 headroom screen runs first, per family. If the control already scores
  near-perfect, there is nothing for any arm to improve and the result is a
  ceiling, not a finding. A family that fails its headroom screen is replaced
  or escalated, never run anyway. This is Escalate until separation: four experiments tied at a ceiling, and the ladder is the general fix #35's ladder applied per task family
  rather than per experiment.
• The outcome metric must be mandatory-populated and must not be optional-field
  shaped. league-h2h ranked three model configurations on a field one arm
  simply never filled in. Registered metrics must be things every arm produces by
  construction.
• Pre-register the ladder, not just the thresholds (Escalate until separation: four experiments tied at a ceiling, and the ladder is the general fix #35). Each family
  declares its rungs before the first dial, so "escalate until separation" is
  applied by code rather than by judgement after a tie.
• Acceptance is not correctness. t8's "106 of 106 answered" measured that a
  reply arrived in the declared shape. W1 kept acceptance out of its numerator on
  purpose. In this suite, acceptance is a gate, correctness is the metric, and
  they are reported in separate columns.
• Thinking is a quality decision, not a cost knob. Any arm that varies the
  reasoning setting must grade correctness at both settings. A token budget met
  by removing reasoning is optimising the metric, not the outcome.

Blocking prerequisites — measuring quality before these is measuring noise

Honest quality measurement is blocked on four known defects. Listing them as
prerequisites rather than caveats, because three of them silently corrupt a
correctness score rather than failing loudly.

#	defect	what it does to a quality score
#37	ModelResponse carries no finish_reason	a truncated answer and a deliberately short one are the same object — a cut-off answer scores as a wrong answer
#32	handed tools, the model writes a tool call and no prose	scores as NO_ANSWER; measured 9 of 16 runs in one arm
#33	command sent as a string rather than an array	74% of one arm's calls refused — scores as inability
—	WorkerSeam has no thinking/extra-body parameter	the reasoning mode cannot be pinned, so a graded result is not reproducible (drone-economics.md, "What is owed")

#37 is the one that matters most, and it is cheap. Without it, every wrong
answer in the suite is ambiguous between the model was wrong and the clock cut
it off — which is the exact failure this cycle spent four clock fixes on, moved
one layer up.

The four axes

The four the request names, made measurable and kept distinct — they answer
different questions and conflating them is how the last ranking went wrong.

axis	question	metric shape	fails when
Quality	is the answer right	graded accuracy against ground truth	saturates on easy tasks
Performance	what did it cost	items/sec, tokens/item, wall clock	already solved; carry it as a column, not a verdict
Intelligence	does it generalise	accuracy held-out vs in-distribution; transfer between families	the held-out set leaks
Achievement	can it complete a whole task	end-to-end pass rate on multi-step goals	partial credit hides where it stopped

Achievement is the one worth adding most, because it is the only axis that
cannot be gamed by a good single turn — and it is the closest to what an
embodiment actually is.

The task families

Four, chosen so their difficulty is tunable (so headroom is recoverable when a
rung saturates) and their grading is mechanical (so no grader has to be
trusted — corrections.md §2 is four separate cases of a grader being wrong in
the flattering direction).

1. Coding — graded by the repo's own test suites

Metric: fraction of held-out tasks where the produced patch makes a
previously-failing test pass and breaks no passing test.

Mechanical, unfakeable, and this repo is an unusually good corpus for it: 5,141
tests, and a git history where each commit's diff and its tests are both
recorded. Rungs: single-function fix → multi-file change → a change that must
add its own test → a change under a constraint (e.g. no new dependency, which
test_zero_deps.py already enforces).

Headroom note: must be screened. If the cortex passes the easy rung 10/10,
that rung is a ceiling and the suite starts one rung up.

2. League of Agents — but with the outcome metric fixed

The existing league harnesses (league_h2h.py, league_commander.py,
league_seat.py) already exist and are dialable. What failed last time was the
metric, not the harness: outcome tied 0–0 in all six matches and the ranking
came from an optional team-message field.

Metric: net_margin against a fixed house bot ladder of increasing
strength, not head-to-head between arms. Head-to-head between two agents that
both tie the house produces a tie; a strength ladder produces a rating, which
does not saturate.

The scripted control already showed identical minds never separate (4/4 rungs
INCONCLUSIVE) and that the paired net_margin cancels colour bias exactly
(0.0 everywhere) — so the instrument for this is already validated.

3. Math — the family with the cleanest ground truth

Metric: exact-match accuracy on held-out problems with numeric or symbolic
answers, plus disagreement rate across N samples as the leading indicator
for #48.

This family exists mainly to serve #48 (reasoning + replication): a small
enumerable answer space makes majority voting well-defined in a way free text
does not. Rungs: arithmetic word problems → multi-step algebra → proof-shaped
problems where the steps are graded, not just the final value.

This is where the N=3 non-reasoning control belongs — without it a win is
unattributable between "replication helps" and "reasoning helps".

4. Computer science / algorithms — where achievement is testable

Metric: does the produced implementation pass a hidden test set and meet
a stated complexity bound (checked by instrumented operation counts, not by
reading the code).

The complexity bound is the interesting part: it is a constraint the model
cannot satisfy by pattern-matching a solution, and it is mechanically checkable.
Rungs: known algorithms → a variant with a twist → an algorithm under a memory or
call-count budget.

What the base set is for

Concretely, the questions it lets us answer that we currently cannot:

• Redundancy for quality: spend the measured width on N answers to one question, not N distinct questions #48 — does N=3 reasoning with majority vote beat N=1 on graded accuracy at
  ~3× tokens and ~1× wall clock? Needs a non-saturating graded metric, which is
  exactly what is missing today.
• The muse question, finally — t18 returned INCONCLUSIVE because arm A's
  confidently-wrong rate was 0 of 8. A family with headroom by construction is
  the only way that question gets a real answer.
• d15's premise — the muse-off decision rests on cost evidence and on
  INCONCLUSIVE counsel results, not on a measured quality comparison. A base set
  would let that decision be checked rather than only justified.
• Arms B0 / B2 / P — built, hermetically tested, never dialled. They have
  no outcome metric that grades correctness to be dialled against. This supplies
  one.
• Thinking on vs off — currently unanswerable, and blocking the drone tier's
  token target.

Cost model, from measured numbers

Grounded in W1 rather than guessed. At width 8, unit grain: 14.09 items/s
graded, 14.21 completion tokens per scoped call, prefill-bound at 13.9:1.

suite shape	items	est. wall clock at width 8	note
headroom screen, per family	~50	~4 s + reasoning time	cheap enough to run every cycle
one graded rung, per family	~320	~23 s at scoped-call size	W1's exact cell size
full base set, 4 families × 3 rungs	~3,840	minutes, not hours	at scoped-call size

The caveat that matters: those numbers are for scoped calls of tens of
tokens. A coding task with a real reasoning trace is 100–1000× more completion
tokens per item, so the coding and CS families are hours, not minutes. The
math and league families are closer to the W1 profile. Sizing each family
against its own measured per-item cost is a task, not an assumption — and
sizing a clock against the wrong quantity is the mistake this cycle made four
times.

Proposed sequencing

1. Prerequisites — ModelResponse carries no finish_reason, so a truncated turn and a stopped turn are indistinguishable — a C3 violation #37 at minimum, plus the WorkerSeam thinking parameter.
   Without these the results are ambiguous by construction.
2. Headroom screens, all four families. Cheap, and they decide which rungs are
   worth building. Publishing "family X saturates at rung 1" is a real result.
3. One family end to end — math first, because its ground truth is cleanest
   and it directly serves Redundancy for quality: spend the measured width on N answers to one question, not N distinct questions #48.
4. The remaining three, then the cross-family transfer measurement that makes
   the intelligence axis meaningful.
5. Re-dial the parked arms (B0/B2/P) against the new metric.

Non-goals, stated up front

• Not a replacement for public benchmarks, and not a claim of comparability
  with them. Held-out sets drawn from this repo's own history are contaminated for
  any model trained on public GitHub; the base set measures our arms against each
  other on a fixed task, not absolute capability.
• Not a promise that any arm wins. Every family may return INCONCLUSIVE, and
  a family that saturates its headroom screen is a published result, not a failure.
• Not a new harness family. Where a harness exists (league_*, arch_*), the
  suite dials it; the new code should be tasks, graders and the runner.

What would falsify the premise

If the headroom screens come back saturated on all four families, then the
tasks are too easy for this rig and the correct conclusion is that this rig
cannot discriminate its own arms on quality at all — which would be a genuinely
important negative, and would redirect effort to harder task sources rather than
to more arms.

---

Filed as a proposal, not a plan. It wants a /scope → /think → /challenge
pass before any of it is built, and the prerequisites above are the part I would
argue should land first regardless of whether the rest is approved.

Related: #35 (escalate until separation — the ladder), #40 (the ceiling gate),
#48 (reasoning + replication), #44 (the Bee-Hive arms awaiting a metric),
#37/#32/#33 (the prerequisites), #34/#36/#39 (existing experiment proposals this
would give a shared metric to).

• embodiment (Claude)