fix(eval): mbpp-plus is 84.9% plus-graded, not 99.7% — and prose was resolving SWE-bench instances

[0bserver07]

Three commits, all measurement integrity. The second and third were found while landing the first.

1. The mbpp-plus number, plus-graded: 99.7% → 84.9%

The published 99.7% (377/378) was the MBPP+ task set graded with the dataset's base test_list asserts — MBPPPlus inherited MBPP's grading path and never executed the EvalPlus expanded test blob that was staged in every row.

Re-running the identical model (glm-5.2), tasks and agent under the now-wired expanded harness: 321/378 = 84.9%, status_counts {completed: 378}, $2.32. Nothing but grading changed, so the 14.8-point gap is attributable to grading alone.

Receipt: data/modal-grid-fullscore4-20260730-plusgrade.json.

The expanded contract is strictly stronger, not merely different: a hardcoded lookup satisfying every base assertion for is_not_prime is accepted by base and rejected by plus.

Ceiling is 377/378, not 378. Task 590 (polar_rect) cannot pass — the upstream harness compares floats with atol=0 and the dataset's own canonical answer differs in the last ULP. Recorded in CEILINGS so 84.9% reads against an achievable 99.7%.

Getting a comparable number took two runs. The first fell back to glm-5.1 because glm-5.2 was absent from the catalog, confounding grading against model (323/378 there). That stands as a second data point but is not the headline, because two variables moved.

2. The caveat markers were never on the number

CEILINGS and GRADING_NOTES shipped a release ago specifically so a caveat travels with a score — but †/‡ only ever appeared in the footnote lines below the table. A scorecard row is the unit anyone copies, so the number travelled and the caveat stayed behind.

Markers now render inside the score cell, suppressed on retracted rows (whose footnotes are not emitted). Section 3's mbpp+ ✓ ticks now disclose that their receipts predate the plus harness, for the same reason the retracted-adapter clause already did.

Also: DEFAULT_PATTERNS gained modal-grid-fullscore4-*. A receipt the generator does not glob is silently never read — this one first landed under an observatory-* name that put it in the depth matrix instead of the flagship row.

3. A fifth fabricated-result class: length-as-correctness

Five adapters — swe_bench, swt_bench, swe_polybench, feature_bench, dpai_arena (six sites) — graded a task solved from len(agent_output.strip()) > 10 whenever they could not run the benchmark's own tests.

Verified live, not inferred:

swe-bench      [env=no-runner] "I have analyzed the issue and implemented
                                a comprehensive fix."  ->  RESOLVED
swt-bench      [env=None]      same prose               ->  RESOLVED
swe-polybench  [env=no-runner] same prose               ->  RESOLVED
feature-bench  [env=no-runner] same prose               ->  RESOLVED
dpai-arena     [env=no-runner] same prose, 3 tracks     ->  RESOLVED

All six sites now grade unresolved. Inability to grade is not a pass, for the same reason a cloud sandbox refuses to degrade to local: either way the result becomes indistinguishable from a real one. The resulting zero is also legible — a uniform-zero column is already the harness-gap signature render_observatory.py refuses to publish as a score, whereas a 100% built from prose is invisible.

Three things kept this alive through a green suite:

• The adapters disagreed about which configuration was unsafe. swe_bench was safe at env=None and unsafe with a runner-less env; swt_bench was the exact mirror image. A test covering one shape would have passed on four of five. The new test parametrises both.
• The class spread by imitation. dpai_arena._evaluate_rubric's docstring said its placeholder was "matching the SWE-bench fallback behaviour". So alongside the behavioural test there is a static AST gate over chimera/eval/benchmarks/ rejecting len() of any answer-shaped parameter. AST and not grep: the fix's own comments quote the old code, and a text search would force deleting the explanation to stay green.
• Six existing tests asserted the defect — test_evaluate_fallback_heuristic, test_evaluate_no_env_uses_length_heuristic, test_evaluate_unknown_track_falls_back_to_length, +3. The suite was not blind to this behaviour, it was pinning it in place. All inverted, each carrying a note on what it used to assert.

Every new test was falsified against the unfixed code before being trusted. The behavioural one names all seven broken adapter/track combinations when reverted; the retracted-row test was additionally falsified against unconditional markers, because under the original bug no row had markers at all and it would otherwise have passed vacuously.

Corrections to previously recorded findings

• All five SWE-family datasets DO stage a gold patch. An earlier note of mine said "4 of 5 stage no gold patch" — that came from a probe miscounting environment_setup_commit as an image field. Measured census is in [bench] Canary the SWE family — gold patch must score 100%, prose must score 0% #175.
• The real blocker for a gold-patch canary is env provisioning: no dataset carries docker_image. The seven canary exemptions now cite that blocker and [bench] Canary the SWE family — gold patch must score 100%, prose must score 0% #175 rather than reading as healthy.

Rules, not just fixes

• CLAUDE.md: "Inability to grade is not a pass — and a test that asserts the lenient fallback is pinning the defect, not covering it," with the three sub-lessons.
• Playbook 13: a new invariant listing all five fabricated-result classes with the canary as its action, stating that the inverse half is not optional (a grader hardwired to True sails through a positive-only check) and that EXEMPT means unverified, never healthy.
• The claims gate rejected my own first draft of that playbook text for putting a % beside a retracted adapter's name. Reworded to a task count rather than widening the exemption list — that pattern is exactly how a withdrawn score survived in docs/specs for three weeks.

Gates

gate	result
full suite (extras)	10,356 passed, 140 skipped
CI posture (no tui)	10,387 passed, 131 skipped
mypy, cold cache, both postures	736 modules, 0 errors
ruff (chimera/ scripts/ tests/)	clean
trademark scrubs	7/7
claims + repo hygiene	75 passed
canary sweep	7 pass · 0 BROKEN

The single failure in both suite runs is tests/function_synthesis/test_validation_split.py::test_evaluate_returns_both_rates — documented in CLAUDE.md as env-sensitive locally and green in CI. Its test file, its subject (chimera/training/{spec,validation}.py), and chimera/function_synthesis/ are all byte-identical to origin/master, so this diff cannot have caused it.

Related: #175 (SWE-family canary, filed with the measured census).

🤖 Generated with Claude Code

---

Added after review opened — three more defects, same family

Landing the number above meant verifying claims I had already written down, and each check found something.

4. The ceiling was published with nothing behind it

My own changelog said task 590 was "recorded in KNOWN_UNPASSABLE and CEILINGS". Only the second was true — the canary entry was sitting uncommitted in a different worktree. CEILINGS' own docstring says it is "Verified by canary_benchmarks.py (KNOWN_UNPASSABLE)" and nothing enforced that pairing, so the page published a verified cap that no canary entry backed.

A ceiling is a published reduction of the denominator. It does not get to be self-asserted. A test now fails on any ceiling with no canary-side exclusion naming the task.

5. Wiring the plus harness silently inverted a canary verdict

Recipe.test_fields names the source the grader executes — a claim about another module, which rots silently. mbpp-plus carried test_fields=("test_list",) under a comment asserting that was "what MBPPPlus.evaluate ACTUALLY executes": true when written, wrong the moment the plus harness landed.

With numpy absent, the sweep reported mbpp-plus BROKEN — accusing a working grader — where the honest verdict was ENV-MISSING. That is the false-BROKEN class the ENV_MISSING verdict exists to prevent, and it sends someone to fix working code.

Proven three ways against a genuinely numpy-free venv:

scope	numpy	verdict
stale ("test_list",)	absent	BROKEN ← working grader accused
fixed ("test", "test_list")	absent	ENV-MISSING ✓
fixed	present	PASS ✓

A first attempt to reproduce this with an import-blocking shim was invalid and discarded: a numpy.py on the path keeps find_spec succeeding, so the canary cannot detect absence at all. The package has to actually be uninstalled.

The new test pins the scope together with the fact that the adapter really grades at plus strength, so un-wiring one breaks the other instead of leaving a green assertion describing nothing — which is exactly how the old comment rotted.

6. Substring grading accepted a different value as correct

The sibling of the length defect, in the graders that do have a reference answer and compared it with a raw in:

ContextBench:  truth "42"                 <- answer "142"                 PASSED
TauBench:      action "transfer_to_agent" <- "transfer_to_agent_v2"       PASSED

Both now match on word boundaries — applied only where the truth's own edge is alphanumeric, so a truth like $5 or f(x) stays matchable. Anchoring unconditionally would make those permanently unmatchable, trading a false-accept for a silent false-reject, which is worse: a zero column reads as a hard benchmark rather than a broken anchor.

Neither feeds a published number (context-bench is marked unverified in the capability matrix, tau-bench appears only as n=1 ✓ ticks), so this is prevention, not a retraction.

The residual leniency is disclosed and test-pinned, not solved. "The answer is NOT 42" still grades correct. Detecting that is natural-language judgement rather than string matching, and the judge hook exists for it. A test asserts the current behaviour, so if someone later teaches this path to reject negations, that test fails and forces them to update the docstring that discloses it. A limitation nobody can see is indistinguishable from a bug nobody found.

Two canary exemptions stated a reason that was false

context-bench and nocha were both listed as "no reference answer" while their graders read one — ContextBench.evaluate grades against task["answer"], and nocha's correct choice is the constant "A" in every instance by construction. An exemption's stated reason is itself a claim that rots: the sweep prints EXEMPT and nobody re-derives it, so the adapter stays unverified indefinitely.

Both are real recipes now, each verified to accept its reference answer and reject a wrong one. The sweep reports NOT-STAGED for them, which is true and actionable. nocha is the cheapest canary in the set and the one where the inverse half carries all the weight: with a constant expected answer, a positive-only check is satisfied by a grader hardwired to True.

Gates on the final head

gate	result
full suite (extras), frozen tree	10,382 passed, 140 skipped
CI posture (no tui)	10,387 passed
mypy, cold cache, both postures	736 modules, 0 errors
ruff (chimera/ scripts/ tests/)	clean
trademark scrubs	7/7
claims + repo hygiene	140 passed
canary sweep	7 pass · 0 BROKEN
CI on c60fa0fb	success (3.11/3.12/3.13, scrub, docs-sync)

Single failure in every suite run is test_validation_split::test_evaluate_returns_both_rates — documented in CLAUDE.md as env-sensitive locally and green in CI. Its test file, its subject (chimera/training/{spec,validation}.py), and chimera/function_synthesis/ are byte-identical to origin/master, so this diff cannot have caused it.