fix(evals): stop the judge grading mixed runners or skipping a grown condition set

[Matthew-Selvam]

Summary

scripts/judge.py silently graded the wrong thing in two ways. Neither produced an error, a warning, or a non-zero exit.

1. A responses file with two runners was graded as one. group_responses() keyed groups on (case_id, trial) alone and assigned into a dict:

groups[(row["case_id"], row["trial"])][row["condition"]] = row["response"]

run_evals.py keys its own resume on (case_id, trial, condition, runner), so a file holding rows from two runner invocations is a shape the harness produces by design. The later runner's row overwrote the earlier one, and every condition was then scored from whichever runner happened to be written last — reported under the conditions of the file, with no indication that half the input was discarded.

Before (4 rows in, 2 retained; the claude rows are gone):

{'baseline': 'baseline-from-codex', 'candidate': 'candidate-from-codex'}

2. A group judged under a narrower --conditions was skipped forever. judged was a set of (case_id, trial), so --conditions widening between passes changed nothing:

$ python3 scripts/judge.py --responses ... --conditions baseline candidate      # exit 0
$ python3 scripts/judge.py --responses ... --conditions baseline candidate comparator
skip judged direct-answer/trial 1
exit=0
conditions scored: ['baseline', 'candidate']   # comparator never graded

Judging baseline+candidate and appending comparator responses afterwards is the documented flow, because run_evals.py writes one condition per invocation. evals/README.md advertises the comparator as a supported third condition, and run_evals.py score then gates on the two conditions that happen to be present while the operator believes three were measured.

Observable behavior

	before	after
Two runners, same (case, trial, condition)	later row wins, both passages look fine, exit 0	ValueError naming the case, trial, and condition
Rerun with a wider --conditions after a narrower pass	"skip judged", missing condition ungraded, exit 0	ValueError naming the group and the conditions it lacks
Rerun with unchanged --conditions	"skip judged"	"skip judged" (unchanged)

Why the second case raises instead of re-judging

The scorer rejects duplicate rows for one (case, trial, condition) (_check_pairing in run_evals.py), so re-writing the earlier conditions would fail later anyway. Skipping is what lost the condition without a word, so the remaining honest option is to refuse and name the two ways forward. This is a behavior change to a documented flow: the error text says to point --output at a fresh file, or remove the rows first.

Authorship and provenance — select exactly one

• Human-authored — substantive implementation and text were produced by a human.
• Autonomous agent-authored — an agent planned and produced most of the substantive change.
• Hybrid — a human and one or more agents both made substantive contributions.

Agent/tool and model/version: Hermes Agent (deepseek-v4.1-flash) ran the review and wrote the fix and tests. Four parallel review subagents (same tool) each audited one subsystem; the two defects above came from the evals-harness reviewer, were then reproduced independently by the submitting agent before any code was changed. Model/version of the subagents was not recorded by the harness.

Agent contribution: Identified both defects, reproduced each against main before touching code, wrote the fix, wrote four regression tests, and verified by mutation that each test fails against the pre-fix source.

Human verification: The submitting human (Matthew-Selvam) reviewed the complete diff, read scripts/judge.py and tests/test_judge.py in full, and ran the commands in the Verification section below. The reproduction transcripts above were produced by the agent and re-run by the human.

Known limitations or uncertain results: The duplicate-runner defect was reproduced with synthetic rows; no real two-runner responses.jsonl exists in the repo to test against. Raised rather than fixed by adding the runner to the group key, because grouping by runner would silently halve the compared set instead of reporting the mixed file — a maintainer may prefer the other repair. scripts/judge.py --retries still bills retries that a failed group discards from the reported total (the harness's own comment says the ledger feeds budget decisions); that is a separate defect and is not fixed here.

Labels

Target label: Target:Evals

Author label: Author:Hybrid

Workflow labels: bug

Safety and side effects

• The change does not access or expose secrets, private files, or unrelated user/repository data.
• Scripts, hooks, workflows, and evals are bounded and do not create surprising or irreversible side effects.
• No destructive, privileged, production, externally visible, or persistent action occurs without explicit user intent and appropriate safeguards.
• Network access, third-party code, permissions, and provider costs are minimized and documented.
• Prompt text, examples, and fixtures contain no hidden instructions that weaken safety or expand agent authority.

Side effects, permissions, network access, and cost: None. No new files, no network, no paid model calls: the tests use sh -c stub runners on temporary files, matching the existing pattern in tests/test_judge.py. The change only turns two silent-wrong-result paths into errors.

Compatibility

• This is not a breaking change.
• This is a breaking change; it was discussed, and migration/deprecation documentation is included below.
• Canonical and mirrored skill files are synchronized when applicable.
• Relevant platform manifests and installation documentation were reviewed.

Migration or rollback notes: Rolling back restores the silent behavior; no data is modified by the change. The only operator-visible difference is that a rerun with widened --conditions over an existing --output now stops with an actionable message instead of writing nothing and reporting success. The unchanged --conditions resume path keeps its existing "skip judged" output and exit 0, verified by test.

skills/i-have-adhd/SKILL.md and its .cursor mirror are untouched, so the sync check is unaffected.

Verification

• python3 -m unittest discover -s tests — 46 tests, OK (baseline origin/main is 43; 3 new: duplicate-condition rejection, the single-runner control case, and the end-to-end widened---conditions guard)
• python3 scripts/run_evals.py validate — Evaluation cases are valid.
• git diff --check main..HEAD — clean
• Mutation check (fix 1 reverted in a scratch clone, tests kept): test_two_responses_for_one_condition_are_rejected_not_overwritten → FAILED, confirming the test detects the defect rather than passing vacuously
• Mutation check (fix 2 reverted the same way): test_a_group_judged_under_narrower_conditions_is_not_silently_skipped → FAILED
• Reproduction on unmodified main, before the fix, with synthetic two-runner input: 4 rows in → 2 retained, claude responses dropped (transcript above)
• Reproduction on unmodified main, before the fix: second pass with --conditions baseline candidate comparator printed skip judged direct-answer/trial 1 and exited 0

Behavior evals: Not run. This changes the eval tooling itself, not the ruleset, so a baseline/candidate comparison of the skill would not measure it; the unit tests above exercise the changed code paths directly. No model calls were made and no cost was incurred.

Final accountability

• I reviewed the complete diff, removed unrelated generated changes, and take responsibility for the submitted content.
• All failed, skipped, or unrun checks are disclosed above.