When the judge provider is down, we record the model as having failed the task.
src/clawbench/runner/judge.py:247-253 returns match=None on judge_call_failed (HTTP 402/timeout after retries).
src/clawbench/runner/run.py:657-664 marks pass only when match is True; run.py:743-754 then exits 1.
src/clawbench/runner/batch.py:324-327 counts exit 1 as failed.
So a judge outage is indistinguishable in batch-summary.json from genuine agent failure. This is not hypothetical — a judge account hitting HTTP 402 mid-sweep has already produced whole batches recorded as failures that needed a manual rescore pass to recover.
Ask:
- Give
match=None a distinct outcome (e.g. exit code 3 / status: "judge_inconclusive"), separate from agent failure.
- Count it separately in
batch-summary.json totals so a reader can see "N runs unjudged" instead of silently deflated reward.
- Have
--resume / clawbench-rescore target exactly those runs for re-judging.
This pairs with #243 (always report Stage-1 and Stage-2 separately): both are about not letting stage-2 problems silently corrupt the headline number.
When the judge provider is down, we record the model as having failed the task.
src/clawbench/runner/judge.py:247-253returnsmatch=Noneonjudge_call_failed(HTTP 402/timeout after retries).src/clawbench/runner/run.py:657-664markspassonly when match isTrue;run.py:743-754then exits 1.src/clawbench/runner/batch.py:324-327counts exit 1 asfailed.So a judge outage is indistinguishable in
batch-summary.jsonfrom genuine agent failure. This is not hypothetical — a judge account hitting HTTP 402 mid-sweep has already produced whole batches recorded as failures that needed a manual rescore pass to recover.Ask:
match=Nonea distinct outcome (e.g. exit code 3 /status: "judge_inconclusive"), separate from agent failure.batch-summary.jsontotals so a reader can see "N runs unjudged" instead of silently deflated reward.--resume/clawbench-rescoretarget exactly those runs for re-judging.This pairs with #243 (always report Stage-1 and Stage-2 separately): both are about not letting stage-2 problems silently corrupt the headline number.