Skip to content

Judge outages are scored as agent failures (match=None → exit 1 → 'failed') #299

Description

@reacher-z

When the judge provider is down, we record the model as having failed the task.

  • src/clawbench/runner/judge.py:247-253 returns match=None on judge_call_failed (HTTP 402/timeout after retries).
  • src/clawbench/runner/run.py:657-664 marks pass only when match is True; run.py:743-754 then exits 1.
  • src/clawbench/runner/batch.py:324-327 counts exit 1 as failed.

So a judge outage is indistinguishable in batch-summary.json from genuine agent failure. This is not hypothetical — a judge account hitting HTTP 402 mid-sweep has already produced whole batches recorded as failures that needed a manual rescore pass to recover.

Ask:

  1. Give match=None a distinct outcome (e.g. exit code 3 / status: "judge_inconclusive"), separate from agent failure.
  2. Count it separately in batch-summary.json totals so a reader can see "N runs unjudged" instead of silently deflated reward.
  3. Have --resume / clawbench-rescore target exactly those runs for re-judging.

This pairs with #243 (always report Stage-1 and Stage-2 separately): both are about not letting stage-2 problems silently corrupt the headline number.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions