|
| 1 | +# ClawBench — Scoring Logic |
| 2 | + |
| 3 | +This document specifies how a ClawBench run is scored. It is the canonical reference for the numbers shown on: |
| 4 | + |
| 5 | +- **Live leaderboard:** https://huggingface.co/spaces/TIGER-Lab/ClawBench |
| 6 | +- **Website snapshot:** https://claw-bench.com/leaderboard |
| 7 | +- **HF data card table:** https://huggingface.co/datasets/NAIL-Group/ClawBench |
| 8 | + |
| 9 | +Anyone can reproduce every number on the leaderboard from the public traces in [`NAIL-Group/ClawBenchV1Trace`](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace) and [`TIGER-Lab/ClawBenchV2Trace`](https://huggingface.co/datasets/TIGER-Lab/ClawBenchV2Trace) by running `scripts/clawbench_rescore.py` (see [Reproducibility](#reproducibility) below). |
| 10 | + |
| 11 | +## Summary |
| 12 | + |
| 13 | +Scoring is two stages applied in order. Each stage produces a boolean per (task × run). |
| 14 | + |
| 15 | +``` |
| 16 | + ┌──────────────┐ ┌──────────────┐ |
| 17 | +agent run ───► │ Interception │ ──true─►│ LLM judge │ ──true─► reward = 1 |
| 18 | + └──────────────┘ └──────────────┘ |
| 19 | + │ false │ false |
| 20 | + └─► reward = 0 ◄────────┘ |
| 21 | +``` |
| 22 | + |
| 23 | +Aggregate metrics (per model × corpus): |
| 24 | + |
| 25 | +```text |
| 26 | +intercepted_rate = sum(intercepted) / N |
| 27 | +reward_rate = sum(intercepted ∧ judge_match) / N |
| 28 | +``` |
| 29 | + |
| 30 | +`N` = number of tasks in the corpus (V1: 153, V2: 130). |
| 31 | + |
| 32 | +## Stage 1 — Final-request interception |
| 33 | + |
| 34 | +A **request interceptor** runs inside the sandbox container. It blocks the *final* outgoing HTTP request whose URL and method match the task's `eval_schema`. The intent is to capture the agent's commit-intent (checkout, form submit, post, etc.) **before** it actually hits the live website — both for evaluation and for safety. |
| 35 | + |
| 36 | +Per-task interceptor config lives in `test-cases/<corpus>/<slug>/task.json`: |
| 37 | + |
| 38 | +```jsonc |
| 39 | +{ |
| 40 | + "eval_schema": { |
| 41 | + "url_pattern": "myrecipes\\.com/api/v\\d+/review/save", |
| 42 | + "method": "POST" |
| 43 | + } |
| 44 | +} |
| 45 | +``` |
| 46 | + |
| 47 | +A run is **intercepted** iff the agent's final outgoing request matched the URL regex *and* the HTTP method. The result lives in `data/interception.json`: |
| 48 | + |
| 49 | +```jsonc |
| 50 | +{ |
| 51 | + "intercepted": true, |
| 52 | + "url": "https://www.myrecipes.com/api/v2/review/save", |
| 53 | + "method": "POST", |
| 54 | + "body": {"rating": 4, "tip": "Add a pinch of salt for balance.", "recipe_id": 12345} |
| 55 | +} |
| 56 | +``` |
| 57 | + |
| 58 | +If `intercepted: false`, **reward = 0** for that task regardless of how close the agent got. Common reasons: |
| 59 | + |
| 60 | +- agent timed out (`time_limit_exceeded` in `run-meta.json`) |
| 61 | +- agent gave up before reaching the terminal step |
| 62 | +- agent reached a different endpoint than the rubric expects (alternate-flow miss) |
| 63 | +- agent hit a CAPTCHA / login / verification wall and could not solve it |
| 64 | + |
| 65 | +Note that **Stage 1 alone is too lenient** — an agent that intercepts but submits the wrong payload would pass. Stage 2 closes that gap. |
| 66 | + |
| 67 | +## Stage 2 — LLM judge |
| 68 | + |
| 69 | +For every run where `intercepted: true`, an LLM judge reviews the intercepted body against the natural-language instruction. The judge returns one of: |
| 70 | + |
| 71 | +- `match: true` — body satisfies the instruction |
| 72 | +- `match: false` — body does not satisfy |
| 73 | +- `match: null` — judge could not decide (rare; treated as `false` in aggregate) |
| 74 | + |
| 75 | +The judge is configured per run; default is `deepseek-v4-pro` at temperature 0. Output lives in `data/judge.json`: |
| 76 | + |
| 77 | +```jsonc |
| 78 | +{ |
| 79 | + "match": true, |
| 80 | + "reason": "The POST body submits rating=4 with a non-trivial tip mentioning 'add a pinch of salt', which fulfills the instruction to rate 4 stars and add a helpful tip.", |
| 81 | + "judge_model": "deepseek-v4-pro", |
| 82 | + "raw": "<full JSON response from the model>", |
| 83 | + "error": null |
| 84 | +} |
| 85 | +``` |
| 86 | + |
| 87 | +### Judge prompt (verbatim) |
| 88 | + |
| 89 | +The judge sees the task instruction and the intercepted body (truncated to 4 KB for headers + body), and is asked to return strict JSON with `match` (bool) and `reason` (≤ 300 chars). The exact prompt is in [`src/clawbench/runner/judge.py`](../src/clawbench/runner/judge.py) — search for `SYSTEM_PROMPT`. |
| 90 | + |
| 91 | +System prompt sketch: |
| 92 | + |
| 93 | +> You are evaluating whether an AI agent's final HTTP request fulfills a natural-language instruction. You will be given the instruction and the request body. Return strict JSON `{"match": boolean, "reason": "≤ 300 chars"}`. Be strict: a body that matches on form but not on the user's intent should be `match: false`. |
| 94 | +
|
| 95 | +User message contains the `instruction` and a compact JSON dump of `(url, method, body)`. Headers are stripped of authentication. Payload is truncated past 4 KB. |
| 96 | + |
| 97 | +### Final score |
| 98 | + |
| 99 | +```python |
| 100 | +final_pass = intercepted AND (judge_match is True) |
| 101 | +``` |
| 102 | + |
| 103 | +Per-run record in `run-meta.json` gets: |
| 104 | + |
| 105 | +```jsonc |
| 106 | +{ |
| 107 | + "intercepted": true, |
| 108 | + "judge_match": true, |
| 109 | + "final_pass": true, |
| 110 | + "result_category": "passed" |
| 111 | +} |
| 112 | +``` |
| 113 | + |
| 114 | +## Why two stages |
| 115 | + |
| 116 | +| Stage | What it catches | What it misses | |
| 117 | +|---|---|---| |
| 118 | +| Interception only | "agent reached the right endpoint" | wrong payload submitted | |
| 119 | +| Judge only (no interception gate) | "agent's reasoning was right" | agent never actually completed the action | |
| 120 | +| **Both** | **end-to-end task completion with correct payload** | very edge-case: agent intercepts a syntactically-equivalent endpoint not in the regex | |
| 121 | + |
| 122 | +Empirically, requiring both moves headline scores down sharply (typical Stage-1-only is 1.5–2× Stage-2 numbers), surfacing models that "almost get there" vs. models that actually complete the task. The two-stage system also makes failure diagnosis cheap — the run-meta tells you which stage cut off. |
| 123 | + |
| 124 | +## Aggregating to a leaderboard row |
| 125 | + |
| 126 | +Each (model × harness × corpus) batch produces one `rescore-summary.json`: |
| 127 | + |
| 128 | +```jsonc |
| 129 | +{ |
| 130 | + "batch_dir": "/path/to/batch", |
| 131 | + "n_total": 130, |
| 132 | + "n_intercepted": 63, |
| 133 | + "n_judge_match": 24, |
| 134 | + "n_judge_mismatch": 34, |
| 135 | + "n_judge_error": 5, |
| 136 | + "pass_rate_stage1_only": 0.4846, |
| 137 | + "pass_rate_with_judge": 0.1846, |
| 138 | + "tasks": [ ... per-task records ... ] |
| 139 | +} |
| 140 | +``` |
| 141 | + |
| 142 | +The leaderboard row is one row per batch, with columns from the script's output: |
| 143 | + |
| 144 | +```csv |
| 145 | +model,harness,dataset,passed,total,pass_rate,wall_hours |
| 146 | +glm-5.1,hermes,v2,24,130,18.46,11.35 |
| 147 | +``` |
| 148 | + |
| 149 | +## Reproducibility |
| 150 | + |
| 151 | +To re-grade an existing trace bundle (no agent re-run required): |
| 152 | + |
| 153 | +```bash |
| 154 | +# 1. install the package |
| 155 | +pip install clawbench-eval |
| 156 | + |
| 157 | +# 2. download the trace bundle for the model you want to re-score |
| 158 | +hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace \ |
| 159 | + --include "*-claude-sonnet-4-6-*" \ |
| 160 | + --local-dir ./v2-traces |
| 161 | + |
| 162 | +# 3. set your judge model's API key in env |
| 163 | +export DEEPSEEK_API_KEY=sk-... |
| 164 | + |
| 165 | +# 4. (one-time) add the judge model to models.yaml — see docs/models.md |
| 166 | + |
| 167 | +# 5. rescore |
| 168 | +python scripts/clawbench_rescore.py \ |
| 169 | + --judge-model deepseek-v4-pro \ |
| 170 | + --only-batch ./v2-traces \ |
| 171 | + --force # re-judge existing judge.json files |
| 172 | +``` |
| 173 | + |
| 174 | +Output: per-run `judge.json` updated in place, plus a fresh `rescore-summary.json` at the batch root. |
| 175 | + |
| 176 | +## Common questions |
| 177 | + |
| 178 | +- **Why DeepSeek instead of Claude / GPT?** Open weights (closer to reproducible) and substantially cheaper for what we need. Swap with `--judge-model <other>` if you want — see `docs/models.md` for setting one up. |
| 179 | +- **Does the judge see the screenshot?** No, by design. The judge sees the intercepted HTTP request + instruction only. Visual judgment lives in a separate (out-of-scope, future) stage. |
| 180 | +- **What if interception fires before the agent has finished?** The interceptor only fires on requests matching `eval_schema.url_pattern` *and* `method`. Setting this regex correctly is a per-task curation responsibility; mistakes are caught in human review (see `docs/contributing/adding-a-task.md`). |
| 181 | +- **Why is `n_judge_error > 0`?** Network blips, rate limits, the judge returning non-JSON. In aggregate we treat these as `match: false` (no credit). Persistent errors flag a config bug. |
| 182 | +- **Where does the `33.3%` headline number come from?** Sonnet 4.6 on V1: `n_intercepted=51, n_judge_match=51, n_total=153`. Stage 1 + Stage 2 collapse onto the same number because Sonnet's intercepted payloads almost always match the instruction on V1. |
| 183 | + |
| 184 | +## See also |
| 185 | + |
| 186 | +- [`src/clawbench/runner/judge.py`](../src/clawbench/runner/judge.py) — the judge implementation (~250 lines). |
| 187 | +- [`scripts/clawbench_rescore.py`](../scripts/clawbench_rescore.py) — the rescoring CLI. |
| 188 | +- [`test-cases/task.schema.json`](../test-cases/task.schema.json) — `eval_schema` field definition. |
| 189 | +- [Trace dataset (V1)](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace) — every layer of every V1 run. |
| 190 | +- [Trace dataset (V2)](https://huggingface.co/datasets/TIGER-Lab/ClawBenchV2Trace) — V2 traces (rolling, as new models are evaluated). |
0 commit comments