feat: inline LLM judge + V2 leaderboard 6-tab toggle + reproduce-leaderboard recipe - #196
Merged
Conversation
Three workstreams (sequenced low-risk first): - WS1: README + both HF data cards announce ClawBenchV1Trace + cross-link - WS2: claw-bench.com v1/v2 tab switcher (refactor server.py from hardcoded to data-driven, add v2 corpus tab) - WS3: deferred — v2 leaderboard, blocked on aggregating raw runs and evaluating more models against v2 corpus Open questions captured inline (Q1.1, Q1.2, Q2.1, Q2.2). Awaiting user approval before invoking writing-plans.
…tion - Replace single V1 success-rate table with 6 collapsible tabs by corpus × harness (V2 Hermes/OpenClaw/Codex/Claude Code, V1 Hermes/OpenClaw) matching the live claw-bench.com leaderboard - V2 Hermes table populated with 8 models judged by deepseek-v4-pro under lenient + strict rubrics (claude-opus-4-7 44.6% lenient anchor) - Add "Reproduce the leaderboard" section above ClawBench-Lite: one-shot vs two-stage flag examples for OpenRouter deepseek-v4-flash, plus the "skip the run and re-judge our published traces" path via TIGER-Lab/ClawBenchV2Trace - Reproduction success criterion: Intercept% / Reward (lenient) / Reward (strict) within ±2 pp of published row
- src/clawbench/runner/judge_llm.py: new lenient-rubric judge
("no explicit contradiction -> match", drop-in replacement for judge.py).
Adds Anthropic messages API support alongside openai-completions.
- scripts/clawbench_rescore.py: --rubric {lenient|strict|both}, default lenient.
Default judge model deepseek-v4-pro (matches the published leaderboard).
Emits per-task CSV (task_id + intercepted + match_lenient + match_strict
+ reasons) and summary.json (percentages formatted XX.X%) under
./eval_results/<batch>/ for easy inspection.
Backward-compat aliases preserved in rescore-summary.json
(n_judge_match -> n_match_strict, pass_rate_with_judge -> reward_pct_strict).
- scripts/clawbench_reproduce.py: one-shot CLI that pulls a model's V2 trace
from TIGER-Lab/ClawBenchV2Trace, re-judges with deepseek-v4-pro under both
rubrics, and diffs against the published leaderboard row (PASS within
+/- tolerance, default 2pp). Built-in baseline table for the 8 V2 hermes
rows currently on the site.
…ppers
- src/clawbench/eval/{rescore,reproduce}.py (moved from scripts/) are now
proper package modules; pyproject.toml exposes them as the
clawbench-rescore and clawbench-reproduce console scripts.
- scripts/{rescore,reproduce}.sh are thin shell wrappers that exec
`uv run python -m clawbench.eval.{rescore,reproduce}`.
- src/clawbench/runner/judge_llm.py stays the publishable core: drop-in
judge module with both openai-completions and anthropic-messages APIs.
- Drop the stale scripts/-path sys.path shim in rescore.py now that it
lives inside the installed package.
- Fix design doc owner attribution.
- Path A: re-run agent + re-judge (verifies full pipeline) - Path B: skip the run, re-judge our published traces (verifies judge alone) - One-shot `clawbench-reproduce --model X` equivalent for Path B - Pass criterion: each of Intercepted / Reward-lenient / Reward-strict within ±2 pp of published row
Collaborator
Author
|
@Perry2004 Please test this PR. |
V2 is the published / supported corpus across claw-bench.com, the README 6-tab leaderboard, the TIGER-Lab/ClawBenchV2Trace HF dataset, and the reproduce CLI's baseline table. Default the batch runner to v2 so any new user invoking `clawbench-batch` without --cases-suite lands on the same corpus that matches our scoring + traces. V1 still works (`--cases-suite v1`) and the V1 NAIL-Group/ClawBenchV1Trace dataset is unchanged.
…six harnesses - V2 is now the default across claw-bench.com, the repo, and the TIGER-Lab/ClawBenchV2Trace HF dataset; intercept-only design means no real charges + no irreversible actions on live sites. - Lenient judge rubric is published (judge_llm.py, --rubric lenient, the rubric that the public leaderboard uses). - Six harnesses are first-class: hermes / openclaw / codex / claude-code / claude-code-chrome-extension / browser-use. - `clawbench-reproduce --model deepseek-v4-flash` reproduces any published row within ±2 pp.
main added the claw-eval suite mapping after this branch diverged (commit 2e5ccbf, "feat: port some claw-eval tasks #72"). Without restoring it here, the merge to main would silently drop the suite alias from this branch's view of CASE_SUITES. No behavior change for the v2 default; just keeps the alias available for users invoking `clawbench-batch --cases-suite claw-eval`.
Adds the canonical scoring rubric. V2 traces now point to TIGER-Lab/ClawBenchV2Trace (the public mirror under the TIGER-Lab org) and the live leaderboard space links to TIGER-Lab/ClawBench instead of the legacy NAIL-Group placeholder. V1 trace dataset (NAIL-Group) stays unchanged — only V2 surface is rewritten.
…b/ClawBench The ICLR 2026 review-1 rebuttal references the live leaderboard twice. Both refs now use the canonical TIGER-Lab/ClawBench HF Space URL instead of the legacy NAIL-Group/clawbench-leaderboard placeholder so the camera-ready links resolve to the actual hosted leaderboard.
The hero SVG strapline reflected V1's 153/144 stats; with the V2 corpus as the default surface (per recent default-flip) the embedded banner is rewritten to V2's 130 tasks / 64 live websites so the README hero and the rendered project page agree with the README headline copy. README markdown headline copy is updated separately.
Mirrors CITATION.cff preferred-citation style. Keeps the author list and arXiv URL unchanged; just normalizes the entry type so downstream tools (BibLaTeX with biblatex-software, sourcebib resolvers) treat the entry as a journal preprint instead of a generic misc.
Resolves README News section conflict by keeping the 2026-05-20 V2-default entry (HEAD) plus main's 2026-05-16 through 2026-05-11 entries. Also fixes the 2026-05-11 V2Trace link to TIGER-Lab/ClawBenchV2Trace. batch.py auto-merged: V2 default + claw-eval restored together. pyproject.toml auto-merged: 0.3.3 + rescore/reproduce scripts.
… (limited-attention principle)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR: ships V2 as the default corpus (
--cases-suite v2), adds the lenientjudge_llmrubric, and theclawbench-reproduceCLI for re-scoring againstTIGER-Lab/ClawBenchV2Tracewith one OpenRouter key.Summary
deepseek/deepseek-v4-projudge on intercepted payload (Stage 2, lenientno-contradiction → matchrubric).Reproduce the leaderboardsection above ClawBench-Lite with two recipes: (a) run + judge in one shot, (b) skip the run and re-judge our published traces fromTIGER-Lab/ClawBenchV2Trace. Reproduction passes when Intercept% / Reward (lenient) / Reward (strict) land within ±2 pp of the published row.Test plan
uv run clawbench-batch --models deepseek/deepseek-v4-flash --cases-suite v2 --case-range 273-273 --harness hermes --no-judge(smoke test single task)uv run python scripts/clawbench_rescore.py <batch_dir> --judge deepseek/deepseek-v4-pro(verify rescore pipeline emitsjudge.jsonper task +rescore-summary.jsonper batch)gh pr viewafter merge — the rendered Markdown shows 6 collapsible tabs and Reproduce-the-leaderboard recipe