Skip to content

Update: ClawBench entry β€” V2 corpus (283 tasks, 163 platforms) - #60

Merged
nibzard merged 1 commit into
steel-dev:mainfrom
reacher-z:update-clawbench-v2
May 26, 2026
Merged

Update: ClawBench entry β€” V2 corpus (283 tasks, 163 platforms)#60
nibzard merged 1 commit into
steel-dev:mainfrom
reacher-z:update-clawbench-v2

Conversation

@reacher-z

Copy link
Copy Markdown
Contributor

ClawBench is already in the Benchmarks & Research section, but the blurb
is from the V1 release. This PR refreshes it to reflect the V2 corpus
that shipped last week.

What changed since the entry was added:

  • Task count: 153 β†’ 283 (V1 153 + V2 130)
  • Platforms covered: 144 β†’ 163
  • Scoring: was just HTTP-request interception; now a two-stage
    pipeline
    β€” Stage 1 final-request interception, Stage 2 LLM-judge
    reviews the intercepted body. This addresses the "agent reached the
    endpoint but submitted the wrong thing" failure mode.
  • Public leaderboard at https://claw-bench.com with 6 tabs by corpus Γ—
    harness pairing.
  • All eval runs come with a 5-layer execution-trace bundle (recording,
    action log, request log, agent messages, interception JSON) β€” useful
    for anyone who wants to audit, debug, or build on top of the runs.

No other entries touched.

@nibzard
nibzard merged commit 1ecc392 into steel-dev:main May 26, 2026
1 of 3 checks passed
@nibzard

nibzard commented May 26, 2026

Copy link
Copy Markdown
Contributor

Merged β€” thanks for the V2 update! Two tiny follow-ups whenever you have a minute (no rush, happy to take them as a separate PR):

  1. The repo now lives at TIGER-AI-Lab/ClawBench β€” the old reacher-z/ClawBench URL still redirects, but updating the README link (and the stars badge path) would make it canonical.
  2. Small mismatch between the entry ("163 platforms") and the upstream README ("144 websites"). Whichever number is right, would be great to have them agree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants