|
10 | 10 | [](https://github.com/reacher-z/ClawBench) |
11 | 11 | [](https://arxiv.org/abs/2604.08523) |
12 | 12 | [](https://huggingface.co/papers/2604.08523) |
13 | | -[](https://huggingface.co/datasets/NAIL-Group/ClawBench) |
| 13 | +[](https://huggingface.co/spaces/TIGER-Lab/ClawBench) |
14 | 14 | [](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace) |
15 | 15 | [](https://claw-bench.com) |
16 | 16 | [](https://github.com/reacher-z/ClawBench) |
@@ -122,7 +122,7 @@ order food, book travel, apply for jobs, write reviews, manage projects.<br/> |
122 | 122 | <td align="center" valign="top"> |
123 | 123 |
|
124 | 124 | 📦 **Download the data**<br/> |
125 | | -[`hf download NAIL-Group/ClawBench`](https://huggingface.co/datasets/NAIL-Group/ClawBench)<br/> |
| 125 | +[`hf download NAIL-Group/ClawBench`](https://huggingface.co/spaces/TIGER-Lab/ClawBench)<br/> |
126 | 126 | <sub>Tasks · rubrics · metadata</sub> |
127 | 127 |
|
128 | 128 | </td> |
@@ -178,11 +178,11 @@ ClawBench ships **three** Hugging Face datasets — task definitions plus full e |
178 | 178 | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | |
179 | 179 | | **[NAIL-Group/ClawBench](https://huggingface.co/datasets/NAIL-Group/ClawBench)** _(also mirrored at [TIGER-Lab/ClawBench](https://huggingface.co/datasets/TIGER-Lab/ClawBench))_ | Task definitions, rubrics, and metadata for V1 (153 tasks) and V2 (130 tasks) — what to attempt and how it's judged. | `hf download --repo-type dataset NAIL-Group/ClawBench` | |
180 | 180 | | **[NAIL-Group/ClawBenchV1Trace](https://huggingface.co/datasets/NAIL-Group/ClawBenchV1Trace)** | One directory per V1 model run, each with `recording.mp4`, `requests.jsonl`, `actions.jsonl`, `agent-messages.jsonl`, `interception.json`, and `run-meta.json` — everything we used to score the run. | `hf download --repo-type dataset NAIL-Group/ClawBenchV1Trace` | |
181 | | -| **[NAIL-Group/ClawBenchV2Trace](https://huggingface.co/datasets/NAIL-Group/ClawBenchV2Trace)** | Same 5-layer bundle for **V2** model runs. Rolling — new models added as they're evaluated. | `hf download --repo-type dataset NAIL-Group/ClawBenchV2Trace` | |
| 181 | +| **[TIGER-Lab/ClawBenchV2Trace](https://huggingface.co/datasets/TIGER-Lab/ClawBenchV2Trace)** | Same 5-layer bundle for **V2** model runs. Rolling — new models added as they're evaluated. | `hf download --repo-type dataset TIGER-Lab/ClawBenchV2Trace` | |
182 | 182 |
|
183 | 183 | > The trace datasets are large; use `hf download --include "<pattern>"` to pull a single model or a single task. |
184 | 184 |
|
185 | | -> **🏆 Live leaderboard:** [`claw-bench.com/leaderboard`](https://claw-bench.com/leaderboard) (V2 default, two-stage scoring — interception + LLM judge). Full scoring formula in [`eval/scoring.md`](eval/scoring.md). Add your run: PR to [`leaderboard/results.csv`](https://huggingface.co/datasets/NAIL-Group/ClawBench/blob/main/leaderboard/results.csv). |
| 185 | +> **🏆 Live leaderboard:** [`claw-bench.com/leaderboard`](https://claw-bench.com/leaderboard) (V2 default, two-stage scoring — interception + LLM judge). Full scoring formula in [`eval/scoring.md`](eval/scoring.md). Add your run: PR to the current [`leaderboard/results.csv`](https://huggingface.co/datasets/TIGER-Lab/ClawBench/blob/main/leaderboard/results.csv) source. |
186 | 186 |
|
187 | 187 | ## How It Works |
188 | 188 |
|
@@ -215,8 +215,8 @@ uv tool install clawbench-eval |
215 | 215 | ``` |
216 | 216 |
|
217 | 217 | You can also use `pipx install clawbench-eval` or `python -m pip install clawbench-eval`. |
218 | | -The installed commands are still `clawbench`, `clawbench-run`, and |
219 | | -`clawbench-batch`. |
| 218 | +The installed commands are still `clawbench`, `clawbench-run`, |
| 219 | +`clawbench-batch`, and `clawbench-harbor-adapt`. |
220 | 220 |
|
221 | 221 | For those want more granular control and contribution, clone the repo and run the root `uv` package entrypoint: |
222 | 222 |
|
@@ -326,6 +326,78 @@ Open the noVNC URL the script prints, complete the task by hand, then close the |
326 | 326 |
|
327 | 327 | **(d) Pair with an external browser agent** — run in Human mode, open the noVNC URL, and let an external browser agent control that browser session while ClawBench records and intercepts it. |
328 | 328 |
|
| 329 | +**(e) Run V2 through Harbor Framework** — convert the V2 cases into a local Harbor dataset, then let Harbor start the ClawBench browser runtime and connect its agent over CDP. |
| 330 | + |
| 331 | +Harbor runs use Harbor's Docker provider, so make sure Docker is available even if you normally use Podman for native ClawBench runs. |
| 332 | + |
| 333 | +```bash |
| 334 | +# Convert all V2 tasks into Harbor-compatible task directories. |
| 335 | +uv run clawbench-harbor-adapt \ |
| 336 | + --output-dir ./harbor-datasets/clawbench-v2 \ |
| 337 | + --overwrite |
| 338 | + |
| 339 | +# Optional smoke dataset with one generated task. |
| 340 | +uv run clawbench-harbor-adapt \ |
| 341 | + --output-dir ./harbor-datasets/clawbench-v2-smoke \ |
| 342 | + --limit 1 \ |
| 343 | + --overwrite |
| 344 | + |
| 345 | +# Configure the verifier judge used for Harbor rewards. |
| 346 | +export CLAWBENCH_JUDGE_BASE_URL="https://your-judge-provider.example/v1" |
| 347 | +export CLAWBENCH_JUDGE_API_KEY="your-judge-api-key" |
| 348 | +export CLAWBENCH_JUDGE_MODEL="deepseek-v4-pro" |
| 349 | +export CLAWBENCH_JUDGE_API_TYPE="openai-completions" |
| 350 | + |
| 351 | +# Run with Harbor. Use "harbor run" directly if Harbor is already installed. |
| 352 | +uvx --from harbor==0.15.0 harbor run \ |
| 353 | + -p ./harbor-datasets/clawbench-v2 \ |
| 354 | + -a "<agent>" \ |
| 355 | + -m "<model>" \ |
| 356 | + --env-file .env \ |
| 357 | + --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \ |
| 358 | + --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \ |
| 359 | + --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \ |
| 360 | + --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}" |
| 361 | +``` |
| 362 | + |
| 363 | +The generated Harbor environment contains Chromium, the ClawBench recorder/interceptor, noVNC, and runtime helper scripts, but no ClawBench-native harness. Harbor installs/runs the selected agent from `-a` inside the task container. |
| 364 | + |
| 365 | +PurelyMail credentials come from `.env` via `--env-file .env`. Scoring requires an intercepted request and a judge match; pass judge credentials to Harbor's verifier with `--ve CLAWBENCH_JUDGE_*`. If the judge base URL or API key is missing, intercepted tasks receive reward `0` with `missing judge configuration`. |
| 366 | + |
| 367 | +Concrete agent examples: |
| 368 | + |
| 369 | +```bash |
| 370 | +# OpenClaw through OpenRouter's OpenAI-compatible endpoint. |
| 371 | +export OPENAI_BASE_URL="https://openrouter.ai/api/v1" |
| 372 | +export OPENAI_API_KEY="$OPENROUTER_API_KEY" |
| 373 | + |
| 374 | +uvx --from harbor==0.15.0 harbor run \ |
| 375 | + -p ./harbor-datasets/clawbench-v2 \ |
| 376 | + -a openclaw \ |
| 377 | + -m openai/deepseek/deepseek-v4-flash \ |
| 378 | + --ak thinking=off \ |
| 379 | + --env-file .env \ |
| 380 | + --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \ |
| 381 | + --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \ |
| 382 | + --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \ |
| 383 | + --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}" \ |
| 384 | + --jobs-dir ./harbor-jobs/openclaw-deepseek-flash |
| 385 | + |
| 386 | +# Hermes through OpenRouter. |
| 387 | +export OPENROUTER_API_KEY="your-openrouter-key" |
| 388 | + |
| 389 | +uvx --from harbor==0.15.0 harbor run \ |
| 390 | + -p ./harbor-datasets/clawbench-v2 \ |
| 391 | + -a hermes \ |
| 392 | + -m deepseek/deepseek-v4-flash \ |
| 393 | + --env-file .env \ |
| 394 | + --ve CLAWBENCH_JUDGE_BASE_URL="$CLAWBENCH_JUDGE_BASE_URL" \ |
| 395 | + --ve CLAWBENCH_JUDGE_API_KEY="$CLAWBENCH_JUDGE_API_KEY" \ |
| 396 | + --ve CLAWBENCH_JUDGE_MODEL="${CLAWBENCH_JUDGE_MODEL:-deepseek-v4-pro}" \ |
| 397 | + --ve CLAWBENCH_JUDGE_API_TYPE="${CLAWBENCH_JUDGE_API_TYPE:-openai-completions}" \ |
| 398 | + --jobs-dir ./harbor-jobs/hermes-deepseek-flash |
| 399 | +``` |
| 400 | + |
329 | 401 | <details> |
330 | 402 | <summary><b>Develop from source</b> — clone + ``./run.sh`` for contributors</summary> |
331 | 403 |
|
@@ -583,23 +655,20 @@ ClawBench's niche: **live consumer websites, everyday tasks, end-to-end recordin |
583 | 655 | ┌─────────────────────────────────────────────────┐ |
584 | 656 | │ Container (Docker / Podman) │ |
585 | 657 | │ │ |
586 | | -│ ┌───────────┐ DOM events ┌──────────────┐ │ |
587 | | -│ │ content.js├──────────────►│ background.js│ │ |
588 | | -│ │ (per tab) │ │ (service │ │ |
589 | | -│ └───────────┘ │ worker) │ │ |
590 | | -│ └──┬──────┬────┘ │ |
591 | | -│ │ │ │ |
592 | | -│ actions │ │ screenshots |
593 | | -│ │ │ │ |
594 | | -│ ┌──────────┐ ┌──────▼──────▼────┐ │ |
| 658 | +│ ┌──────────┐ CDP Fetch/Runtime/Page events │ |
| 659 | +│ │ Chromium ├─────────────────────────────┐ │ |
| 660 | +│ │ :9222 CDP│ │ │ |
| 661 | +│ └──────────┘ │ │ |
| 662 | +│ │ │ |
| 663 | +│ ┌──────────┐ ┌────────────────▼─┐ │ |
595 | 664 | │ │ Xvfb │◄──ffmpeg──►│ FastAPI Server │ │ |
596 | 665 | │ │ :99 │ x11grab │ :7878 │ │ |
597 | 666 | │ └──────────┘ └──────────────────┘ │ |
598 | 667 | │ │ │ |
599 | | -│ ┌──────────┐ ┌───────▼─────────┐ │ |
600 | | -│ │ Chromium │ │ /data │ │ |
601 | | -│ │ :9222 CDP│ │ actions.jsonl │ │ |
602 | | -│ └──────────┘ │ requests.jsonl │ │ |
| 668 | +│ ┌───────▼─────────┐ │ |
| 669 | +│ │ /data │ │ |
| 670 | +│ │ actions.jsonl │ │ |
| 671 | +│ │ requests.jsonl │ │ |
603 | 672 | │ │ screenshots/ │ │ |
604 | 673 | │ │ recording.mp4 │ │ |
605 | 674 | │ └─────────────────┘ │ |
@@ -636,6 +705,18 @@ uv run clawbench-batch --models claude-sonnet-4-6 --cases-suite claw-eval --all- |
636 | 705 |
|
637 | 706 | # Batch a custom case directory: |
638 | 707 | uv run clawbench-batch --models claude-sonnet-4-6 --cases-dir custom-cases --all-cases |
| 708 | + |
| 709 | +# Convert V2 tasks into a local Harbor dataset: |
| 710 | +uv run clawbench-harbor-adapt --output-dir ./harbor-datasets/clawbench-v2 --overwrite |
| 711 | + |
| 712 | +# Run the generated Harbor dataset: |
| 713 | +uvx --from harbor==0.15.0 harbor run -p ./harbor-datasets/clawbench-v2 -a "<agent>" -m "<model>" --env-file .env |
| 714 | + |
| 715 | +# Examples: |
| 716 | +# openclaw via OpenRouter/OpenAI-compatible API: |
| 717 | +# -a openclaw -m openai/deepseek/deepseek-v4-flash --ak thinking=off |
| 718 | +# hermes via OpenRouter: |
| 719 | +# -a hermes -m deepseek/deepseek-v4-flash |
639 | 720 | ``` |
640 | 721 |
|
641 | 722 | V1 tasks are in [`test-cases/v1/`](test-cases/v1/) (153 tasks). V2 tasks are in `test-cases/v2/` (130 tasks), Lite is in `test-cases/v1-lite/` (20 tasks), and converted Claw-Eval tasks live in `test-cases/claw-eval/` (19 tasks). All suites use [`test-cases/task.schema.json`](test-cases/task.schema.json). For test case authoring details, see [CONTRIBUTING.md](CONTRIBUTING.md). For output structure and evaluation guidance, see [eval/README.md](eval/README.md). |
|
0 commit comments