1- <p align =" center " >
2- <img src =" site/public/favicon.svg " width =" 110 " alt =" PawBench logo " >
3- </p >
4-
5- <h1 align =" center " >PawBench</h1 >
1+ <h1 align =" center " >🐾 PawBench</h1 >
62
73<p align =" center " >
84 <a href =" README.md " ><strong >English</strong ></a > ·
95 <a href =" README.zh-CN.md " >简体中文</a >
106</p >
117
128<p align =" center " >
13- <img alt =" tasks " src =" https://img.shields.io/badge/tasks-150-2ea44f " >
14- <img alt =" models " src =" https://img.shields.io/badge/models-9-0969da " >
15- <img alt =" harnesses " src =" https://img.shields.io/badge/harnesses-3-8250df " >
9+ <a href =" #tasks " >
10+ <img alt="tasks" src="https://img.shields.io/badge/tasks-150-2ea44f">
11+ </a >
12+ <a href =" https://agentscope-ai.github.io/PawBench/ " >
13+ <img alt="models" src="https://img.shields.io/badge/models-9-0969da">
14+ </a >
15+ <a href =" #harnesses " >
16+ <img alt="harnesses" src="https://img.shields.io/badge/harnesses-3-8250df">
17+ </a >
1618 <a href =" https://agentscope-ai.github.io/PawBench/ " >
1719 <img alt="leaderboard" src="https://img.shields.io/badge/leaderboard-live-cf222e">
1820 </a >
19- <a href =" https://github.com/agentscope-ai/pawbench " >
21+ <a href =" https://github.com/agentscope-ai/PawBench " >
2022 <img alt="GitHub repo" src="https://img.shields.io/badge/GitHub-pawbench-24292f">
2123 </a >
2224</p >
2325
2426<p align =" center " >
25- <strong >A benchmark for evaluating foundation models and the harnesses that run them .</strong ><br >
26- 150 agent tasks · 4,050 model-harness-task cells · score slices, transcripts, grader artifacts, and replayable runs.
27+ <strong >A benchmark for evaluating LLM × harness performance .</strong ><br >
28+ 150 agent tasks | 4,050 model-harness-task cells | score slices, transcripts, grader artifacts, and replayable runs.
2729</p >
2830
2931## Overview
@@ -32,11 +34,11 @@ The same model can behave very differently in different agent runtimes. When a t
3234
3335PawBench evaluates the model and the harness together:
3436
35- $$ \text{Agent Performance} = f(\text{Foundation Model }, \text{Harness}) $$
37+ $$ \text{Agent Performance} = f(\text{LLM }, \text{Harness}) $$
3638
3739v1.0 covers ** 9 models × 3 harnesses × 150 tasks** . The first run shows a ** 5.6-point** average gap between the strongest and weakest harnesses, comparable to some model upgrades. On ` qwen3.6-35b-a3b ` , switching only the harness moves the score by ** 11.5 points** .
3840
39- ![ PawBench leaderboard overview] ( site/public/pawbench-leaderboard-overview.png )
41+ ![ PawBench leaderboard overview] ( site/public/pawbench-leaderboard-overview.svg )
4042
4143## Quick Start
4244
@@ -117,14 +119,14 @@ PawBench follows a **Reuse & Tag** methodology. Instead of writing every task fr
117119| Modality | ` modality ` | ` text ` or ` multimodal ` (` image ` , ` audio ` , ` video ` ) |
118120| Environment | ` environment ` | ` closed ` (offline, reproducible) or ` open ` (live internet / SaaS APIs) |
119121
120- v1.0 contains ** 150 tasks** from ` claweval ` , ` qwenclawbench ` , ` pinchbench ` , ` qwenpawbench ` , ` skillsbench ` , and ` wildclawbench ` .
122+ v1.0 contains ** 150 tasks** from ` claweval ` , ` qwenclawbench ` , ` pinchbench ` , PawBench self-built tasks , ` skillsbench ` , and ` wildclawbench ` .
121123
122124| Source | # | Main coverage |
123125| :--- | ---: | :--- |
124126| [ ` claweval ` ] ( https://github.com/claw-eval/claw-eval ) | 52 | Office productivity, data analytics, content creation |
125127| [ ` qwenclawbench ` ] ( https://github.com/SKYLENAGE-AI/QwenClawBench ) | 29 | Automation, software engineering, safety alignment |
126- | ` pinchbench ` | 23 | Office workflows, software engineering, information retrieval |
127- | ` qwenpawbench ` | 21 | Automation , information retrieval, safety alignment |
128+ | [ ` pinchbench ` ] ( https://github.com/pinchbench/skill ) | 23 | Office workflows, software engineering, information retrieval |
129+ | PawBench | 21 | Self-built tasks covering automation , information retrieval, and safety alignment |
128130| [ ` skillsbench ` ] ( https://github.com/benchflow-ai/skillsbench ) | 15 | Long-horizon skills, domain automation |
129131| [ ` wildclawbench ` ] ( https://github.com/InternLM/WildClawBench ) | 10 | Office workflows, safety alignment |
130132
0 commit comments