Skip to content

Commit ff9137a

Browse files
Polish README presentation
Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 81525fa commit ff9137a

4 files changed

Lines changed: 198 additions & 32 deletions

File tree

‎README.md‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,31 @@
1-
<p align="center">
2-
<img src="site/public/favicon.svg" width="110" alt="PawBench logo">
3-
</p>
4-
5-
<h1 align="center">PawBench</h1>
1+
<h1 align="center">🐾 PawBench</h1>
62

73
<p align="center">
84
<a href="README.md"><strong>English</strong></a> ·
95
<a href="README.zh-CN.md">简体中文</a>
106
</p>
117

128
<p align="center">
13-
<img alt="tasks" src="https://img.shields.io/badge/tasks-150-2ea44f">
14-
<img alt="models" src="https://img.shields.io/badge/models-9-0969da">
15-
<img alt="harnesses" src="https://img.shields.io/badge/harnesses-3-8250df">
9+
<a href="#tasks">
10+
<img alt="tasks" src="https://img.shields.io/badge/tasks-150-2ea44f">
11+
</a>
12+
<a href="https://agentscope-ai.github.io/PawBench/">
13+
<img alt="models" src="https://img.shields.io/badge/models-9-0969da">
14+
</a>
15+
<a href="#harnesses">
16+
<img alt="harnesses" src="https://img.shields.io/badge/harnesses-3-8250df">
17+
</a>
1618
<a href="https://agentscope-ai.github.io/PawBench/">
1719
<img alt="leaderboard" src="https://img.shields.io/badge/leaderboard-live-cf222e">
1820
</a>
19-
<a href="https://github.com/agentscope-ai/pawbench">
21+
<a href="https://github.com/agentscope-ai/PawBench">
2022
<img alt="GitHub repo" src="https://img.shields.io/badge/GitHub-pawbench-24292f">
2123
</a>
2224
</p>
2325

2426
<p align="center">
25-
<strong>A benchmark for evaluating foundation models and the harnesses that run them.</strong><br>
26-
150 agent tasks · 4,050 model-harness-task cells · score slices, transcripts, grader artifacts, and replayable runs.
27+
<strong>A benchmark for evaluating LLM × harness performance.</strong><br>
28+
150 agent tasks | 4,050 model-harness-task cells | score slices, transcripts, grader artifacts, and replayable runs.
2729
</p>
2830

2931
## Overview
@@ -32,11 +34,11 @@ The same model can behave very differently in different agent runtimes. When a t
3234

3335
PawBench evaluates the model and the harness together:
3436

35-
$$\text{Agent Performance} = f(\text{Foundation Model}, \text{Harness})$$
37+
$$\text{Agent Performance} = f(\text{LLM}, \text{Harness})$$
3638

3739
v1.0 covers **9 models × 3 harnesses × 150 tasks**. The first run shows a **5.6-point** average gap between the strongest and weakest harnesses, comparable to some model upgrades. On `qwen3.6-35b-a3b`, switching only the harness moves the score by **11.5 points**.
3840

39-
![PawBench leaderboard overview](site/public/pawbench-leaderboard-overview.png)
41+
![PawBench leaderboard overview](site/public/pawbench-leaderboard-overview.svg)
4042

4143
## Quick Start
4244

@@ -117,14 +119,14 @@ PawBench follows a **Reuse & Tag** methodology. Instead of writing every task fr
117119
| Modality | `modality` | `text` or `multimodal` (`image`, `audio`, `video`) |
118120
| Environment | `environment` | `closed` (offline, reproducible) or `open` (live internet / SaaS APIs) |
119121

120-
v1.0 contains **150 tasks** from `claweval`, `qwenclawbench`, `pinchbench`, `qwenpawbench`, `skillsbench`, and `wildclawbench`.
122+
v1.0 contains **150 tasks** from `claweval`, `qwenclawbench`, `pinchbench`, PawBench self-built tasks, `skillsbench`, and `wildclawbench`.
121123

122124
| Source | # | Main coverage |
123125
| :--- | ---: | :--- |
124126
| [`claweval`](https://github.com/claw-eval/claw-eval) | 52 | Office productivity, data analytics, content creation |
125127
| [`qwenclawbench`](https://github.com/SKYLENAGE-AI/QwenClawBench) | 29 | Automation, software engineering, safety alignment |
126-
| `pinchbench` | 23 | Office workflows, software engineering, information retrieval |
127-
| `qwenpawbench` | 21 | Automation, information retrieval, safety alignment |
128+
| [`pinchbench`](https://github.com/pinchbench/skill) | 23 | Office workflows, software engineering, information retrieval |
129+
| PawBench | 21 | Self-built tasks covering automation, information retrieval, and safety alignment |
128130
| [`skillsbench`](https://github.com/benchflow-ai/skillsbench) | 15 | Long-horizon skills, domain automation |
129131
| [`wildclawbench`](https://github.com/InternLM/WildClawBench) | 10 | Office workflows, safety alignment |
130132

‎README.zh-CN.md‎

Lines changed: 18 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,31 @@
1-
<p align="center">
2-
<img src="site/public/favicon.svg" width="110" alt="PawBench logo">
3-
</p>
4-
5-
<h1 align="center">PawBench</h1>
1+
<h1 align="center">🐾 PawBench</h1>
62

73
<p align="center">
84
<a href="README.md">English</a> ·
95
<a href="README.zh-CN.md"><strong>简体中文</strong></a>
106
</p>
117

128
<p align="center">
13-
<img alt="tasks" src="https://img.shields.io/badge/tasks-150-2ea44f">
14-
<img alt="models" src="https://img.shields.io/badge/models-9-0969da">
15-
<img alt="harnesses" src="https://img.shields.io/badge/harnesses-3-8250df">
9+
<a href="#任务构建">
10+
<img alt="tasks" src="https://img.shields.io/badge/tasks-150-2ea44f">
11+
</a>
12+
<a href="https://agentscope-ai.github.io/PawBench/">
13+
<img alt="models" src="https://img.shields.io/badge/models-9-0969da">
14+
</a>
15+
<a href="#harness">
16+
<img alt="harnesses" src="https://img.shields.io/badge/harnesses-3-8250df">
17+
</a>
1618
<a href="https://agentscope-ai.github.io/PawBench/">
1719
<img alt="leaderboard" src="https://img.shields.io/badge/leaderboard-live-cf222e">
1820
</a>
19-
<a href="https://github.com/agentscope-ai/pawbench">
21+
<a href="https://github.com/agentscope-ai/PawBench">
2022
<img alt="GitHub repo" src="https://img.shields.io/badge/GitHub-pawbench-24292f">
2123
</a>
2224
</p>
2325

2426
<p align="center">
25-
<strong>面向通用智能体的评测基准,用来评估基座模型和 Harness 的联合效果。</strong><br>
26-
150 道 Agent 任务 · 4,050 个 model-harness-task cell · 支持分数切片、执行轨迹、grader 产物和可复盘运行。
27+
<strong>面向通用智能体的评测基准,用来评估 LLM 和 Harness 的联合效果。</strong><br>
28+
150 道 Agent 任务 | 4,050 个 model-harness-task cell | 支持分数切片、执行轨迹、grader 产物和可复盘运行。
2729
</p>
2830

2931
## Overview
@@ -32,11 +34,11 @@
3234

3335
PawBench 同时评估模型和承载模型运行的 Harness:
3436

35-
$$\text{Agent 表现} = f(\text{基座模型}, \text{Harness})$$
37+
$$\text{Agent 表现} = f(\text{LLM}, \text{Harness})$$
3638

3739
v1.0 覆盖 **9 个模型 × 3 个 Harness × 150 道任务**。第一批结果显示,三家 Harness 的平均分最高和最低相差 **5.6 分**,已经接近一些模型版本升级带来的收益。以 `qwen3.6-35b-a3b` 为例,只切换 Harness,分数差距达到 **11.5 分**。
3840

39-
![PawBench 榜单概览](site/public/pawbench-leaderboard-overview.png)
41+
![PawBench 榜单概览](site/public/pawbench-leaderboard-overview.svg)
4042

4143
## 快速开始
4244

@@ -117,14 +119,14 @@ PawBench 采用 **Reuse & Tag** 方法。它不是从零手写所有任务,而
117119
| 模态 | `modality` | `text` 或 `multimodal`(`image`、`audio`、`video`) |
118120
| 环境 | `environment` | `closed`(离线、可复现)或 `open`(需要联网或真实 SaaS API) |
119121

120-
v1.0 包含 **150 道任务**,来源包括 `claweval`、`qwenclawbench`、`pinchbench`、`qwenpawbench`、`skillsbench` 和 `wildclawbench`。
122+
v1.0 包含 **150 道任务**,来源包括 `claweval`、`qwenclawbench`、`pinchbench`、PawBench 自建任务、`skillsbench` 和 `wildclawbench`。
121123

122124
| 来源 | 数量 | 主要覆盖 |
123125
| :--- | ---: | :--- |
124126
| [`claweval`](https://github.com/claw-eval/claw-eval) | 52 | 办公协同、数据分析、内容创作 |
125127
| [`qwenclawbench`](https://github.com/SKYLENAGE-AI/QwenClawBench) | 29 | 自动化、软件工程、安全对齐 |
126-
| `pinchbench` | 23 | 办公流程、软件工程、信息检索 |
127-
| `qwenpawbench` | 21 | 自动化、信息检索、安全对齐 |
128+
| [`pinchbench`](https://github.com/pinchbench/skill) | 23 | 办公流程、软件工程、信息检索 |
129+
| PawBench | 21 | 自建任务,覆盖自动化、信息检索、安全对齐 |
128130
| [`skillsbench`](https://github.com/benchflow-ai/skillsbench) | 15 | 长程 Skill、领域自动化 |
129131
| [`wildclawbench`](https://github.com/InternLM/WildClawBench) | 10 | 办公流程、安全对齐 |
130132

-71.1 KB
Binary file not shown.
Lines changed: 162 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)