Skip to content

Commit 2bb2f4d

Browse files
authored
Merge pull request #1 from agentscope-ai/feat/local-env-opusjudge-submissions
refresh leaderboard data and readme
2 parents 2758de4 + c9dc00a commit 2bb2f4d

81 files changed

Lines changed: 7692 additions & 6015 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
.DS_Store
22

3+
# Python bytecode / cache
4+
__pycache__/
5+
*.py[cod]
6+
*$py.class
7+
38
# generated single-file snapshot (rebuild via `npm run build:snapshot`)
49
pawbench-snapshot.html
510

@@ -13,3 +18,11 @@ site/pawbench-site.zip
1318
# raw evaluation runs (large, not source-of-truth — aggregated outputs land
1419
# in submissions/ via site/scripts/aggregate_results.py)
1520
result/
21+
22+
#harness
23+
.claude/
24+
.ruff_cache
25+
.attribution/
26+
analysis/
27+
task_analysis/
28+
attribution/

‎README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@
22

33
> Languages: **English** · [简体中文](README.zh-CN.md)
44
5-
**An AI Agent benchmark evaluating the performance of (Model × Harness) combinations on production-grade tasks.**
5+
**PawBench is a production-grade AI Agent benchmark focused on evaluating the joint performance of (Foundation Model × Harness) combinations.**
66

7-
PawBench is built on the view that agent performance is a property of the `(Model × Harness)` combination. The same set of tasks is run across different `Model × Harness` combinations, so you can see each axis independently and tell whether a bottleneck is **model-limited** (needs a stronger LLM) or **harness-limited** (needs better retries, memory, tool routing, or error recovery).
7+
As AI moves toward production deployment, the role of the Harness has become increasingly critical, and overall Agent performance is truly a product of both the foundation model and the Harness working in synergy. PawBench breaks away from traditional single-dimension model evaluations by allowing both dimensions to be observed independently. This not only precisely diagnoses whether system bottlenecks are 'model-limited' or 'harness-limited' (such as optimizing retries, context management, tool routing, or error recovery), but also comprehensively evaluates the collaborative synergy of different technical combinations—providing scientific, quantitative guidance for architectural design, component selection, and system optimization, thereby accelerating the full lifecycle iteration of AI Agents.
88

9-
$$\text{Agent Performance} = f(\text{Model}, \text{Harness})$$
9+
$$\text{Agent Performance} = f(\text{Foundation Model}, \text{Harness})$$
10+
11+
PawBench v1.0 curates 150 production-grade representative tasks from the community, deeply annotated based on a five-dimensional orthogonal taxonomy (Scenario, Capability, Complexity, Modality, Environment). Together with a secure, container-isolated Docker sandbox, it provides developers with a multi-dimensional, scientifically reproducible Agent comprehensive evaluation capability and leaderboard.
1012

1113
## Quick Start
1214

‎README.zh-CN.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@
22

33
> Languages: [English](README.md) · **简体中文**
44
5-
**一个评估 (Model × Harness) 组合在生产环境任务中表现的 AI Agent 评测基准。**
5+
**PawBench 是一个专注于评估“基座模型 × Harness”联合效果的生产级 AI Agent 评测基准。**
66

7-
PawBench 的核心立场是:Agent 的表现是 `(模型 × Harness)` 组合的产物。同一套任务在不同的 `模型 × Harness` 组合下运行,让两条轴的能力都能被独立观察,帮助你判断瓶颈到底是 **模型受限**(需要换更强的 LLM)还是 **Harness 受限**(需要更好的重试、内存管理、工具路由或错误恢复)。
7+
随着 AI 走向落地,在生产环境中 Harness 的作用越来越重要,整个 Agent 的表现实际上是大模型与 Harness 共同作用的乘积效应。PawBench 打破了传统的单一维度大模型评测,让双轴能力能够被独立观察;这不仅能精准诊断系统瓶颈究竟是“模型受限”还是“Harness 受限”(如优化重试、上下文管理、工具路由或错误恢复),更能全面评估不同技术组合的协同效应,为架构设计、组件选型与系统优化提供科学的量化指导,加速 Agent 的全生命周期迭代。
88

9-
$$\text{Agent 表现} = f(\text{模型}, \text{Harness})$$
9+
$$\text{Agent 表现} = f(\text{基座模型}, \text{Harness})$$
10+
11+
PawBench v1.0 从社区精选了 150 个生产级代表性任务,并基于一套包含领域场景、原子能力、复杂度、交互模态、运行环境的五维正交标签体系进行深度打标,配合安全的 Docker 隔离沙箱,为开发者提供多维度、科学可复现的 Agent 全景实力评测能力与可视化排行榜。
1012

1113
## 快速开始
1214

‎pawbench/agents/base.py‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -28,16 +28,6 @@ async def teardown(self, environment: BaseEnvironment) -> None: ...
2828
@abc.abstractmethod
2929
def version(self) -> Optional[str]: ...
3030

31-
def get_system_prompt(self) -> Optional[str]:
32-
"""Return the system prompt used for this agent's last run.
33-
34-
Subclasses populate ``self._last_system_prompt`` inside
35-
``post_run_collect()`` (while the container is still reachable) so
36-
that ``backend.py`` can prepend it to the transcript after the
37-
container stops. The default implementation returns ``None``.
38-
"""
39-
return getattr(self, "_last_system_prompt", None)
40-
4131

4232
class ContainerAgent(BaseAgent):
4333
"""Base for agents that need installation inside the environment."""
@@ -62,6 +52,16 @@ async def post_run_collect(self, environment: BaseEnvironment) -> None:
6252
The default implementation is a no-op.
6353
"""
6454

55+
def get_system_prompt(self) -> Optional[str]:
56+
"""Return the system prompt used for this agent's last run.
57+
58+
Subclasses populate ``self._last_system_prompt`` inside
59+
``post_run_collect()`` (while the container is still reachable) so
60+
that ``backend.py`` can prepend it to the transcript after the
61+
container stops. The default implementation returns ``None``.
62+
"""
63+
return getattr(self, "_last_system_prompt", None)
64+
6565
def extract_transcript(
6666
self,
6767
local_workspace: "Path | None",

0 commit comments

Comments
 (0)