Skip to content

Commit 78bee54

Browse files
Gui-YueGui-Yue
committed
refactor(benchmark): simplify to SWE-Verified + TB2 official flow, fix TB2 scoring, and trim dead TAU
 assets - unify benchmark pipeline around --benchmark=swe|tb2|both - switch SWE runner to verified-only instances - add official TB2 harbor wrapper runner (run-tb2-official.ts) - fix TB2 score parsing to ignore top-level summary result.json - remove legacy mini/full SWE + TAU + HTML benchmark code paths - delete unused TAU domain fixtures under tests/benchmark/tau/domains - update benchmark result docs (EN/ZH) to confirmed latest scores
1 parent 70d3de3 commit 78bee54

36 files changed

Lines changed: 1107 additions & 4001 deletions

‎.github/workflows/benchmark.yml‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
name: Benchmark Full Suite
2+
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
benchmark:
7+
description: "Which benchmark to run"
8+
type: choice
9+
required: true
10+
default: both
11+
options:
12+
- both
13+
- swe
14+
- tb2
15+
provider:
16+
description: "SWE provider filter"
17+
type: choice
18+
required: true
19+
default: all
20+
options:
21+
- all
22+
- anthropic
23+
- openai
24+
- gemini
25+
tb2_model:
26+
description: "TB2 model in provider/model format"
27+
type: string
28+
required: true
29+
default: openai/glm-5
30+
31+
env:
32+
NODE_VERSION: "20"
33+
34+
permissions:
35+
contents: read
36+
37+
jobs:
38+
benchmark:
39+
name: Benchmark
40+
runs-on: ubuntu-latest
41+
timeout-minutes: 360
42+
43+
steps:
44+
- name: Checkout
45+
uses: actions/checkout@v4
46+
47+
- name: Setup Node.js
48+
uses: actions/setup-node@v4
49+
with:
50+
node-version: ${{ env.NODE_VERSION }}
51+
cache: npm
52+
53+
- name: Setup uv
54+
uses: astral-sh/setup-uv@v4
55+
56+
- name: Login to Docker Hub (optional)
57+
if: ${{ secrets.DOCKERHUB_USERNAME != '' && secrets.DOCKERHUB_TOKEN != '' }}
58+
uses: docker/login-action@v3
59+
with:
60+
username: ${{ secrets.DOCKERHUB_USERNAME }}
61+
password: ${{ secrets.DOCKERHUB_TOKEN }}
62+
63+
- name: Install dependencies
64+
run: npm ci
65+
66+
- name: Create benchmark environment
67+
run: |
68+
cat > .env.test << 'EOT'
69+
ANTHROPIC_API_KEY=${{ secrets.ANTHROPIC_API_KEY }}
70+
ANTHROPIC_MODEL_ID=${{ vars.ANTHROPIC_MODEL_ID }}
71+
ANTHROPIC_BASE_URL=${{ vars.ANTHROPIC_BASE_URL }}
72+
73+
OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }}
74+
OPENAI_MODEL_ID=${{ vars.OPENAI_MODEL_ID }}
75+
OPENAI_BASE_URL=${{ vars.OPENAI_BASE_URL }}
76+
77+
GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
78+
GEMINI_MODEL_ID=${{ vars.GEMINI_MODEL_ID }}
79+
GEMINI_BASE_URL=${{ vars.GEMINI_BASE_URL }}
80+
81+
BENCHMARK_DOCKER_PROXY=${{ vars.BENCHMARK_DOCKER_PROXY }}
82+
BENCHMARK_TIMEOUT_MS=${{ vars.BENCHMARK_TIMEOUT_MS }}
83+
EOT
84+
85+
- name: Run unified benchmark command
86+
run: |
87+
mkdir -p tests/tmp
88+
args=(
89+
--benchmark=${{ inputs.benchmark }}
90+
--tb2-model=${{ inputs.tb2_model }}
91+
--tb2-agent=oracle
92+
--tb2-runner=uvx
93+
--tb2-python=3.12
94+
--tb2-jobs-dir=./tests/tmp/jobs
95+
--output=json
96+
--output-file=tests/tmp/benchmark-report.json
97+
)
98+
99+
if [[ "${{ inputs.provider }}" != "all" && "${{ inputs.benchmark }}" != "tb2" ]]; then
100+
args+=(--provider=${{ inputs.provider }})
101+
fi
102+
103+
npm run test:benchmark -- "${args[@]}"
104+
105+
- name: Write step summary
106+
if: ${{ always() }}
107+
run: |
108+
node - <<'NODE' >> "$GITHUB_STEP_SUMMARY"
109+
const fs = require('fs');
110+
function readJson(p) {
111+
if (!fs.existsSync(p)) return null;
112+
try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch { return null; }
113+
}
114+
115+
const report = readJson('tests/tmp/benchmark-report.json');
116+
console.log('## Benchmark Report');
117+
console.log('');
118+
119+
if (!report) {
120+
console.log('- report not found');
121+
process.exit(0);
122+
}
123+
124+
if (Array.isArray(report.swe) && report.swe.length > 0) {
125+
console.log('### SWE-bench-Verified');
126+
console.log('');
127+
console.log('| Provider / Model | Resolved | Rate |');
128+
console.log('|---|---:|---:|');
129+
for (const r of report.swe) {
130+
const name = `${r.provider.id} / ${r.provider.model}`;
131+
const resolved = `${r.summary.resolved}/${r.summary.total}`;
132+
const rate = `${(r.summary.rate * 100).toFixed(1)}%`;
133+
console.log(`| ${name} | ${resolved} | ${rate} |`);
134+
}
135+
console.log('');
136+
}
137+
138+
if (report.tb2) {
139+
const tb2 = report.tb2;
140+
console.log('### Terminal Bench 2.0');
141+
console.log('');
142+
console.log(`- Agent: \`${tb2.agent}\``);
143+
if (tb2.model) console.log(`- Model: \`${tb2.model}\``);
144+
console.log(`- Passed: **${tb2.passed}/${tb2.total}**`);
145+
console.log(`- Rate: **${(tb2.rate * 100).toFixed(1)}%**`);
146+
console.log('');
147+
}
148+
NODE
149+
150+
- name: Upload benchmark artifacts
151+
if: ${{ always() }}
152+
uses: actions/upload-artifact@v4
153+
with:
154+
name: benchmark-artifacts-${{ github.run_id }}
155+
if-no-files-found: warn
156+
path: |
157+
tests/tmp/benchmark-report.json
158+
tests/tmp/jobs/*/result.json

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,7 @@ See [docs/en/guides/architecture.md](./docs/en/guides/architecture.md) for detai
146146
| [Providers](./docs/en/guides/providers.md) | Model provider configuration |
147147
| [Database](./docs/en/guides/database.md) | SQLite/PostgreSQL persistence |
148148
| [Resume & Fork](./docs/en/guides/resume-fork.md) | Crash recovery & branching |
149+
| [Benchmark Results](./docs/en/guides/benchmark-results.md) | Confirmed benchmark score tables |
149150
| **Project** | |
150151
| [Contribution Guide](./docs/en/contribution.md) | How to contribute |
151152
| **Reference** | |

‎README.zh-CN.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ npm run example:room # 多Agent协作
106106
| [Provider 配置](./docs/zh-CN/guides/providers.md) | 模型 Provider 配置 |
107107
| [数据库存储](./docs/zh-CN/guides/database.md) | SQLite/PostgreSQL 持久化 |
108108
| [恢复与分叉](./docs/zh-CN/guides/resume-fork.md) | 崩溃恢复与分支 |
109+
| [Benchmark 结果](./docs/zh-CN/guides/benchmark-results.md) | 已确认的跑分结果表格 |
109110
| **项目** | |
110111
| [贡献指南](./docs/zh-CN/contribution.md) | 提交 PR 的要求与流程 |
111112
| **参考** | |

‎docs/en/guides/benchmark-results.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Benchmark Results (Confirmed)
2+
3+
Last updated: 2026-02-26
4+
5+
## SWE-bench-Verified
6+
7+
| Provider / Model | Instances | Resolved | Rate | Avg Tokens | Avg Duration |
8+
|---|---:|---:|---:|---:|---:|
9+
| openai / glm-5 | 12 | 12/12 | 100.0% | 17.2k | 134.5k ms |
10+
11+
Source: local full run log (`2026-02-25__21-06-21`).
12+
13+
## Terminal Bench 2.0
14+
15+
| Agent / Model | Passed | Parseable | Unknown | Rate (parseable) | Notes |
16+
|---|---:|---:|---:|---:|---|
17+
| oracle / glm-5 | 1 | 31 | 58 | 3.2% | From the same full run; many tasks ended with runtime/timeout errors. |
18+
19+
## Reproduce
20+
21+
```bash
22+
npm run test:benchmark -- \
23+
--benchmark=both \
24+
--tb2-model=openai/glm-5 \
25+
--tb2-agent=oracle \
26+
--tb2-runner=uvx \
27+
--tb2-jobs-dir=./tests/tmp/jobs \
28+
--output=json \
29+
--output-file=tests/tmp/benchmark-report.json
30+
```
31+
32+
The JSON report includes both `swe` and `tb2` sections.

0 commit comments

Comments
 (0)