Skip to content

Commit a2d097e

Browse files
wangyuchao.404TRAE CLI
andcommitted
docs(benchmark): document EdgeBench SForge boundary
Document the provider ownership split, no-execution command flow, long-run cost gate, and compact evidence boundary. Co-authored-by: TRAE CLI <noreply@bytedance.com>
1 parent 5a2f4cf commit a2d097e

2 files changed

Lines changed: 91 additions & 0 deletions

File tree

docs/development/benchmark-developer-workflow.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1437,10 +1437,72 @@ selected reducer can be rerun without private material.
14371437
| Terminal-Bench | Cloud Codex CLI runs the task on a dedicated benchmark host; LoopX ingests compact no-upload evidence. | Prior split-control adapters remain useful reducers, but the next run should prefer direct cloud-host Codex plus container runtime. |
14381438
| SkillsBench | Cloud Codex CLI and BenchFlow run on the same dedicated host; LoopX records compact base/test mini-pair evidence. | Prior host-local ACP relay work is historical route-repair evidence. Do not add more bridge layers before trying the cloud-host path. |
14391439
| Agents' Last Exam | Cloud Codex CLI drives the local-Docker-capable ALE route on the dedicated host; LoopX ingests compact no-upload evidence. | Formal task runs still need task-data and public-claim gates, but Docker/Codex colocation should replace the earlier local-host split-control assumption. |
1440+
| EdgeBench | SForge owns isolated work/judge containers and iterative feedback; LoopX gates one public task plan and ingests compact best-over-time evidence. | The built-in adapter is no-execution and fail-closed. A real task still requires explicit task/image/judge readiness, private model credentials, launch authority, and a cost review. |
14401441

14411442
This table is intentionally about runner maturity, not leaderboard score.
14421443
Score claims require separate public-safe result ingestion and review.
14431444

1445+
### EdgeBench SForge Gate
1446+
1447+
EdgeBench publishes 51 open tasks from a 134-task benchmark. Tasks are designed
1448+
for long-horizon environment learning, use isolated work and judge containers,
1449+
and score the best submission observed during the run. Official reports compare
1450+
2, 4, 6, 8, 10, and 12 hour checkpoints. One frontier-model task can cost
1451+
hundreds to more than one thousand USD, so repository validation must never
1452+
start a task.
1453+
1454+
Start with the compact preflight:
1455+
1456+
```bash
1457+
loopx benchmark edgebench-preflight \
1458+
--task-id ad_placement_optimization \
1459+
--backend docker \
1460+
--task-catalog-ready \
1461+
--task-images-ready \
1462+
--judge-ready \
1463+
--require-ready \
1464+
--format json > edgebench-preflight.json
1465+
```
1466+
1467+
This checks only public-safe readiness signals plus local Linux, SForge, and
1468+
container-runtime availability. It does not fetch tasks, pull images, start
1469+
containers, read task bodies, invoke a model, upload, or submit. When the
1470+
preflight is ready, create a bounded private launch plan:
1471+
1472+
```bash
1473+
loopx benchmark edgebench-run-plan \
1474+
--preflight-json edgebench-preflight.json \
1475+
--agent codex \
1476+
--model <public-model-label> \
1477+
--run-id <public-run-id> \
1478+
--timeout-seconds 7200 \
1479+
--require-ready \
1480+
--format json
1481+
```
1482+
1483+
The plan names required credential and endpoint environment variables but never
1484+
reads or records their values. It rejects budgets above the official 43,200
1485+
second ceiling and does not claim official-setting or leaderboard
1486+
comparability. Launch SForge separately only after the operator approves model
1487+
cost, credentials, task assets, and the selected backend.
1488+
1489+
After a private run, reduce only SForge's compact `final_result.json`:
1490+
1491+
```bash
1492+
loopx benchmark edgebench-result-reduce \
1493+
--result-json <private-final-result.json> \
1494+
--task-id <public-task-id> \
1495+
--run-id <public-run-id> \
1496+
--require-countable \
1497+
--format json
1498+
```
1499+
1500+
The reducer retains best score/pass rate, best round, submission counts,
1501+
runtime, timeout, and resume count. It ignores raw agent output and archive
1502+
metadata. A compact result is still not an official score or treatment claim;
1503+
use the normal benchmark ledger and comparison review gates before making
1504+
either claim.
1505+
14441506
### SkillsBench Split-Control Preflight
14451507

14461508
This preflight is retained for historical split-control debugging and for

docs/reference/benchmark-architecture.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,3 +55,32 @@ benchmark-specific read models.
5555
This is a product boundary, not an isolation claim: benchmark execution still
5656
uses LoopX state and receipts, and it remains covered by the same release and
5757
public-evidence policies.
58+
59+
## EdgeBench Provider Boundary
60+
61+
EdgeBench uses the same ownership rule. The built-in
62+
`loopx.benchmark_adapters.edgebench` module owns only LoopX-facing readiness,
63+
single-task planning, and compact result normalization. The upstream SForge
64+
harness remains the runner provider and owns task acquisition, work/judge
65+
container isolation, iterative submissions, hidden evaluation, Docker or
66+
Kubernetes execution, and agent/model launch.
67+
68+
The first integration slice deliberately does not wrap `sforge fetch-tasks`,
69+
`sforge pull`, `sforge serve`, or `sforge run`. Those commands can download
70+
large task assets, pull images, start containers, invoke paid model APIs, and
71+
run for up to 12 hours. LoopX instead exposes fail-closed commands that make the
72+
boundary reviewable before any of those effects:
73+
74+
```bash
75+
loopx benchmark edgebench-preflight --task-id <public-task-id>
76+
loopx benchmark edgebench-run-plan --preflight-json <compact-preflight.json> \
77+
--agent <agent> --model <model> --run-id <public-run-id>
78+
loopx benchmark edgebench-result-reduce --result-json <final-result.json> \
79+
--task-id <public-task-id> --run-id <public-run-id>
80+
```
81+
82+
The result reducer accepts only SForge's compact final result shape and keeps
83+
best score, pass rate, best round, submission counts, runtime, timeout, and
84+
resume count. It does not preserve raw agent output, archives, task bodies,
85+
hidden tests, logs, trajectories, credentials, paths, uploads, or leaderboard
86+
claims.

0 commit comments

Comments
 (0)