You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/development/benchmark-developer-workflow.md
+62Lines changed: 62 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1437,10 +1437,72 @@ selected reducer can be rerun without private material.
1437
1437
| Terminal-Bench | Cloud Codex CLI runs the task on a dedicated benchmark host; LoopX ingests compact no-upload evidence. | Prior split-control adapters remain useful reducers, but the next run should prefer direct cloud-host Codex plus container runtime. |
1438
1438
| SkillsBench | Cloud Codex CLI and BenchFlow run on the same dedicated host; LoopX records compact base/test mini-pair evidence. | Prior host-local ACP relay work is historical route-repair evidence. Do not add more bridge layers before trying the cloud-host path. |
1439
1439
| Agents' Last Exam | Cloud Codex CLI drives the local-Docker-capable ALE route on the dedicated host; LoopX ingests compact no-upload evidence. | Formal task runs still need task-data and public-claim gates, but Docker/Codex colocation should replace the earlier local-host split-control assumption. |
1440
+
| EdgeBench | SForge owns isolated work/judge containers and iterative feedback; LoopX gates one public task plan and ingests compact best-over-time evidence. | The built-in adapter is no-execution and fail-closed. A real task still requires explicit task/image/judge readiness, private model credentials, launch authority, and a cost review. |
1440
1441
1441
1442
This table is intentionally about runner maturity, not leaderboard score.
1442
1443
Score claims require separate public-safe result ingestion and review.
1443
1444
1445
+
### EdgeBench SForge Gate
1446
+
1447
+
EdgeBench publishes 51 open tasks from a 134-task benchmark. Tasks are designed
1448
+
for long-horizon environment learning, use isolated work and judge containers,
1449
+
and score the best submission observed during the run. Official reports compare
1450
+
2, 4, 6, 8, 10, and 12 hour checkpoints. One frontier-model task can cost
1451
+
hundreds to more than one thousand USD, so repository validation must never
1452
+
start a task.
1453
+
1454
+
Start with the compact preflight:
1455
+
1456
+
```bash
1457
+
loopx benchmark edgebench-preflight \
1458
+
--task-id ad_placement_optimization \
1459
+
--backend docker \
1460
+
--task-catalog-ready \
1461
+
--task-images-ready \
1462
+
--judge-ready \
1463
+
--require-ready \
1464
+
--format json > edgebench-preflight.json
1465
+
```
1466
+
1467
+
This checks only public-safe readiness signals plus local Linux, SForge, and
1468
+
container-runtime availability. It does not fetch tasks, pull images, start
1469
+
containers, read task bodies, invoke a model, upload, or submit. When the
1470
+
preflight is ready, create a bounded private launch plan:
1471
+
1472
+
```bash
1473
+
loopx benchmark edgebench-run-plan \
1474
+
--preflight-json edgebench-preflight.json \
1475
+
--agent codex \
1476
+
--model <public-model-label> \
1477
+
--run-id <public-run-id> \
1478
+
--timeout-seconds 7200 \
1479
+
--require-ready \
1480
+
--format json
1481
+
```
1482
+
1483
+
The plan names required credential and endpoint environment variables but never
1484
+
reads or records their values. It rejects budgets above the official 43,200
1485
+
second ceiling and does not claim official-setting or leaderboard
1486
+
comparability. Launch SForge separately only after the operator approves model
1487
+
cost, credentials, task assets, and the selected backend.
1488
+
1489
+
After a private run, reduce only SForge's compact `final_result.json`:
1490
+
1491
+
```bash
1492
+
loopx benchmark edgebench-result-reduce \
1493
+
--result-json <private-final-result.json> \
1494
+
--task-id <public-task-id> \
1495
+
--run-id <public-run-id> \
1496
+
--require-countable \
1497
+
--format json
1498
+
```
1499
+
1500
+
The reducer retains best score/pass rate, best round, submission counts,
1501
+
runtime, timeout, and resume count. It ignores raw agent output and archive
1502
+
metadata. A compact result is still not an official score or treatment claim;
1503
+
use the normal benchmark ledger and comparison review gates before making
1504
+
either claim.
1505
+
1444
1506
### SkillsBench Split-Control Preflight
1445
1507
1446
1508
This preflight is retained for historical split-control debugging and for
0 commit comments