diff --git a/docs/development/benchmark-developer-workflow.md b/docs/development/benchmark-developer-workflow.md index 9c30904dd..7a42076f7 100644 --- a/docs/development/benchmark-developer-workflow.md +++ b/docs/development/benchmark-developer-workflow.md @@ -1437,10 +1437,72 @@ selected reducer can be rerun without private material. | Terminal-Bench | Cloud Codex CLI runs the task on a dedicated benchmark host; LoopX ingests compact no-upload evidence. | Prior split-control adapters remain useful reducers, but the next run should prefer direct cloud-host Codex plus container runtime. | | SkillsBench | Cloud Codex CLI and BenchFlow run on the same dedicated host; LoopX records compact base/test mini-pair evidence. | Prior host-local ACP relay work is historical route-repair evidence. Do not add more bridge layers before trying the cloud-host path. | | Agents' Last Exam | Cloud Codex CLI drives the local-Docker-capable ALE route on the dedicated host; LoopX ingests compact no-upload evidence. | Formal task runs still need task-data and public-claim gates, but Docker/Codex colocation should replace the earlier local-host split-control assumption. | +| EdgeBench | SForge owns isolated work/judge containers and iterative feedback; LoopX gates one public task plan and ingests compact best-over-time evidence. | The built-in adapter is no-execution and fail-closed. A real task still requires explicit task/image/judge readiness, private model credentials, launch authority, and a cost review. | This table is intentionally about runner maturity, not leaderboard score. Score claims require separate public-safe result ingestion and review. +### EdgeBench SForge Gate + +EdgeBench publishes 51 open tasks from a 134-task benchmark. Tasks are designed +for long-horizon environment learning, use isolated work and judge containers, +and score the best submission observed during the run. Official reports compare +2, 4, 6, 8, 10, and 12 hour checkpoints. One frontier-model task can cost +hundreds to more than one thousand USD, so repository validation must never +start a task. + +Start with the compact preflight: + +```bash +loopx benchmark edgebench-preflight \ + --task-id ad_placement_optimization \ + --backend docker \ + --task-catalog-ready \ + --task-images-ready \ + --judge-ready \ + --require-ready \ + --format json > edgebench-preflight.json +``` + +This checks only public-safe readiness signals plus local Linux, SForge, and +container-runtime availability. It does not fetch tasks, pull images, start +containers, read task bodies, invoke a model, upload, or submit. When the +preflight is ready, create a bounded private launch plan: + +```bash +loopx benchmark edgebench-run-plan \ + --preflight-json edgebench-preflight.json \ + --agent codex \ + --model \ + --run-id \ + --timeout-seconds 7200 \ + --require-ready \ + --format json +``` + +The plan names required credential and endpoint environment variables but never +reads or records their values. It rejects budgets above the official 43,200 +second ceiling and does not claim official-setting or leaderboard +comparability. Launch SForge separately only after the operator approves model +cost, credentials, task assets, and the selected backend. + +After a private run, reduce only SForge's compact `final_result.json`: + +```bash +loopx benchmark edgebench-result-reduce \ + --result-json \ + --task-id \ + --run-id \ + --require-countable \ + --format json +``` + +The reducer retains best score/pass rate, best round, submission counts, +runtime, timeout, and resume count. It ignores raw agent output and archive +metadata. A compact result is still not an official score or treatment claim; +use the normal benchmark ledger and comparison review gates before making +either claim. + ### SkillsBench Split-Control Preflight This preflight is retained for historical split-control debugging and for diff --git a/docs/reference/benchmark-architecture.md b/docs/reference/benchmark-architecture.md index 6ea7a1d65..d0c5399e0 100644 --- a/docs/reference/benchmark-architecture.md +++ b/docs/reference/benchmark-architecture.md @@ -55,3 +55,32 @@ benchmark-specific read models. This is a product boundary, not an isolation claim: benchmark execution still uses LoopX state and receipts, and it remains covered by the same release and public-evidence policies. + +## EdgeBench Provider Boundary + +EdgeBench uses the same ownership rule. The built-in +`loopx.benchmark_adapters.edgebench` module owns only LoopX-facing readiness, +single-task planning, and compact result normalization. The upstream SForge +harness remains the runner provider and owns task acquisition, work/judge +container isolation, iterative submissions, hidden evaluation, Docker or +Kubernetes execution, and agent/model launch. + +The first integration slice deliberately does not wrap `sforge fetch-tasks`, +`sforge pull`, `sforge serve`, or `sforge run`. Those commands can download +large task assets, pull images, start containers, invoke paid model APIs, and +run for up to 12 hours. LoopX instead exposes fail-closed commands that make the +boundary reviewable before any of those effects: + +```bash +loopx benchmark edgebench-preflight --task-id +loopx benchmark edgebench-run-plan --preflight-json \ + --agent --model --run-id +loopx benchmark edgebench-result-reduce --result-json \ + --task-id --run-id +``` + +The result reducer accepts only SForge's compact final result shape and keeps +best score, pass rate, best round, submission counts, runtime, timeout, and +resume count. It does not preserve raw agent output, archives, task bodies, +hidden tests, logs, trajectories, credentials, paths, uploads, or leaderboard +claims. diff --git a/examples/edgebench-benchmark-adapter-smoke.py b/examples/edgebench-benchmark-adapter-smoke.py new file mode 100644 index 000000000..8d8eede55 --- /dev/null +++ b/examples/edgebench-benchmark-adapter-smoke.py @@ -0,0 +1,286 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +if str(ROOT) not in sys.path: + sys.path.insert(0, str(ROOT)) + +from loopx.benchmark_adapters.edgebench import ( # noqa: E402 + EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS, + build_edgebench_sforge_preflight, + build_edgebench_sforge_run_plan, + reduce_edgebench_final_result, +) + + +def assert_boundary(payload: dict[str, object]) -> None: + boundary = payload["boundary"] + assert isinstance(boundary, dict) + assert boundary["container_started"] is False + assert boundary["task_body_read"] is False + assert boundary["hidden_tests_read"] is False + assert boundary["model_api_invoked"] is False + assert boundary["raw_logs_read"] is False + assert boundary["raw_trajectory_read"] is False + assert boundary["credential_values_recorded"] is False + assert boundary["local_paths_recorded"] is False + assert boundary["command_argv_recorded"] is False + assert boundary["upload_invoked"] is False + assert boundary["submit_invoked"] is False + assert boundary["leaderboard_evidence"] is False + + +def ready_preflight() -> dict[str, object]: + return build_edgebench_sforge_preflight( + task_id="ad_placement_optimization", + backend="docker", + linux_host=True, + sforge_available=True, + container_runtime_available=True, + task_catalog_ready=True, + task_images_ready=True, + judge_ready=True, + ) + + +def test_contracts() -> None: + ready = ready_preflight() + assert ready["ready"] is True + assert ready["open_task_count"] == 51 + assert ready["first_blocker"] == "ready_for_edgebench_run_plan" + assert ready["score_policy"]["selection"] == "best_submission_over_time" + assert_boundary(ready) + + blocked = build_edgebench_sforge_preflight( + task_id="ad_placement_optimization", + backend="docker", + linux_host=True, + sforge_available=False, + container_runtime_available=False, + task_catalog_ready=False, + task_images_ready=False, + judge_ready=False, + ) + assert blocked["ready"] is False + assert blocked["first_blocker"] == "sforge_not_available" + assert_boundary(blocked) + + plan = build_edgebench_sforge_run_plan( + preflight=ready, + agent="codex", + model="gpt-5.5", + timeout_seconds=7_200, + run_id="edgebench-smoke-001", + ) + assert plan["ready"] is True + assert plan["runner"]["entrypoint"] == "sforge run" + assert plan["runner"]["required_secret_names"] == ["SFORGE_AGENT_API_KEY"] + assert plan["evaluation"]["official_setting"] is False + assert plan["evaluation"]["leaderboard_eligible"] is False + assert_boundary(plan) + + forged = build_edgebench_sforge_run_plan( + preflight={ + "ready": True, + "task_id": "ad_placement_optimization", + "backend": "docker", + }, + agent="codex", + model="gpt-5.5", + timeout_seconds=7_200, + run_id="edgebench-smoke-forged", + ) + assert forged["ready"] is False + assert forged["first_blocker"] == "edgebench_preflight_contract_invalid" + assert_boundary(forged) + + try: + build_edgebench_sforge_run_plan( + preflight=ready, + agent="codex", + model="gpt-5.5", + timeout_seconds=EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS + 1, + run_id="edgebench-smoke-002", + ) + except ValueError as exc: + assert "timeout_seconds" in str(exc) + else: + raise AssertionError("run plan accepted a budget above 12 hours") + + compact = reduce_edgebench_final_result( + task_id="ad_placement_optimization", + run_id="edgebench-smoke-001", + result={ + "best_score": 62.9, + "best_pass_rate": 0.8, + "best_round": "agent-4", + "total_rounds": 7, + "agent_submissions": 4, + "auto_submissions": 3, + "runtime_seconds": 7_199.5, + "resume_count": 1, + "timed_out": True, + "agent_output": "must not be projected", + "archive_size_bytes": 1234, + }, + ) + assert compact["countable"] is True + assert compact["metrics"] == { + "best_score": 62.9, + "best_pass_rate": 0.8, + "best_round": "agent-4", + "total_rounds": 7, + "agent_submissions": 4, + "auto_submissions": 3, + "runtime_seconds": 7_199.5, + "resume_count": 1, + "timed_out": True, + } + assert "agent_output" not in json.dumps(compact) + assert "archive_size_bytes" not in json.dumps(compact) + assert_boundary(compact) + + mismatched = reduce_edgebench_final_result( + task_id="ad_placement_optimization", + run_id="edgebench-smoke-mismatch", + result={ + "best_score": 62.9, + "total_rounds": 7, + "agent_submissions": 4, + "auto_submissions": 2, + "runtime_seconds": 7_199.5, + }, + ) + assert mismatched["countable"] is False + assert mismatched["first_blocker"] == "edgebench_submission_rounds_mismatch" + assert_boundary(mismatched) + + for invalid_pass_rate in (1.5, -0.1): + invalid = reduce_edgebench_final_result( + task_id="ad_placement_optimization", + run_id=f"edgebench-smoke-pass-rate-{invalid_pass_rate}", + result={ + "best_pass_rate": invalid_pass_rate, + "total_rounds": 1, + "agent_submissions": 1, + "auto_submissions": 0, + "runtime_seconds": 60.0, + }, + ) + assert invalid["countable"] is False + assert invalid["first_blocker"] == "edgebench_pass_rate_out_of_range" + assert_boundary(invalid) + + +def run_cli(*args: str) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, "-m", "loopx.cli", "--format", "json", *args], + cwd=ROOT, + text=True, + capture_output=True, + check=False, + ) + + +def test_cli() -> None: + preflight = run_cli( + "benchmark", + "edgebench-preflight", + "--task-id", + "ad_placement_optimization", + "--no-environment-probe", + ) + assert preflight.returncode == 0, preflight.stderr + blocked = json.loads(preflight.stdout) + assert blocked["ok"] is True + assert blocked["ready"] is False + assert blocked["first_blocker"] == "edgebench_linux_host_required" + assert_boundary(blocked) + + required = run_cli( + "benchmark", + "edgebench-preflight", + "--task-id", + "ad_placement_optimization", + "--no-environment-probe", + "--require-ready", + ) + assert required.returncode == 1 + assert json.loads(required.stdout)["error"] == "edgebench_linux_host_required" + + with tempfile.TemporaryDirectory(prefix="loopx-edgebench-smoke-") as temp_dir: + root = Path(temp_dir) + preflight_path = root / "preflight.json" + preflight_path.write_text(json.dumps(ready_preflight()), encoding="utf-8") + plan = run_cli( + "benchmark", + "edgebench-run-plan", + "--preflight-json", + str(preflight_path), + "--agent", + "codex", + "--model", + "gpt-5.5", + "--run-id", + "edgebench-smoke-001", + "--timeout-seconds", + "7200", + "--require-ready", + ) + assert plan.returncode == 0, plan.stderr + plan_payload = json.loads(plan.stdout) + assert plan_payload["ok"] is True + assert plan_payload["ready"] is True + assert_boundary(plan_payload) + + result_path = root / "final_result.json" + result_path.write_text( + json.dumps( + { + "best_score": 62.9, + "best_pass_rate": 0.8, + "best_round": "agent-4", + "total_rounds": 7, + "agent_submissions": 4, + "auto_submissions": 3, + "runtime_seconds": 7199.5, + "resume_count": 1, + "timed_out": True, + } + ), + encoding="utf-8", + ) + reduced = run_cli( + "benchmark", + "edgebench-result-reduce", + "--result-json", + str(result_path), + "--task-id", + "ad_placement_optimization", + "--run-id", + "edgebench-smoke-001", + "--require-countable", + ) + assert reduced.returncode == 0, reduced.stderr + reduced_payload = json.loads(reduced.stdout) + assert reduced_payload["ok"] is True + assert reduced_payload["countable"] is True + assert_boundary(reduced_payload) + + +def main() -> int: + test_contracts() + test_cli() + print("edgebench-benchmark-adapter-smoke ok") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/loopx/benchmark_adapters/edgebench.py b/loopx/benchmark_adapters/edgebench.py new file mode 100644 index 000000000..03cecca46 --- /dev/null +++ b/loopx/benchmark_adapters/edgebench.py @@ -0,0 +1,303 @@ +from __future__ import annotations + +import math +import re +from collections.abc import Mapping +from typing import Any + + +EDGEBENCH_BENCHMARK_ID = "edgebench" +EDGEBENCH_UPSTREAM_REPOSITORY = "https://github.com/ByteDance-Seed/EdgeBench" +EDGEBENCH_OPEN_TASK_COUNT = 51 +EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS = 43_200 +EDGEBENCH_TIME_CHECKPOINT_SECONDS = (7_200, 14_400, 21_600, 28_800, 36_000, 43_200) +EDGEBENCH_PREFLIGHT_SCHEMA_VERSION = "edgebench_sforge_preflight_v0" +EDGEBENCH_RUN_PLAN_SCHEMA_VERSION = "edgebench_sforge_run_plan_v0" +EDGEBENCH_RESULT_SCHEMA_VERSION = "edgebench_compact_result_v0" + +_PUBLIC_LABEL = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,159}") + + +def _public_label(value: Any, *, field: str) -> str: + text = str(value or "").strip() + if not _PUBLIC_LABEL.fullmatch(text): + raise ValueError(f"{field} must be a public-safe label") + return text + + +def _optional_number(value: Any) -> float | None: + if value is None or isinstance(value, bool): + return None + try: + number = float(value) + except (TypeError, ValueError): + return None + return number if math.isfinite(number) else None + + +def _optional_non_negative_int(value: Any) -> int | None: + if value is None or isinstance(value, bool): + return None + try: + number = int(value) + except (TypeError, ValueError): + return None + return number if number >= 0 else None + + +def _boundary() -> dict[str, bool]: + return { + "no_execution": True, + "container_started": False, + "task_body_read": False, + "hidden_tests_read": False, + "model_api_invoked": False, + "raw_logs_read": False, + "raw_trajectory_read": False, + "credential_values_recorded": False, + "local_paths_recorded": False, + "command_argv_recorded": False, + "upload_invoked": False, + "submit_invoked": False, + "leaderboard_evidence": False, + } + + +def build_edgebench_sforge_preflight( + *, + task_id: str, + backend: str, + linux_host: bool, + sforge_available: bool, + container_runtime_available: bool, + task_catalog_ready: bool, + task_images_ready: bool, + judge_ready: bool, +) -> dict[str, Any]: + task_label = _public_label(task_id, field="task_id") + if backend not in {"docker", "kubernetes"}: + raise ValueError("backend must be docker or kubernetes") + + checks = { + "linux_host": bool(linux_host), + "sforge_available": bool(sforge_available), + "container_runtime_available": bool(container_runtime_available), + "task_catalog_ready": bool(task_catalog_ready), + "task_images_ready": bool(task_images_ready), + "judge_ready": bool(judge_ready), + } + blocker_order = ( + ("linux_host", "edgebench_linux_host_required"), + ("sforge_available", "sforge_not_available"), + ("container_runtime_available", f"{backend}_runtime_not_available"), + ("task_catalog_ready", "edgebench_task_catalog_not_ready"), + ("task_images_ready", "edgebench_task_images_not_ready"), + ("judge_ready", "edgebench_judge_not_ready"), + ) + blockers = [blocker for check, blocker in blocker_order if not checks[check]] + ready = not blockers + return { + "schema_version": EDGEBENCH_PREFLIGHT_SCHEMA_VERSION, + "benchmark_id": EDGEBENCH_BENCHMARK_ID, + "upstream_repository": EDGEBENCH_UPSTREAM_REPOSITORY, + "open_task_count": EDGEBENCH_OPEN_TASK_COUNT, + "task_id": task_label, + "backend": backend, + "ready": ready, + "first_blocker": blockers[0] if blockers else "ready_for_edgebench_run_plan", + "blockers": blockers, + "checks": checks, + "score_policy": { + "selection": "best_submission_over_time", + "time_checkpoints_seconds": list(EDGEBENCH_TIME_CHECKPOINT_SECONDS), + "official_timeout_seconds": EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS, + }, + "boundary": _boundary(), + "decision": { + "next_allowed_action": ( + "build_single_task_edgebench_run_plan" + if ready + else "repair_edgebench_preflight_before_run_plan" + ), + "must_not_claim": [ + "task execution", + "official EdgeBench score", + "leaderboard comparability", + "LoopX treatment benefit", + ], + }, + } + + +def build_edgebench_sforge_run_plan( + *, + preflight: Mapping[str, Any], + agent: str, + model: str, + timeout_seconds: int, + run_id: str, +) -> dict[str, Any]: + task_id = _public_label(preflight.get("task_id"), field="task_id") + agent_label = _public_label(agent, field="agent") + model_label = _public_label(model, field="model") + run_label = _public_label(run_id, field="run_id") + if timeout_seconds <= 0 or timeout_seconds > EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS: + raise ValueError( + f"timeout_seconds must be between 1 and {EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS}" + ) + + checks = preflight.get("checks") + canonical_preflight = ( + preflight.get("schema_version") == EDGEBENCH_PREFLIGHT_SCHEMA_VERSION + and preflight.get("benchmark_id") == EDGEBENCH_BENCHMARK_ID + and preflight.get("backend") in {"docker", "kubernetes"} + and isinstance(checks, Mapping) + and all( + checks.get(field) is True + for field in ( + "linux_host", + "sforge_available", + "container_runtime_available", + "task_catalog_ready", + "task_images_ready", + "judge_ready", + ) + ) + ) + preflight_ready = preflight.get("ready") is True and canonical_preflight + blockers = ( + [] + if preflight_ready + else [ + ( + str(preflight.get("first_blocker") or "edgebench_preflight_not_ready") + if canonical_preflight + else "edgebench_preflight_contract_invalid" + ) + ] + ) + return { + "schema_version": EDGEBENCH_RUN_PLAN_SCHEMA_VERSION, + "benchmark_id": EDGEBENCH_BENCHMARK_ID, + "task_id": task_id, + "ready": preflight_ready, + "first_blocker": blockers[0] if blockers else "ready_for_private_sforge_launch", + "blockers": blockers, + "runner": { + "provider": "sforge", + "backend": preflight.get("backend"), + "entrypoint": "sforge run", + "agent": agent_label, + "model": model_label, + "run_id": run_label, + "timeout_seconds": timeout_seconds, + "required_secret_names": ["SFORGE_AGENT_API_KEY"], + "optional_endpoint_names": ["SFORGE_AGENT_API_BASE_URL"], + }, + "evaluation": { + "iterative_feedback": True, + "selection": "best_submission_over_time", + "judge_isolated_from_work_container": True, + "official_setting": False, + "leaderboard_eligible": False, + }, + "boundary": _boundary(), + "decision": { + "next_allowed_action": ( + "launch_private_single_task_sforge_run" + if preflight_ready + else "repair_edgebench_preflight_before_launch" + ), + "requires_explicit_launch_authority": True, + "requires_private_credentials": True, + "must_not_claim": [ + "official EdgeBench setting", + "leaderboard comparability", + "task success before compact result ingest", + ], + }, + } + + +def reduce_edgebench_final_result( + *, + task_id: str, + run_id: str, + result: Mapping[str, Any], +) -> dict[str, Any]: + task_label = _public_label(task_id, field="task_id") + run_label = _public_label(run_id, field="run_id") + best_score = _optional_number(result.get("best_score")) + best_pass_rate = _optional_number(result.get("best_pass_rate")) + total_rounds = _optional_non_negative_int(result.get("total_rounds")) + agent_submissions = _optional_non_negative_int(result.get("agent_submissions")) + auto_submissions = _optional_non_negative_int(result.get("auto_submissions")) + runtime_seconds = _optional_number(result.get("runtime_seconds")) + resume_count = _optional_non_negative_int(result.get("resume_count")) + best_round = result.get("best_round") + best_round_label = ( + _public_label(best_round, field="best_round") + if isinstance(best_round, str) and best_round.strip() + else None + ) + + blockers: list[str] = [] + if best_score is None and best_pass_rate is None: + blockers.append("edgebench_best_metric_missing") + if best_pass_rate is not None and not 0.0 <= best_pass_rate <= 1.0: + blockers.append("edgebench_pass_rate_out_of_range") + if total_rounds is None or total_rounds <= 0: + blockers.append("edgebench_submission_rounds_missing") + elif ( + agent_submissions is not None + and auto_submissions is not None + and total_rounds != agent_submissions + auto_submissions + ): + blockers.append("edgebench_submission_rounds_mismatch") + if runtime_seconds is None or runtime_seconds < 0: + blockers.append("edgebench_runtime_missing") + + countable = not blockers + return { + "schema_version": EDGEBENCH_RESULT_SCHEMA_VERSION, + "benchmark_id": EDGEBENCH_BENCHMARK_ID, + "task_id": task_label, + "run_id": run_label, + "countable": countable, + "first_blocker": blockers[0] if blockers else "compact_edgebench_result_ready", + "blockers": blockers, + "metrics": { + "best_score": best_score, + "best_pass_rate": best_pass_rate, + "best_round": best_round_label, + "total_rounds": total_rounds, + "agent_submissions": agent_submissions, + "auto_submissions": auto_submissions, + "runtime_seconds": runtime_seconds, + "resume_count": resume_count, + "timed_out": result.get("timed_out") is True, + }, + "score_policy": { + "selection": "best_submission_over_time", + "official_score_claimed": False, + "leaderboard_comparable": False, + }, + "boundary": { + **_boundary(), + "compact_result_read": True, + "raw_logs_read": False, + "raw_trajectory_read": False, + }, + "decision": { + "next_allowed_action": ( + "ingest_compact_edgebench_result_into_benchmark_ledger" + if countable + else "repair_compact_edgebench_result" + ), + "must_not_claim": [ + "official EdgeBench score", + "leaderboard comparability", + "LoopX treatment benefit", + ], + }, + } diff --git a/loopx/cli_commands/benchmark_dispatch.py b/loopx/cli_commands/benchmark_dispatch.py index b3df26b5d..fb6708178 100644 --- a/loopx/cli_commands/benchmark_dispatch.py +++ b/loopx/cli_commands/benchmark_dispatch.py @@ -28,6 +28,10 @@ handle_benchmark_run_ledger_command, register_benchmark_run_ledger_commands, ) +from .edgebench import ( + handle_edgebench_command, + register_edgebench_commands, +) from .terminal_bench_adapter import ( handle_terminal_bench_adapter_command, register_terminal_bench_adapter_commands, @@ -62,6 +66,7 @@ def register_benchmark_command_group( register_terminal_bench_adapter_commands(benchmark_sub, add_subcommand_format) register_agents_last_exam_commands(benchmark_sub, add_subcommand_format) + register_edgebench_commands(benchmark_sub, add_subcommand_format) register_benchmark_review_lifecycle_commands(benchmark_sub, add_subcommand_format) register_terminal_bench_environment_result_commands(benchmark_sub, add_subcommand_format) @@ -118,6 +123,14 @@ def handle_benchmark_command( if agents_last_exam_result is not None: return agents_last_exam_result + edgebench_result = handle_edgebench_command( + args, + print_payload=print_payload, + output_format=output_format, + ) + if edgebench_result is not None: + return edgebench_result + benchmark_review_lifecycle_result = handle_benchmark_review_lifecycle_command( args, registry_path=registry_path, diff --git a/loopx/cli_commands/edgebench.py b/loopx/cli_commands/edgebench.py new file mode 100644 index 000000000..827677858 --- /dev/null +++ b/loopx/cli_commands/edgebench.py @@ -0,0 +1,205 @@ +from __future__ import annotations + +import argparse +import importlib.util +import json +import platform +import shutil +from collections.abc import Callable +from pathlib import Path + +from ..benchmark_adapters.edgebench import ( + EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS, + build_edgebench_sforge_preflight, + build_edgebench_sforge_run_plan, + reduce_edgebench_final_result, +) + + +PrintPayload = Callable[ + [dict[str, object], str, Callable[[dict[str, object]], str]], + None, +] +OutputFormat = Callable[[argparse.Namespace], str] + +EDGEBENCH_COMMANDS = { + "edgebench-preflight", + "edgebench-run-plan", + "edgebench-result-reduce", +} + + +def _render(title: str, payload: dict[str, object]) -> str: + lines = [ + f"# {title}", + "", + f"- Schema: `{payload.get('schema_version')}`", + f"- Task: `{payload.get('task_id')}`", + f"- Ready/countable: `{payload.get('ready', payload.get('countable'))}`", + f"- First blocker: `{payload.get('first_blocker')}`", + ] + decision = payload.get("decision") + if isinstance(decision, dict): + lines.append(f"- Next action: {decision.get('next_allowed_action')}") + boundary = payload.get("boundary") + if isinstance(boundary, dict): + lines.extend( + [ + f"- No execution: `{boundary.get('no_execution')}`", + f"- Container started: `{boundary.get('container_started')}`", + f"- Model API invoked: `{boundary.get('model_api_invoked')}`", + f"- Upload/submit invoked: `{boundary.get('upload_invoked')}`/`{boundary.get('submit_invoked')}`", + ] + ) + return "\n".join(lines) + "\n" + + +def _load_json_object(path: str) -> dict[str, object]: + payload = json.loads(Path(path).read_text(encoding="utf-8")) + if not isinstance(payload, dict): + raise ValueError("expected a JSON object") + return payload + + +def register_edgebench_commands( + benchmark_subparsers: argparse._SubParsersAction, + add_subcommand_format: Callable[[argparse.ArgumentParser], None], +) -> None: + preflight = benchmark_subparsers.add_parser( + "edgebench-preflight", + help=( + "Check compact EdgeBench/SForge readiness without fetching tasks, " + "pulling images, starting containers, invoking models, uploading, or submitting." + ), + ) + add_subcommand_format(preflight) + preflight.add_argument("--task-id", required=True) + preflight.add_argument( + "--backend", + choices=["docker", "kubernetes"], + default="docker", + ) + preflight.add_argument("--task-catalog-ready", action="store_true") + preflight.add_argument("--task-images-ready", action="store_true") + preflight.add_argument("--judge-ready", action="store_true") + preflight.add_argument( + "--no-environment-probe", + action="store_true", + help="Skip host/SForge/container-runtime probes for dependency-free smokes.", + ) + preflight.add_argument("--require-ready", action="store_true") + + run_plan = benchmark_subparsers.add_parser( + "edgebench-run-plan", + help=( + "Build a private single-task SForge launch plan from a compact " + "preflight. This never launches SForge or reads credentials." + ), + ) + add_subcommand_format(run_plan) + run_plan.add_argument("--preflight-json", required=True) + run_plan.add_argument("--agent", required=True) + run_plan.add_argument("--model", required=True) + run_plan.add_argument("--run-id", required=True) + run_plan.add_argument( + "--timeout-seconds", + type=int, + default=7_200, + help=( + "Bounded task budget. Must not exceed the official 12-hour " + f"budget ({EDGEBENCH_OFFICIAL_TIMEOUT_SECONDS} seconds)." + ), + ) + run_plan.add_argument("--require-ready", action="store_true") + + result = benchmark_subparsers.add_parser( + "edgebench-result-reduce", + help=( + "Reduce an SForge final_result.json into compact public-safe " + "best-score evidence without reading raw logs or trajectories." + ), + ) + add_subcommand_format(result) + result.add_argument("--result-json", required=True) + result.add_argument("--task-id", required=True) + result.add_argument("--run-id", required=True) + result.add_argument("--require-countable", action="store_true") + + +def handle_edgebench_command( + args: argparse.Namespace, + *, + print_payload: PrintPayload, + output_format: OutputFormat, +) -> int | None: + if args.benchmark_command not in EDGEBENCH_COMMANDS: + return None + + try: + if args.benchmark_command == "edgebench-preflight": + probe = not args.no_environment_probe + container_command = "docker" if args.backend == "docker" else "kubectl" + payload = build_edgebench_sforge_preflight( + task_id=args.task_id, + backend=args.backend, + linux_host=(platform.system().lower() == "linux") if probe else False, + sforge_available=( + importlib.util.find_spec("sforge") is not None + or shutil.which("sforge") is not None + ) + if probe + else False, + container_runtime_available=shutil.which(container_command) is not None + if probe + else False, + task_catalog_ready=args.task_catalog_ready, + task_images_ready=args.task_images_ready, + judge_ready=args.judge_ready, + ) + payload["ok"] = not args.require_ready or payload["ready"] is True + if not payload["ok"]: + payload["error"] = payload["first_blocker"] + title = "EdgeBench SForge Preflight" + elif args.benchmark_command == "edgebench-run-plan": + payload = build_edgebench_sforge_run_plan( + preflight=_load_json_object(args.preflight_json), + agent=args.agent, + model=args.model, + timeout_seconds=args.timeout_seconds, + run_id=args.run_id, + ) + payload["ok"] = not args.require_ready or payload["ready"] is True + if not payload["ok"]: + payload["error"] = payload["first_blocker"] + title = "EdgeBench SForge Run Plan" + else: + payload = reduce_edgebench_final_result( + task_id=args.task_id, + run_id=args.run_id, + result=_load_json_object(args.result_json), + ) + payload["ok"] = not args.require_countable or payload["countable"] is True + if not payload["ok"]: + payload["error"] = payload["first_blocker"] + title = "EdgeBench Compact Result" + except (OSError, ValueError, json.JSONDecodeError): + payload = { + "ok": False, + "schema_version": "edgebench_command_error_v0", + "error": "edgebench_input_invalid", + "boundary": { + "no_execution": True, + "container_started": False, + "model_api_invoked": False, + "upload_invoked": False, + "submit_invoked": False, + }, + } + title = "EdgeBench Command Error" + + print_payload( + payload, + output_format(args), + lambda item: _render(title, item), + ) + return 0 if payload.get("ok") else 1