Phase 3: Local End-to-End Scripted Run

[rmax]

Description

Wire everything together for a complete end-to-end scripted run against the payment webhook benchmark. Implement Docker sandbox, actual evaluator commands, the payment webhook benchmark repository, hidden tests, the controller assembly, and reports.

Key Deliverables

1. Docker Sandbox — container isolation, resource limits, read-only mounts
2. Payment Webhook Benchmark — FastAPI service with duplicate-delivery bug
3. Visible Tests — pytest tests that check basic webhook behavior
4. Hidden Tests — concurrent delivery, transaction rollback, idempotency
5. Controller Assembly — complete controller that orchestrates the full pipeline
6. Completion Manifest — tamper-evident manifest with all run metadata
7. Reports — JSON report, Markdown report, terminal summary
8. Benchmark Validation Script — validates benchmark integrity

Acceptance Criteria

• Docker sandbox executes commands with resource limits
• Protected tests mounted read-only
• Payment webhook benchmark has an intentional duplicate-delivery bug
• At least one visible test initially fails
• Hidden tests are not writable by the sandbox
• Controller orchestrates: goal → sandbox → repo → eval → decision
• Scripted agent run produces a completion manifest
• Artifact chain verifies correctly
• Markdown and JSON reports are generated
• Benchmark validation confirms all integrity checks
• Policy enforcement test: agent attempts protected edit → rejected
• False completion test: model claims completion, hidden tests fail → refused
• Budget exhaustion terminates cleanly

Depends on: #1, #3

[rmax]

Phase 3 Progress

✅ Complete

• Phase 2: Full deterministic core complete — 33 unit tests, all modules implemented

🔄 In Progress

• Batch 1: Docker sandbox + Controller assembly + Completion manifest (Codex running)

⏳ Queued

• Batch 2: Payment webhook benchmark (source repo, visible/hidden tests)
• Batch 3: Reports, CLI commands, benchmark validation, e2e tests