Add Harbor sandbox integration for on-demand PhantomWiki eval

[anmolkabra]

Motivation

PhantomWiki's whole argument is on-demand, contamination-resistant evaluation — yet adopting it today means installing SWI-Prolog, the phantom-wiki package, running the CLI, then learning phantom_eval. The friction pushes people toward the static HuggingFace splits (which defeats the purpose).

This PR ships PhantomWiki as a one-command, fully-Dockerized evaluation via the Harbor framework. A researcher needs only Docker + pip install harbor; every run generates a fresh universe inside the containers and evaluates a frontier model on it.

How it works

run_adapter.py  ──►  N Harbor tasks (one per question), one fresh universe (seed S)
                          ├─ environment/  (AGENT image)
                          │     pw-generate --seed S → keep /corpus + /app/question.txt
                          │                            strip Prolog DB + gold answers
                          └─ tests/        (separate VERIFIER image)
                                pw-generate --seed S → keep gold
                                test.sh: score /logs/artifacts/answer.txt → F1
                                         → /logs/verifier/{reward.txt,result.json}

• One task per question — the Terminal-Bench-faithful mapping; Harbor aggregates per-task rewards into mean F1. The N task dirs are generated (git-ignored), not hand-maintained.
• Generation runs in Docker, not on the host. The environment/ and tests/ Dockerfiles install Prolog + phantom-wiki and run pw-generate.
• Gold answers never reach the agent. Since the agent has a root shell, we use Harbor's separate verifier environment ([verifier] environment_mode = "separate"): a second image, built from tests/, regenerates the same universe to hold the gold. Harbor forwards the agent's /logs/artifacts/ into it for scoring.
• Determinism makes the two generations safe — both images share the same base + pinned phantom-wiki + seed (single-threaded), so the corpus the agent sees and the gold the verifier scores match. Covered by a determinism self-test.

Usage

# fresh dataset (random seed); no Prolog on the host
python harbor/adapters/phantom-wiki/run_adapter.py --output-dir harbor/runs/pw --num-tasks 50
# evaluate any model via the cheap Terminus agent
harbor run -p harbor/runs/pw -a terminus-2 -m anthropic/claude-opus-4-1 --n-concurrent 8
# F1-vs-difficulty diagnostics
python harbor/diagnostics/plot_difficulty.py --run-dir <harbor-run-output-dir>

Once registered (adapters/phantom-wiki/dataset.toml), this collapses to a single harbor run -d phantom-wiki@1 -a terminus-2 -m <model>.

What's included

• harbor/adapters/phantom-wiki/ — adapter (adapter.py/run_adapter.py), task template/, shared in-container select_question.py + score_answer.py, dataset.toml.
• harbor/diagnostics/plot_difficulty.py — F1-vs-difficulty + F1-by-type plots from the run logs.
• tests/harbor/ — unit tests for the selector, scorer, and adapter (18 passing), plus a determinism self-test (auto-skipped where Prolog isn't installed). Lint-clean under the repo's black/isort/flake8 config.

⚠️ Two integration points to verify against the installed Harbor version

I authored this against the public Harbor docs but could not run Harbor/Docker end-to-end in the build sandbox, so please confirm during review:

1. Verifier image build source — assumes environment_mode = "separate" builds the grading image from the task's tests/ dir. If your Harbor expects a prebuilt [verifier.environment].docker_image, the adapter can docker build --tag it instead.
2. Artifact forwarding — assumes Harbor copies the agent's /logs/artifacts/ into the separate verifier container. If the path/declaration differs, adjust instruction.md + tests/test.sh.

A no-code-change fallback exists if separate-verifier plumbing isn't available: environment_mode = "shared" with test.sh itself re-running pw-generate --seed $SEED to recover the gold after the agent finishes (the determinism test guarantees it matches). Documented in harbor/README.md.

https://claude.ai/code/session_019t4Xt48jUEATKWG9A17UTa

---

Generated by Claude Code