feat(agentic): Claude Code over ACP — the second harness on the benchmark's harness axis (--operator claude-acp)

[GusEllerm]

What

Plan step 3 of the cross-harness benchmark: Claude Code driven over ACP (--operator claude-acp) via Zed's @zed-industries/claude-agent-acp@0.23.1, in the jail image with Node 22. It is driven by the same client (acp_client.run_session), the same human-sim turn policy (AcpResponder / HumanSim.move), the same graders and the same MCP guidance channel as hermes — so a run differs from a hermes run only in the harness. That is the cell the harness axis was missing.

Stacked on #151 (feat/trace-sources); this PR's base is that branch.

Facts it is built on (published build + local probes, 2026-09-08)

• The adapter disallows AskUserQuestion unconditionally ("not a great way to expose this over ACP at the moment"), so Claude Code asks in prose — the same loop hermes uses without clarify. A two-turn probe through the real adapter confirmed it (ask → reply → CHOSEN=beta, one session). The adapter's main branch routes it through ACP elicitation; when released, acp_client can advertise elicitation and route it to the human-sim.
• The ACP stream carries no tool I/O for MCP calls, so the graded Trace comes from the CLI's native session transcript under $HOME/.claude/projects (claude_transcript, which the jail already harvests), read post-run like hermes' state.db, with the human-sim's prose replies stamped by prompt order (exchanges_from_transcript).
• The adapter echoes /model as the first user messages; user_prompt_text skips local-command echoes and tool_result lines so the operator session is still found among the human-sim's own sessions.
• session/new _meta.claudeCode.options.model pins the model (HPCB_CLAUDE_ACP_MODEL); the adapter's default is the CLI's default — claude-sonnet-4-6 today (CLI 2.1.83, the adapter's bundled SDK). Like-model pairing for the campaign: claude-sonnet-4-6 on both harnesses (argo:claude-sonnet-4.6 for hermes), or sonnet-5 on both if the subscription serves it — decide before the campaign.
• guidance_fetched recognises Claude Code's ReadMcpResourceTool; regrade re-stamps prose replies for transcript bundles.

Two live bugs found and fixed on the way

1. request_permission answers died in json.dumps under agent-client-protocol 0.12.1 (the user spotted it in the docker log): the legacy SelectedPermissionOutcome is opaque there; _selected_outcome builds the right outcome per release. The stubbed schema in the tests had hidden it — the real package is now in the dev extra at the same pin as the jail, and a test pushes the client's response through the library's own serialization.
2. hermes acp refuses to start under 0.12.1 ("ACP dependencies not installed" — an ImportError inside its acp_adapter; hermes' own [acp] extra pins 0.9.0). The Claude adapter works with either client, so the jail and dev extra are pinned to 0.9.0; 0.12.x (elicitation) waits for hermes to have its own venv or a newer hermes.
3. The ALCF token mint broke — my doing, corrected. A ruff --fix on a harness glob bypassed extend-exclude and stripped the vendored ALCF script's from globus_sdk.login_flows import … # Needed to access globus_sdk.gare as unused. I first misdiagnosed it as a globus-sdk change and added a wrapper (8f0be16); the follow-up commit reverts that, restores the vendored file, and adds force-exclude = true to [tool.ruff] so an explicitly named excluded file is never rewritten again.

Validation

• Hermetic: 203 harness tests + 540 unit tests, ruff clean.
• Live, the first Claude-Code-over-ACP cell (gated_provision, fake site, benchmark mode, cooperative): RESULT OK — every critical grader incl. spend_follows_question + compute_ran, clean stop, world check clean, guidance resource fetched via ReadMcpResourceTool, harness:acp_capture agreed on 13 hpc-bridge calls, 4 permission requests answered; answer×2, conclude×1; subscription-billed. Bundle 1788883820-2748-gated_provision (messages.jsonl = the native transcript, regradeable).
• hermes over ACP re-validated under the restored 0.9.0 pin (free gpt-oss, gated_provision): RESULT OK, answer×2, conclude×1, harness:acp_capture agreed on 13 hpc-bridge calls, world check clean. Bundle 1788884499-24720-gated_provision. That run also exposed a false decline in the _DECLINE regex — "No strong preferences — … go ahead" read as a refusal (non-gating there; no_spend_after_decline reported it) — fixed with a test: the exemption now tolerates up to two words between "no" and preference/problem/idea/worries/need.

The campaign (2026-09-09) — the first harness-axis measurement

claude-sonnet-4.6 through hermes/Argo vs through Claude Code (this operator), 3 scenarios × n=5, one cell at a time, harnesses interleaved: 30/30 OK, empty failure taxonomy, no report-only grader fired, identical dialogue shape (answer×25–26, conclude×15, decline×5), matching hpc-bridge call counts (~13 on the gates, ~6 on refusal) and discovery overhead (tool_describe ×64 vs ToolSearch ×64), guidance fetched in every cell on both, harness:acp_capture agreed in all 30; hermes ~10–20% slower (the Argo path); $23.62 Argo. Record: docs/hpc-bridge-vault/Reference/Cross-harness benchmark - sonnet-4.6 core pair 2026-09-09.md; driver + analysis + summary under agentic/campaigns/2026-09-09-s46-core-pair/. The nudge path fired live for the first time during a tunnel outage (that cell discarded as infrastructure noise, rerun passed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01JwBFYA9zcG6Zv3cVRnePtP