feat(agentic): trace sources for the cross-harness benchmark — Claude CLI transcript reader, persisted ACP event log + cross-check, regrade for every bundle

[GusEllerm]

What

Plan step 2 of the cross-harness benchmark, reshaped by evidence: the trace SOURCES each ACP-driven operator is graded from, plus the instrument check that keeps them honest.

Why the reshaping

Step 2 was "build the Trace from the ACP session/update stream" so a second agent without hermes' state.db could be graded. Reading both adapters' source rules that out as a grading source:

• hermes acp_adapter/tools.py: raw_output=None for any JSON tool result (a truncated rendering goes into content).
• Zed claude-agent-acp tools.ts: sets neither rawInput nor rawOutput; an MCP call gets only its name.

Confirmed live in this PR's validation run: every tool_call_update arrived with raw_output: null. Each agent has a full-fidelity post-run store instead — hermes' state.db (already used) and, for Claude Code, the CLI's native session transcript ($CLAUDE_CONFIG_DIR/projects/<slug>/<session>.jsonl), which the jail's entrypoint already harvests into the bundle.

What landed

• claude_transcript.py — the Claude-side twin of hermes_trace: native transcript → Trace (tool_use/tool_result paired, assistant text, thinking skipped, sidechains optional). AskUserQuestion answers come from the line's toolUseResult.answers when the CLI recorded them structurally, else the graders' existing fallback on the rendered "q"="a" text. select_operator_session picks the operator out of a harvest that also holds the human-sim's own SDK sessions (their first message is the role-play prompt). Prerequisite for the Claude-Code-over-ACP cell (step 3).
• AcpCapture.events — the client's flat, JSON-able event log (user_prompt / message_chunk / tool_call / tool_call_update / permission / turn_end, per prompt turn), persisted into the bundle as acp-updates.jsonl (provenance.write_run_record(extra_jsonl=…)).
• acp_trace.capture_crosscheck → harness:acp_capture (report-only until proven clean): the ordered hpc-bridge tool NAMES in the stream vs in the graded trace — the one thing every adapter carries. A mismatch means the post-run source lagged, truncated or picked the wrong session (the 2026-09-07 mid-run state.db read would have shown here).
• regrade.bundle_trace — format sniffing so every bundle regrades: SDK dict-form, hermes rows (an ACP bundle re-stamps the human-sim's prose exchanges from the record's dialogue by message order), Claude CLI transcript. -z transcript-replay bundles replay trace-only. And regrade now honours the recorded benchmark mode (the preference graders were report-only live, so they no longer decide the replayed verdict).

Validation

• Hermetic: 193 harness tests (4 new files, registered in CI) + 226 unit tests; ruff clean.
• Live (free, gpt-oss over ACP, fake site, benchmark mode): gated_provision RESULT OK; acp-updates.jsonl persisted (42 events, 3 turns); harness:acp_capture agreed with the graded trace on 8 hpc-bridge calls. Bundle 1788881977-55550-gated_provision.
• Offline: regrade over all 568 stored bundles. The ACP-era hermes bundles re-stamp correctly — the sonnet-5 run the old stamping bug had false-failed now regrades PASS on spend_follows_question. An INTERRUPTED bundle (rc 130, zero messages) regrades FAIL on agent_engaged, correctly.

Open for step 3

How claude-agent-acp surfaces AskUserQuestion to the client (it routes canUseTool to session/request_permission; whether the question's options become permission options is not visible in the excerpt read) — verify on the first run and route it to the human-sim rather than auto-approving. Node returns to the jail image.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JwBFYA9zcG6Zv3cVRnePtP