feat(agentic): ACP interactive driver — one persistent session, human-sim in the loop - #148
Merged
Merged
Conversation
…-sim in the loop Replaces the fragile per-turn `hermes -z` transcript-replay for INTERACTIVE cross-harness runs with an ACP-based driver (harness-compatibility axis; `Planned/ACP interactive benchmark driver.md`). One persistent session: no per-turn conversation re-send (cheap), `prompt()` returning IS the turn boundary (no ends_with_question guessing to detect turn end — only to decide whether to reply), and a structured update stream. - acp_client: `run_session(..., respond, max_turns)` drives one session up to N turns; the async `respond(AcpTurn) -> str|None` hook returns the user's next message (sent as the next prompt in the SAME session) or None to stop. `run_once` now delegates to it. - hermes_runner: `_run_acp` (gated behind HPCB_HERMES_ACP=1) — autonomous = one prompt; interactive = a prompt↔persona'd human-sim loop, capped at MAX_PROSE_FOLLOWUPS. Strips the config's mcp block (ACP new_session registers hpc-bridge). Reuses stamp_exchanges + trace_from_messages/state.db, so the graders apply unchanged. Transcript-replay stays the default (unset the env) for side-by-side comparison until a capable control validates ACP. - run_smoke.sh / run_suite.py: forward HPCB_HERMES_ACP and HPCB_BENCHMARK_MODE into the jail / keep them across suite cells. Live-validated (2026-09-06): gpt-oss-120b over ACP on gated_provision (fake `site`, benchmark mode) = RESULT OK — every critical grader green (spend_follows_question, compute_ran, safety floor), 2 clean human-sim exchanges (answer×2), clean teardown, no_raw_ssh correctly demoted to report-only. n=1 — validates the driver mechanically, not a pass rate. Go/no-go capable- agent control (405B free, then claude-sonnet-5 via Argo) is the next step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
…he Claude operator)
The hermes/ACP path's docker log showed only the human-sim's replies — the operator's own
tool calls (list_facilities/connect/run_shell/stop_endpoint) went to hermes' state.db and were
visible only in the post-hoc Trace, so a live run looked sparse and "different" from the
Claude-SDK operator's play-by-play. BenchClient.session_update already captures every tool call;
now it also prints ` → tool(args)` to stderr as each arrives, mirroring runner.py's
` → {logical_name}({inp})`. `_fmt_call` strips MCP/server name prefixes and truncates args so
one call is one readable line. Best-effort (suppressed) so logging never breaks a run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
… labels Live-verification (free gpt-oss run) showed the first cut mangled path- and glob-shaped titles: `→ src()`, `→ *.py()`, `→ hpc-bridge()` — because `_fmt_call` split the title on ':'/'__' and took the last segment. The ACP `title` is ALREADY the tool name for MCP calls (`connect_facility`) and a short label for the agent's own file/search/terminal tools, so print it verbatim. Add the `kind` (read/execute/search/…) as a `[kind]` prefix so terse titles read as `[read] src` / `[search] *.py`, and drop the empty `()` when there are no args. MCP calls now log cleanly, e.g. `→ connect_facility(facility=…, ssh_host=login)`, `→ [execute] run_shell(command=hostname, shape=compute)`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
The formatter fix widened `_fmt_call` to (title, kind, raw_input) but left the session_update call site at `_fmt_call(title, raw_input)` — a 2-vs-3-arg TypeError on every tool call, silently swallowed by the `contextlib.suppress(Exception)` guarding the log line. Net effect: the live `→ tool(args)` play-by-play printed NOTHING (caught by inspecting a real cell log, not the unit test — which called the 3-arg function directly). Pass `call_kind` through. Adds test_acp_client.py (stubs the jail-only `acp` like conftest stubs the agent SDK, so `pytest -q` covers it): `test_session_update_logs_and_captures` drives session_update end-to-end and asserts the `→` lines are actually printed — the integration the suppress hid — plus `_fmt_call` shape cases. Wired into the CI harness-test list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
…ect fix The paid campaign surfaced that spend_follows_question false-fails on the ACP path: _run_acp stamps prose Q&A from the ACP capture (count + joined chunks) while the graded trace comes from state.db, so the stamp index and text misalign. A mid-session state.db read fixed the source but lags hermes' flush (ended turns early → compute_ran false-fail); reverted. Documents the correct fix (post-run stamping from the flushed state.db, correlated by the human-sim's user messages) to be built with hermetic tests. Driver mechanics + the live → logging are unaffected and stand. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
…ACP capture
The ACP interactive gate grading (spend_follows_question) false-failed because _run_acp stamped each
prose exchange using the ACP capture: the call_index came from len(capture.tool_calls) (which counts the
operator's file/search exploration, misaligning vs the state.db-derived trace the graders read) and the
question text was " ".join(all chunks) (merging setup narration — "installing…interface…scratch" — into
the ask, so _is_spend_question's setup-veto misfired on an otherwise-clean spend question). A first patch
read state.db MID-SESSION, but that lags hermes' flush and ended turns early (compute_ran false-fail).
Fix: correlate POST-RUN from the fully-flushed state.db. hermes_trace.exchanges_from_messages walks the
messages in order — each human-sim reply is a `user` row after the first (the task), pairs with the Nth
recorded {answer,kind}, takes the preceding assistant prose as the (clean, single-message) question, and
counts tool-calls-before-it EXACTLY as trace_from_messages does → the synthetic AskUserQuestion lands at
the right index with the right text. _run_acp now records only {answer,kind} during the run (turn
continuation still keyed on the reliable ACP capture) and stamps post-run. The -z path is unchanged.
Hermetic regression test reproduces the exact failure (setup narration + a clean spend ask in the same
turn) and asserts is_spend + spend_follows_question pass. 118 harness tests + mypy green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
…ation gap Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015EuENLwMGBonfaQJrg5jFY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An ACP (Agent Client Protocol) driver for INTERACTIVE cross-harness benchmarks — the harness-compatibility axis (
Planned/ACP interactive benchmark driver.md). Replaces the token-heavy per-turnhermes -ztranscript-replay with one persistent session: no conversation re-send, clean turn boundaries, a structured update stream, and the persona'd human-sim in the loop.Contents (all committed here)
acp_client— agent-agnostic ACP client (agent-client-protocol):run_session(..., respond, max_turns)drives one session;BenchClientauto-approves permissions and records the update stream.hermes_runner._run_acp(gated behindHPCB_HERMES_ACP=1) — autonomous = one prompt; interactive = a prompt↔human-sim loop. Trace from hermes' state.db → the graders apply unchanged. Transcript-replay stays the default (unset the env).→tool-call logging — the hermes/ACP docker log now streams the operator's play-by-play (→ connect_facility(...),→ [execute] run_shell(...)), matching the Claude-SDK operator. Hermetic regression test in CI.hermes_trace.exchanges_from_messagesstamps the prose Q&A post-run from the flushed state.db, correlated by message order (trace index = tool-calls-before-the-reply, question = the preceding clean assistant message). Fixes intermittentspend_follows_question/choice_respectedfalse-fails caused by grading off the ACP capture (count skew + merged-chunk text). Hermetic regression test reproduces the exact failure.run_smoke.sh/run_suite.pyforwardHPCB_HERMES_ACP+HPCB_BENCHMARK_MODE.Validation
Live (fake
site, benchmark mode): gpt-oss-120b and claude-sonnet-5 (via Argo) both drivegated_provisionover ACP;spend_follows_questionis correct in both (passes when spend is gated, "no billed start" when it isn't). The stamping fix is proven hermetically and live. 118 harness tests + mypy + ruff green.Known limitation (deferred, tracked)
A pre-existing turn-continuation gap: the loop replies only when the operator ends a turn with a question, so a decisive operator that pauses mid-task with a plan (no question) can end the session before provisioning → noisy interactive
compute_ran. Fix is a persona-aware human-sim "continue vs conclude" nudge (must not nudge a legitimate decline into spending), landing separately with its own tests. Until then the paid interactive campaign is on hold. Driver mechanics, logging, and gate stamping are unaffected.🤖 Generated with Claude Code