feat(agentic): ACP turn-continuation — the human-sim decides every turn (reply / nudge / conclude) - #150
Merged
Merged
Conversation
…rn (reply / nudge / conclude)
The ACP driver replied only when a regex said the operator's turn "ends with a question". A decisive operator
(sonnet-5) that brought up the login node, ran sinfo and ended its turn on a PLAN got no reply, the session
ended, and `compute_ran` false-failed. Now `HumanSim.move` judges EVERY operator turn in persona:
- reply — it asked, or set out a concrete step and is waiting for a go-ahead (anything that would start or
pay for compute is always a reply, decided per persona)
- nudge — a mid-task pause with nothing to decide → "carry on"
- conclude — goal met / declined and wrapped up → end the session
Deterministic guards in the sim (hermetically tested): a STANDING decline is never nudged — only a later answer
to a SPEND-ish question supersedes it (the grader's own `_is_spend_question`, so guard and grader agree; a
config answer after a decline does not, as seen live); nudges have their own budget (`MAX_NUDGES`, separate from
`MAX_PROSE_FOLLOWUPS`, which moves to human_sim) so a decisive-but-chatty operator is not scored as looping; each
budget ends in `conclude`; the parse fallback is the neutral reply, never a nudge or an approval.
Stamping: a nudge is a user row, so it is recorded for the post-run correlation, but `stamp_exchanges` stamps it
as a `user_nudge` marker — never an AskUserQuestion — because "next I'll provision a debug node" + "carry on"
would otherwise satisfy `spend_follows_question` through the spend-ish regex. A reply at a proposal-pause is
stamped as a question (the operator put the spend to the user and yielded).
`hermes_runner.AcpResponder` replaces the closure; `harness:prose_followups` reports nudges (the answer cap still
gates); the transcript labels a pause / wrap-up as such. New `test_hermes_runner.py` (in CI) drives the loop
against the real sim with only `_ask` scripted.
Live (free, gpt-oss over ALCF, fake site, benchmark mode): gated_provision OK (answer×1, conclude×1) and
spend_refusal OK (decline×1, answer×1, conclude×1; nothing billed). The nudge path is exercised hermetically;
its live check is the one paid sonnet-5 run still owed before the campaign.
Also records the refined objective (2026-09-08): like models through a VARIETY of harnesses over one ACP
driver, and the ordered plan that follows, in the vault plan doc.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JwBFYA9zcG6Zv3cVRnePtP
… human-sim was reading "part ition") Over the Argo tunnel hermes streams, so a turn's AgentMessageChunk texts are token deltas; run_session joined them with ' ' and the human-sim's prompt carried "part ition", "sp ending", "c ost" as the operator's ask (sonnet-5 via Argo, paid validation 2026-09-08 — it still judged correctly, by luck). `_join_chunks` concatenates deltas verbatim and inserts a newline only between two whole (non-streamed) messages that would otherwise fuse. Grading was never affected: the graded question comes from state.db post-run. Hermetic tests for both shapes. Records the paid validation in the plan doc: sonnet-5 gated_provision RESULT OK (answer×3, conclude×1, compute_ran PASS, $1.73 metered). sonnet-5 asked at every step this time, so the nudge path remains hermetically validated only — stated as such. The campaign gate is met. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JwBFYA9zcG6Zv3cVRnePtP
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The ACP interactive driver's turn-continuation blocker (HANDOFF §4). The human-sim now decides every operator turn —
reply/nudge/conclude— instead of replying only whenends_with_questionmatched. A decisive operator that ends a turn on a plan gets a "carry on"; a wrap-up ends the session; a standing decline is never nudged.Why
sonnet-5 over ACP brought up the login node, ran
sinfo, ended its turn on "I'll provision debug next" — no question — got no reply, andcompute_ranfalse-failed. gpt-oss completed only because it keeps asking. The paid campaign has been on hold on this.How
human_sim.HumanSim.move— one in-persona judgement per turn (sees its own prior replies; still context-isolated), then deterministic guards:_is_spend_question, so guard and grader agree — a config answer after a decline does not, as seen live)MAX_NUDGES), separate from the answer cap (MAX_PROSE_FOLLOWUPS, moved tohuman_sim) — a decisive-but-chatty operator is not scored as loopingconclude; parse fallback = the neutral reply, never a nudge or an approvalhermes_trace.stamp_exchanges— a nudge is stamped as auser_nudgemarker, never an AskUserQuestion ("next I'll provision a debug node" + "carry on" would otherwise satisfyspend_follows_questionvia the spend regex). A reply at a proposal-pause IS a question.hermes_runner.AcpResponderreplaces the closure;run.pyreports nudges inharness:prose_followups(answer cap still gates) and labels pauses / wrap-ups in the transcript.test_human_sim.py(policy + guards, incl. the live-observed config-answer case),test_hermes_trace.py(nudge ≠ question — the false-pass guard), newtest_hermes_runner.py(loop-level, real sim with_askscripted; added to CI).Validation
site, benchmark mode):gated_provisionOK (answer×1, conclude×1, all criticals) andspend_refusalOK (decline×1, answer×1, conclude×1, nothing billed,refusal_exercised+no_spend_after_declinepass). Bundles1788878972-93059-gated_provision,1788879164-97189-spend_refusal.gated_provisionOK —answer×3, conclude×1, all criticals incl.compute_ran, guidance resource fetched, $1.73 metered. Stated plainly: sonnet-5 asked at every step this time, so thenudgepath is validated hermetically (loop-level test) but not yet observed live — run-to-run variance; the policy demonstrably does not disturb a capable operator that asks. Bundle1788880401-22038-gated_provision.acp_client._join_chunksnow concatenates deltas verbatim (a newline only between two whole messages that would fuse). Grading was never affected (questions come from state.db post-run). Tests for both shapes.Also
Records the refined objective (2026-09-08) in the vault plan doc: like models through a variety of harnesses over one ACP driver + one human-sim, and the ordered plan (update-stream trace → Claude Code via
claude-agent-acp→ campaign n=5 →spend_revokedviasession/cancel→ judge-agreement pass → elicitation spike).🤖 Generated with Claude Code
https://claude.ai/code/session_01JwBFYA9zcG6Zv3cVRnePtP