feat(acp): park, pause, breaker and replay in the agent harness (T16)

[wiggdevin]

Summary

This branch closes T16: park, pause, breaker, and replay in the ACP agent harness, per
docs/plans/2026-09-06-harness-reliability-design.md. Commits on the branch:

• b4ce86acb — docs(zs): harness reliability, agent health and curator designs (T16 design doc + related plans)
• 6977f9c44 — feat(acp): park, pause, breaker and replay in the agent harness (T16)
• b42de461c — wip: uncommitted agent work at session stop 2026-09-06 05:30 (fix round)
• 3d8281fee — fix(acp): park semantics in the retry tests, clippy, and the replay tests
• c691edb83 — fix(acp): close the verified Sol findings on the reliability harness
• 19c4cbe89 — fix(acp): close the delta findings on the reliability harness (T16)

What the harness now does, per the design doc's decisions:

• Park: a batch that cannot be delivered (retries exhausted, hard timeout, auth error,
  breaker expiry) is written durably to state/parked.jsonl instead of being discarded.
  A batch that had already started producing agent output/tool calls is marked
  needs_review and requires an operator to retry or discard it; a batch that never
  started is replay-eligible.
• Pause: a CapacityExhausted error (session limit, rate limit, 429, overloaded, quota)
  pauses the agent until the parsed reset time (clamped to at most 6 hours), rather than
  burning the retry budget and dead-lettering the message.
• Breaker: three consecutive ProviderInternal/Unknown failures on one scope open a
  breaker that probes every 10 minutes for up to 6 hours before parking.
• Replay: after a pause or breaker recovers via a successful probe turn, not-started
  parked batches for that scope replay oldest-first, framed with a "delivered late" section
  header; started batches never auto-replay.
• Ledger: an append-only state/ledger.jsonl records every state transition
  (turn_started, turn_finished, batch_parked, batch_replayed, agent_paused,
  breaker_opened, etc.) with 30-day retention and size caps.
• State dir: BUZZ_ACP_STATE_DIR (set by desktop at spawn, falling back to
  ~/.buzz/.state/<pubkey-prefix-16>/) holds the ledger and park file at 0700/0600
  permissions.
• Desktop wiring: the desktop sets BUZZ_ACP_STATE_DIR when spawning a managed agent
  and reserves it against user override (managed_agents/reserved_env_keys.rs,
  managed_agents/runtime.rs).

No message is discarded by the harness on this branch's happy path; every failure mode
routes to park, pause, or the breaker instead of the old 10-retry-then-drop behavior.

What changed

By area, from git diff --stat origin/zs/main...HEAD (26 files, +8289/-207):

• crates/buzz-acp/src/reliability/* (new, ~3.5K lines) — the reliability subsystem:
  error_class.rs (failure classification), ledger.rs (append-only durable ledger),
  park.rs (durable park file, replay eligibility, scope caps), runtime.rs (pause/breaker
  state machine, replay orchestration), state.rs (per-scope/per-agent state), state_dir.rs
  (state directory resolution and permissions), notices.rs (channel notice templates).
• crates/buzz-acp/src/lib.rs (+2698/-…) — wires reliability into the main dispatch
  loop: pause/breaker gates, park-or-fallthrough on failure, replay-after-success, control
  frame handling (replay_batch, discard_batch, resume_now, keep_paused).
• crates/buzz-acp/src/queue.rs (+470/-…) — requeue_preserve_timestamps, replay
  staging (stage_replay), scope-aware carryover handling.
• crates/buzz-acp/src/pool.rs (+137/-…) — failure notice posting to the channel
  (pause/park/breaker templates).
• crates/buzz-acp/src/acp.rs (+164 lines) — turn_saw_output/started-state wiring
  from the wire protocol (session/update, Goose usage updates).
• desktop/src-tauri/src/managed_agents/* — runtime.rs, storage.rs,
  reserved_env_keys.rs, plus their test files: sets and reserves BUZZ_ACP_STATE_DIR when
  spawning a managed agent.
• Cargo.toml / Cargo.lock / desktop/src-tauri/Cargo.lock — new dependency for the
  reliability crate additions.
• docs/plans/* — the T16 design doc plus three related design docs from the same
  session (2026-09-04-zs-implementation-plan.md, 2026-09-06-agent-health-design.md,
  2026-09-06-agent-self-improvement-design.md) landed alongside it.

Gates run

No full suite ran locally on this branch — only targeted/scoped gates below plus the
repo's own pre-push hook (lefthook) on the final push. The merge queue runs the full
suite; this PR relies on that run for full-repo coverage.

From C4-t16-acp-tests.txt (pre-second-fix-round baseline, before the Sol-verified fixes landed):

• cargo nextest run -p buzz-acp — not completed in this run (superseded by cargo test)
• cargo test -p buzz-acp — exit 101 (failing, pre-fix state)
• cargo clippy -p buzz-acp --all-targets -- -D warnings — exit 101 (failing, pre-fix state)
• cargo fmt -p buzz-acp -- --check — exit 0

From C4-t16-acp-verify.txt (post-fix re-verify):

• cargo fmt -p buzz-acp -- --check — exit 1 (formatting diffs found), then fmt applied
• cargo clippy -p buzz-acp --all-targets -- -D warnings — exit 0
• cargo test -p buzz-acp reliability — exit 0
• cargo test -p buzz-acp queue — exit 0
• cargo test -p buzz-acp hard_timeout (post-fmt re-verify) — exit 0
• cargo clippy -p buzz-acp --all-targets -- -D warnings (post-fmt re-verify) — exit 0

From C4-t16-tauri-tests.txt:

• cargo test managed_agents — exit 0 (87s)
• cargo clippy --all-targets -- -D warnings — exit 0 (111s)
• cargo fmt -- --check — exit 0 (2s)
• Summary line: all gates exit 0: ok=true

From T16-fix-verify.txt (final fix-round verification, buzz-acp target dir then Tauri target dir):

• cargo fmt -p buzz-acp -- --check — exit 0
• cargo clippy -p buzz-acp --all-targets -- -D warnings — exit 0
• cargo test -p buzz-acp reliability — exit 0
• cargo test -p buzz-acp queue — exit 0
• cargo test -p buzz-acp hard_timeout — exit 0
• cargo test managed_agents (TAURI target dir) — exit 0 (1379 passed, 0 failed)
• cargo clippy --all-targets -- -D warnings (TAURI target dir) — exit 0
• cargo fmt -- --check — exit 0

Pre-push hook (lefthook) on the final push to origin/feat/harness-reliability:

• push-head-scope — pass
• branch-skew — pass
• file-size-check — pass (10 file-size-core tests, all passing)
• desktop-check — pass
• desktop-typecheck — pass
• rust-tests — pass (all 12 workspace crates, including buzz-acp at 25 tests, buzz-mcp-launch, etc.)
• desktop-tauri-checks — pass
• desktop-test — pass (6546 tests, 0 failed, 86 suites) — note: the first push attempt
  failed this gate with 5 test files hitting ERR_MODULE_NOT_FOUND on lucide-react
  because desktop/node_modules was stale in this worktree relative to the merged base
  (the base merge added the dependency). Root cause confirmed environmental, not a code
  defect: pnpm install --frozen-lockfile in desktop/ (no lockfile change) resolved it,
  and the retried push passed this gate cleanly on the same commit.

Tested base OID

d0cbd2ebda9cbc0f8b56f1e7dd425f76e4a141da (git merge-base HEAD origin/zs/main).
origin/zs/main was already an ancestor of HEAD at PR time — no merge was needed.

Sol verdicts

GPT-5.6 Sol ran three adversarial static review passes against this branch. Sol's sandbox
is read-only and cannot execute the test suite, so its closure judgments below are based on
static code/test-seam tracing, not runtime verification.

Full pass (T16-sol-verdict.json, at 3d8281fee): verdict BLOCK, 25 findings total
— 21 BLOCK, 4 WARN.

Verification (T16-sol-verified.md): each of Sol's 25 findings was independently
re-opened against the actual file/line at HEAD and checked against the design doc.
Result: 19 of 21 BLOCK findings confirmed as-is, 2 downgraded BLOCK→WARN (#6 — a
liveness stall requiring five independent config knobs at their extreme, not data loss;
#21 — the symlink/length hardening gap on BUZZ_ACP_STATE_DIR is defense-in-depth for
buzz-acp-as-a-library, since the shipped desktop topology always overwrites the value
from a trusted source). The 4 original WARNs were all reconfirmed as WARN. Net: 19
confirmed BLOCK + 2 downgraded BLOCK→WARN + 4 confirmed WARN = 21 total confirmed findings
requiring changes, 0 refuted.

Delta #1 (T16-sol-delta1.json, after the first fix commit c691edb83): verdict
BLOCK, 22 findings — 16 BLOCK (7 new + 9 sub-findings against 8 prior BLOCK
categories that remained unresolved), 6 WARN. All 16 BLOCK findings were confirmed and
fixed in the second fix commit (19c4cbe89).

Delta #2 (T16-sol-delta2.json, after the second fix commit 19c4cbe89): verdict
BLOCK, 25 findings — 17 BLOCK, 8 WARN. Sol's summary: "2 new BLOCKs and 15
unresolved or reopened prior BLOCK findings... Closure evidence supports original findings
#1, #2, #3, #7, #10, #11, #16, #17, #18 and DELTA-1 findings #2 and #7." Sol could not
execute tests in its read-only sandbox for this pass either, so these closure judgments are
static-inspection-based, not test-verified.

The branch is NOT clean on Sol. The fix loop hit its cap at delta #2 with 17 BLOCK and
8 WARN findings still open. These are listed verbatim below.

Follow-ups (open Sol findings at the loop cap)

BLOCK

• crates/buzz-acp/src/reliability/park.rs:662 — NEW: corrupt-record quarantine grows without bound. Trigger: place one malformed park record in parked.jsonl and repeatedly restart the harness. read_batches excludes but never removes the source record, while quarantine_line appends it to parked.jsonl.corrupt on every open with no byte cap, rotation, or deduplication. Fix: use a bounded, atomically replaced quarantine with deduplication or fail closed until the source record is reconciled.
• crates/buzz-acp/src/reliability/ledger.rs:524 — NEW: ledger recovery scans an unbounded file before enforcing limits. Trigger: supply a very large ledger.jsonl with no newline. Startup seeks backward and reads until BOF without a ledger-size or scan-size ceiling, allowing unbounded startup I/O and memory work. Fix: reject or quarantine ledgers above a fixed byte ceiling before recovery, and cap the backward scan to the maximum permitted record size.
• crates/buzz-acp/src/reliability/park.rs:579 — Prior DELTA-1 feat(desktop): show each agent's role in the @-mention picker (port of #2706) #6 remains: park reader silently abandons unreconciled records. Trigger: a park file larger than 10 MiB, containing over 1,000 valid batches, or containing an invalid line whose .corrupt append fails still opens successfully with only a prefix/subset. A later mutation rewrites that subset and destroys the unread suffix or invalid message record. Fix: fail ParkFile::open on any cap breach or parse failure unless a bounded quarantine commit succeeds atomically; never overwrite a source containing unreconciled records.
• crates/buzz-acp/src/reliability/runtime.rs:269 — Prior DELTA-1 feat(acp): extra MCP servers via env, with a trust boundary for their credentials #4 remains: multi-batch replay commit is still torn. Trigger: let the first replay marker commit, fail a later marker or ledger append, then fail unmark_replayed during rollback. Rollback errors are discarded, leaving some batches permanently replay-stamped even though nothing was sent. Fix: commit every replay marker and ledger intent as one atomic state transition or durable transaction; propagate any rollback failure.
• crates/buzz-acp/src/queue.rs:724 — Prior DELTA-1 spike(pdf): PDF route decision for T8 #3 remains: retry requeue drops cancelled carryover. Trigger: build a batch with 50 cancelled/replay carryover events plus one new event, then return a retryable provider failure. Parking rejects the 51 combined records, and requeue restores only batch.events, silently losing cancelled_events. Fix: preserve the complete batch, including cancelled carryover and metadata, through retry; split it safely before exceeding the durable per-record cap.
• crates/buzz-acp/src/lib.rs:5365 — Prior DELTA-1 fix(desktop): prefer the bundle's harness binaries over workspace target dirs #1 remains: full handoff fallback shifts message loss into the live queue. Trigger: fill the 200-entry returned-handoff queue, then fail another park handoff while its scope or channel queue is at capacity. The fallback requeue invokes queue eviction, discarding either an existing client message or part of the returned batch. Fix: retain failed handoffs in a bounded durable journal or apply backpressure; do not route them through an evicting queue.
• crates/buzz-acp/src/queue.rs:410 — Prior feat(acp): extra MCP servers via env, with a trust boundary for their credentials #4 remains: unavailable durability still consumes and evicts ingress. Trigger: start with reliability state unavailable and deliver a 501st event to one scope, or exceed the aggregate channel cap across scopes. push rejects or evicts an event after the relay delivery has been consumed, without a durable retry record or replay floor. Fix: do not admit subscriptions until durability is available, or durably spool/NACK deliveries and reconnect from an explicit replay floor; disable aggregate eviction during the outage.
• crates/buzz-acp/src/lib.rs:5722 — Prior feat(desktop): open markdown attachments in a viewer panel #5 remains: probe-lease release guards lack binding production tests. Trigger: delete or move the production release_pause_probe_for or release_breaker_probe calls on failed and panicked probe outcomes. Existing state-helper tests still pass, so the first failed probe can permanently wedge the scope. Fix: add tests that dispatch a real probe through PromptResult failure and panic paths and prove a subsequent probe becomes eligible.
• crates/buzz-acp/src/lib.rs:6023 — Prior feat(managed-agents): add OpenSEO-shaped agent config generator (T6-config) #8 remains: panic fallback can erase started provenance. Trigger: panic after provider output, then make direct parking fail or run without a reliability runtime. The fallback queue.requeue deconstructs the started batch; a later flush can classify it as not started and auto-replay duplicate side effects. Fix: carry immutable started provenance in queued records, or retain the intact batch until durable parking succeeds; test both fallback branches.
• crates/buzz-acp/src/acp.rs:2012 — Prior docs(zs): MCP registry design memo (T7) #9 remains: Goose started-state gate compares cumulative rather than turn-local output. Trigger: after one turn produces tokens, begin another turn that fails before output but emits accumulated_output_tokens from the previous turn. The nonzero cumulative value marks the new turn started and changes it from automatic replay to manual review. Fix: compare the usage snapshot with the per-turn baseline or use a current-turn output delta; add a two-turn wire-level regression.
• crates/buzz-acp/src/reliability/state_dir.rs:165 — Prior ci(zs): fix the inbox-live-update scroll baseline and settle layout before measuring #12 and DELTA-1 feat(desktop): open markdown attachments in a viewer panel #5 remain: directory fsync failure is reported as durable success. Trigger: allow rename to succeed and force the parent-directory fsync to fail. The helper logs and returns success, allowing the caller to release its only live copy even though a power loss can erase the renamed directory entry. Fix: return an explicit committed-but-not-durable outcome and retain custody or stop dispatch until durability is confirmed.
• crates/buzz-acp/src/reliability/ledger.rs:379 — Prior feat(desktop): export a markdown document as PDF (T9) #13 remains: failed ledger append poisons later records in-process. Trigger: force write_all to leave a partial JSON record, then restore the filesystem and append again without reopening. O_APPEND joins the next record to the partial tail; startup repair is never invoked between writes. Fix: repair or truncate to the last complete frame after every failed append and before another append; test short-write recovery without reopening.
• crates/buzz-acp/src/reliability/runtime.rs:384 — Prior feat(mcp): MCP registry core: secret store, launcher and proxy, registry loader, staged config generation (T7a) #14 remains: discard removes message custody before its audit record. Trigger: issue discard, let park removal commit, then fail the ledger append. DiscardedUnrecorded honestly reports the problem but cannot restore the deleted messages or the required audit trail. Fix: persist a durable discard intent before removal, or atomically transact the park mutation and ledger/outbox record.
• crates/buzz-acp/src/reliability/runtime.rs:182 — Prior feat(mcp): MCP registry core: secret store, launcher and proxy, registry loader, staged config generation (T7a) #14 remains: ordinary ledger failures are swallowed. Trigger: force batch_parked or batch_needs_review recording to fail. park_batch ignores the false result and returns success, allowing callers to proceed without the required ledger record. Fix: make record return Result and propagate it through a coherent park-plus-ledger transaction or durable outbox.
• crates/buzz-acp/src/pool.rs:5058 — Prior docs(zs): calendar view design memo (T12a) #15 remains: failure notices disappear after bounded retries or restart. Trigger: keep the relay unavailable for all 12 attempts or restart during retry. The detached task has no durable outbox, and a quiet paused scope has no later action that guarantees the missing operator notice is retried. Fix: write notices to a bounded idempotent outbox before sending and resume pending delivery during startup.
• crates/buzz-acp/src/reliability/error_class.rs:198 — Prior docs(zs): agent-minutes script and manifest behind the throughput baselines #19 remains: redactor exposes whitespace-separated credential values. Trigger: return an error such as X-API-Key: secret-value or JSON formatted as "api_key": "secret-value". Tokenization redacts the key-bearing token but leaves the following value token intact for logs, observers, notices, or ledger text. Fix: parse and redact key-value pairs across whitespace and punctuation boundaries; add exact log, observer, notice, and ledger sink tests.
• desktop/src-tauri/src/managed_agents/runtime.rs:1054 — Prior ci(zs): shorten the merge gate and serialize heavy local gates #20 remains: state-directory env ordering test does not bind the spawn builder. Trigger: move apply_state_dir_env before the descriptor.env loop while retaining the proof token. Production still compiles and the test at runtime/tests.rs:1334 still passes because it manually simulates the desired ordering instead of invoking the builder. Fix: extract the real command construction path and assert its final get_envs result, or encode the ordering with a type-state builder that cannot be reordered.

WARN

• crates/buzz-acp/src/reliability/error_class.rs:150 — Provider-error sanitization performs full-size work before final truncation. Trigger: a maximum-size provider error is cloned, normalized, split, and rebuilt before the final output cap is applied, increasing transient CPU and allocation pressure. Fix: apply a bounded input slice before normalization and tokenization, preserving a small allowance for redaction context.
• crates/buzz-acp/src/reliability/state_dir.rs:36 — Standalone state-directory input still follows symlinks and lacks a path-length bound. Trigger: launch the harness directly with BUZZ_ACP_STATE_DIR pointing through an attacker-controlled symlink or an excessively long path; directory and state-file operations follow that path. Fix: bound the environment value, reject symlink components where the trust model requires containment, and document the allowed root.
• crates/buzz-acp/src/lib.rs:1997 — replay_batch acknowledgement still means scheduled rather than immediately dispatched. Trigger: send replay_batch while another turn owns the scope. The control response reports success although replay waits behind the active turn and may later fail before dispatch. Fix: return a distinct scheduled status with the blocking scope state, or acknowledge only after dispatch begins.
• crates/buzz-cli/src/lib.rs:268 — Replay control still has no buzz-cli operator command. Trigger: an operator must construct raw control events because AgentsCmd exposes no typed replay-batch operation, making validated recovery unnecessarily error-prone. Fix: add a typed buzz-cli replay command with batch-id validation and normalized status output.
• crates/buzz-acp/src/lib.rs:5431 — Replay acknowledgements do not expose the newly generated batch identifier. Trigger: a successful replay creates a new batch UUID internally but the control result returns only the requested parked identifier, preventing direct ledger correlation. Fix: include both the requested parked batch IDs and the generated replay batch ID in the acknowledgement.
• crates/buzz-acp/src/lib.rs:5160 — Expired breaker path is recorded with misleading park provenance. Trigger: a scope whose breaker exceeds the expiry window takes Action::Park but is recorded using the generic auth-style park path rather than ParkReason::BreakerExpired, obscuring why work was withheld. Fix: carry the state-machine reason through Action::Park and record BreakerExpired explicitly.
• crates/buzz-acp/src/reliability/state.rs:182 — Breaker-cap enforcement silently removes active containment. Trigger: open breakers for 1,001 scopes. Capacity enforcement evicts the oldest still-active breaker, permitting that scope to dispatch again before recovery was demonstrated. Fix: reject or park new scopes at capacity, or retain an explicit global-open sentinel instead of silently lifting an existing breaker.
• crates/buzz-acp/src/reliability/park.rs:613 — Park records are not validated against their signed event and stored scope. Trigger: tamper a syntactically valid owner-local park line so its events have invalid signatures or channel tags inconsistent with the stored channel_id and scope. The record is accepted and can be replayed under the wrong recovery context. Fix: verify event signatures, permitted kinds, and channel/scope consistency while loading; quarantine records that fail validation.

Security note

During the fix round, an intermediate, uncommitted desktop test (a managed_agents
state-dir env test) spawned /usr/bin/env as a real child process and printed the whole
process environment on assertion failure; that output briefly reached a proof file and an
agent transcript before being redacted (see
$STATE/proof/SECURITY-env-leak-2026-09-07.md for the full incident record and exposed
variable names, values not included). The committed test on this branch does not do
this — it uses Command::get_envs() to inspect the constructed command's environment
in-process and spawns nothing.

---

Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com

🤖 Generated with Claude Code

https://claude.ai/code/session_01P8VoJ9givMm44HNc4gx5Gz

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[wiggdevin]

Re-triggering PR CI: GitHub created no workflow run for the ready_for_review event. Reopening immediately.