feat: controller-level stuck-agent sweep (non-LLM detection) (#571)

[rileywhite]

Summary

Implements a stuckTracker in the controller reconciliation tick that detects agents whose process is alive but conversation context is broken (e.g. Claude Code sessions wedged on a 400 tool_use concurrency error). Detection is non-LLM: pattern-match on pane output + wisp staleness cross-check, with a parallel progress-mismatch axis (activity predates the wisp). On detection, files a pool:dog warrant bead into the existing mol-shutdown-dance recovery flow rather than killing the session directly.

This is the alternative-design counterpart to #659. Where #659 hashes Peek output and kills the session in-controller, #714 separates observation from action: the controller files a warrant, and the existing dog/recovery flow handles termination. Detection composes two independent axes (regex + progress-mismatch) so the sweep stays useful even with zero error patterns configured.

• cmd/gc/stuck_tracker.go — pure-predicate stuckTracker interface + noopStuckTracker + memoryStuckTracker. Immutable compiled-regex slice. Convergent evidence: stale wisp AND (error pattern OR progress mismatch).
• cmd/gc/wisp_freshness.go + wisp_freshness_test.go — wispFreshness opaque struct + sweepWispFreshness() using existing CommandRunner pattern (one bd list --json per sweep, not per session). Tests pin the id, updated_at, metadata.session_name JSON tags against future bd refactors.
• cmd/gc/stuck_sweep.go — CityRuntime.runStuckSweep(): halts inside the halt gate, idempotence-checked (no duplicate warrants on store errors), ProcessAlive race-check before Create, agent.stuck event + gc.agent.stuck_warrants.total OTEL counter on successful warrant Create only.
• internal/config/ — five new [daemon] fields: stuck_sweep, stuck_wisp_threshold, stuck_error_patterns, stuck_peek_lines, stuck_warrant_label. Pre-start pattern validation (ValidateStuckPatterns) with indexed errors.
• internal/events/events.go — AgentStuck = "agent.stuck" constant.
• internal/telemetry/recorder.go — gc.agent.stuck_warrants.total Int64Counter, low-cardinality axis attribute (regex or progress_mismatch). Incremented exactly once per successful Create, after the AgentStuck event so metric and event always agree.
• internal/doctor/checks_semantic.go + tests — daemon-stuck-sweep doctor check surfaces the effective configuration: disabled / misconfigured / regex-axis state / progress-mismatch state, threshold, label. Calls StuckSweepEnabled() rather than re-deriving the predicate.
• cmd/gc/cmd_doctor.go, cmd_start.go, cmd_supervisor.go — wire the new doctor check and ensure stuck-sweep startup logging describes which axes are active.
• Docs — engdocs/architecture/controller.md, engdocs/architecture/health-patrol.md (incl. new Concurrency Model subsection documenting single-goroutine ownership of cr.stuck and cr.runner), docs/reference/config.md, docs/reference/cli.md, docs/schema/city-schema.json regenerated via go run ./cmd/genschema.

Design decisions

• stuck_sweep defaults to false (opt-in) — false-positive risk is high-consequence
• stuck_error_patterns defaults to empty in Go (provider-neutral); example patterns live in user TOML
• stuck_warrant_label configurable (default "pool:dog") for ZFC role-neutrality
• Zero patterns disables the regex axis only (not the whole sweep) — progress-mismatch axis stays live so wisp/activity divergence still triggers warrants
• Fail-open on wisp query error (sweep skips, no warrant flood)
• Fail-open on hasOpenStuckWarrant store error (assume warrant exists → skip)
• Single-goroutine ownership of cr.stuck and cr.runner — built in newCityRuntime, rebuilt only on config reload, accessed only from the reconciler tick

What was tested

• make fmt-check — PASS
• make lint — PASS
• make vet — PASS
• Unit tests: cmd/gc/stuck_tracker_test.go (11 tests: detection logic, config accessors, edge cases)
• Unit tests: cmd/gc/wisp_freshness_test.go (tiebreaker, empty/malformed/error/empty-array paths, JSON tag pinning)
• Integration tests: cmd/gc/stuck_sweep_test.go (5 tests: no-agents, idempotence, halt-gate, fail-open, event-on-create)
• Doctor tests: internal/doctor/checks_semantic_test.go (daemon-stuck-sweep check)

Review outcome

Two-round review council (Code, UAT, Docs, Alignment, Blast Radius, Invariants, Concurrency reviewers). Round 1 found 9 blocking issues (all fixed): idempotence direction, vacuous halt test, illusory interface, operator visibility gaps, doc dissonances. Round 2 added observability and policy refinements: OTEL counter (S1), doctor check (S2), relaxed StuckSweepEnabled() so zero patterns no longer disables the sweep entirely (M3), Concurrency Model docs (M2), JSON-tag pinning tests (M1), and dropped unused freshnessEntry fields (O4). Final pass: clean.

Docs updated

• engdocs/architecture/controller.md — tracker list, main loop, code map, config block
• engdocs/architecture/health-patrol.md — ASCII diagram, data flow prose, key types, events inventory, known limitations, Concurrency Model subsection
• docs/reference/config.md, docs/reference/cli.md, docs/schema/city-schema.json — regenerated via go run ./cmd/genschema
• Follow-up issue filed: ga-60k — operator guide (docs/guides/stuck-agent-detection.md)

Breaking changes

None. All new config fields default to disabled/empty; existing behavior unchanged.

Closes #571

🤖 Generated with Claude Code

[rileywhite]

CI failure here is the known flake tracked in #716 — TestACPConformance/SharedSession/Nudge_RunningSession failing with write |1: broken pipe, preceded by the same acp: readLoop exit: read |0: file already closed flood.

This PR does not touch internal/runtime/acp (changes are confined to cmd/gc/, internal/config, internal/events, internal/telemetry, internal/doctor, and docs/), so the failure is unrelated to the work in this branch.

Marking ready for review.