fix(supervisor): wake promptly after catalog publication (#433)

agent-identity: dev3.direct.omp.4ndwdn2g
agent-persona: generalist
agent-supervisor: unavailable
agent-tool: OMP
agent-tool-version: 18.1.2
agent-runtime: OMP 18.1.2
tooling-profile: dotfiles@7904863

Author: schickling-assistant

.gitignore

@@ -3,6 +3,9 @@
 # Nix build symlink.
 /result
 
+# Agent work state and temporary artifacts.
+/tmp/
+
 # Per-agent runtime overlay (materialized per clone; machine-specific, never product).
 .st2/
 .claude/

INVARIANTS.md

@@ -17,6 +17,7 @@ materialization, messaging, DING, or presence must preserve them.
 | **Bounded idempotent stream ingress** | A running agent accepts events only for a declared stream. Within the retained 128-receipt ring, concurrent or crash-replayed `(stream, event-id)` publication reuses one canonical filename, conflicting content fails, and supersession archives only the matching keyed predecessor or the stream-wide head through ordinary archive semantics. State remains bounded and honestly treats an identity evicted from the ring as new without searching inbox or archive history. Events do not write the Sent ledger and DING marks them as stream work. | `tests/event_e2e.rs::stable_event_identity_publishes_exactly_one_canonical_message`; `tests/event_e2e.rs::concurrent_replays_publish_exactly_one_event`; `tests/event_e2e.rs::conflicting_reuse_and_undeclared_or_suspended_ingress_fail_closed`; `tests/event_e2e.rs::supersede_collapses_only_the_matching_key_and_preserves_archive_receipts`; `tests/event_e2e.rs::keyless_supersede_replaces_the_stream_wide_head`; `tests/event_e2e.rs::crash_replay_honors_an_archive_receipt_and_never_restores_the_inbox_copy`; `tests/event_e2e.rs::subject_frontmatter_injection_is_refused_before_any_write`; `tests/event_e2e.rs::stream_state_is_bounded_and_forgets_only_beyond_its_honest_horizon`; `tests/event_e2e.rs::event_emit_cli_returns_a_stable_json_receipt_and_ding_marks_the_record`; `tests/stream_authoring_cli.rs::a_direct_adapter_launch_executes_the_exact_event_cli_contract`; `tests/run.rs::suspend_and_resume_relaunch_the_agent_and_stream_together` |
 | **Fail-closed observed native DING** | Each unread message becomes one normalized `[DING]` frame. Fresh delivery records ownership, then preserves the one combined bracketed-paste, 0.5 second delay, and Return transaction. PTY and Return success are transport only: `Delivered` additionally requires adapter classification of the expected notice text in a submitted-prompt or queued-message pattern while the lowest live composer is empty or an accepted idle placeholder. Retry never re-pastes and may send one bare Return only after two adjacent `RetainedSafe` observations. A maintained adapter's positive `NotRetained` observation releases only an already archived staged head; unread, blocked, timed-out, errored, unknown, and unrecognized states retain ownership and later FIFO work remains blocked. Ownership prevents duplicate paste across command failures, receipt ambiguity, archive races, and restart adoption without letting a vanished archived head block FIFO indefinitely. Startup backlog otherwise becomes one generic recovery DING; new arrivals remain FIFO; `busy` delivers immediately; only fresh `dnd` defers. | `src/ding/mod.rs::poke_text_normalizes_and_bounds_untrusted_fields`; `src/ding/mod.rs::malicious_controls_cannot_escape_the_single_paste_frame`; `src/ding/mod.rs::pty_delivery_uses_face607_delay_order_and_seconds`; `src/ding/mod.rs::maintained_composer_classifiers_require_exact_idle_state`; `src/ding/mod.rs::successful_transport_with_retained_or_unproven_pixels_is_not_delivered`; `src/ding/mod.rs::ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership`; `src/ding/mod.rs::adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt`; `src/ding/mod.rs::staged_retry_submits_only_retained_safe_and_requires_a_receipt`; `src/ding/mod.rs::staged_retry_keeps_unproven_and_retained_blocked_owned`; `src/ding/mod.rs::staged_ownership_survives_archive_and_never_repastes`; `src/ding/mod.rs::archived_not_retained_releases_fifo_without_repasting_owned_notice`; `src/ding/mod.rs::unread_not_retained_keeps_fifo_ownership_without_repasting`; `src/ding/mod.rs::pty_commands_have_a_real_outer_timeout`; `src/ding/mod.rs::session_watch_has_startup_grace_debounce_and_live_reset`; `src/ding/mod.rs::new_arrivals_is_fifo_and_archive_receipts_prevent_reding`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry`; `src/ding/mod.rs::startup_recovery_notice_retries_in_memory`; `src/ding/mod.rs::startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke` |
 | **Mutation-only filesystem wakeups** | Supervisor and DING filesystem watchers ignore read/open access events and wake early only for create, modify, rename, or remove events. Their own catalog and inbox reads therefore cannot bypass the bounded timer cadence or form a Linux inotify CPU loop. | `src/watch.rs::only_mutations_wake_watch_loops`; `src/watch.rs::linux_reads_are_silent_but_real_mutations_wake`; `src/ding/mod.rs::idle_ding_does_not_spin_on_its_own_inbox_reads`; `src/run.rs::idle_supervisor_does_not_spin_on_its_own_catalog_reads` |
+| **Prompt catalog convergence** | A resident catalog supervisor observes cooperative transaction commits through a constant-cost catalog-generation watcher and authorized direct Agent Spec publication through an independent declaration watcher. Both channels coalesce into the serialized loop without waiting for the periodic audit interval; failure of one leaves the other and the timer fallback available. Runtime/control noise remains excluded. | `src/watch.rs::catalog_generation_commit_wakes_catalog_watch`; `src/watch.rs::atomic_agent_bundle_publication_wakes_production_shaped_catalog`; `src/run.rs::supervisor_wakes_and_launches_a_new_direct_declaration`; `src/run.rs::failed_watch_installation_keeps_supervisor_on_timer_cadence` |
 | **Bounded DING PTY probe churn** | An unsafe or active composer retains its FIFO notice but deferred delivery retries use a bounded backoff, so each inbox poll cannot spawn another short-lived PTY probe. | `src/ding/mod.rs::deferred_delivery_backoff_bounds_short_lived_pty_attempts` |
 | **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/native_only.rs::clean_path_executes_the_maintained_native_authoring_guide`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` |
 | **Stable roster JSON** | `st2 agents --json [--enrich]` preserves field names, order, null handling, presence, typed desired state and rationale, the retirement compatibility projection, opaque declared Resource descriptors, origin-timed activity, inbox counts, and the appended `observedState`, `driverDiagnostic`, and `context` objects. Declared presence, desired lifecycle, observed harness state, native-driver degradation, and harness context are independent axes: none is derived from another; a missing observed or context record is `null`, while missing diagnostic evidence is explicitly `absent`, never healthy. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `src/agents.rs::observed_state_joins_declared_presence_without_touching_either`; `src/agents.rs::driver_diagnostic_wire_exposes_failure_and_evidence_age_without_identity_payloads`; `src/agents.rs::context_is_a_fourth_axis_that_survives_an_indeterminate_observed_state`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence`; `tests/status_agents.rs::roster_keeps_presence_separate_from_suspended_desired_state`; `tests/status_agents.rs::roster_uses_version_1_origin_time_for_last_activity`; `tests/status_agents.rs::roster_joins_a_real_context_record_independently_of_observed_state` |

docs/vrs/.decisions/0015-catalog-commits-and-direct-edits-use-independent-wake-channels.md

@@ -0,0 +1,59 @@
+# Catalog commits and direct edits use independent wake channels
+
+Status: accepted
+
+Requirements change authorized by Johannes on 2026-09-03 through interview decisions Q1–Q3.
+
+## Context
+
+A production `axe agent new` publication waited for the next 30-second supervisor interval. Issue #430 attributed the delay to a timer-only catalog loop, but the exact reported st2 revision already installed the declaration watcher. An isolated test also showed that the watcher receives the atomic directory rename used to publish a new Agent bundle in a small catalog.
+
+The failure is therefore at the declaration notification boundary, not the absence of a catalog event loop. Built-in st2 declaration writers already expose a stronger boundary: after locked publication and readback, each successful transaction atomically advances `.st2/catalog-generation`. Authorized direct atomic KDL replacement does not necessarily use that transaction boundary and must remain prompt.
+
+## Options
+
+| Option | Result | Reason |
+| --- | --- | --- |
+| Independent generation and declaration watchers | Selected | Gives cooperative commits a constant-cost channel while preserving direct atomic authoring. |
+| Durable host-scoped reconcile request | Rejected | Adds a caller and authoring contract, host selection, and a split publish/request failure. |
+| Declaration watcher only | Rejected | Leaves transactional launch latency dependent on the event health of every declaration subscription. |
+| Unix socket or process signal | Rejected | Adds endpoint lifecycle, permissions, PID or socket staleness, and lossy delivery without helping direct authors. |
+
+## Evidence and Argument
+
+The supporting source reads, exact-revision check, caller trace, production
+measurements, and watcher prototypes are recorded in
+[the catalog reconcile wakeup experiment](../.experiments/2026-09-03-catalog-reconcile-wakeup.md).
+The key discriminator is independence: a second watcher instance over one
+control directory does not share the declaration watcher's subscription set or
+backend event queue, while both preserve the existing serialized reconciliation
+owner.
+
+## Decision
+
+The resident catalog supervisor uses two independent filesystem watcher instances that feed the same unit wake channel:
+
+1. A constant-cost watcher subscribes non-recursively to `.st2` and accepts only mutation of `catalog-generation`. This watcher is installed before declaration subscriptions when the control directory exists, revalidates the directory identity after each pass, and reinstalls a stale subscription.
+2. The declaration watcher retains shallow subscriptions to catalog declaration-space directories. It accepts `agent.kdl`, root `catalog.kdl`, `_templates`, and declaration-directory topology mutations while excluding runtime and Resource state.
+
+A callback only queues a wake. The existing single-threaded supervisor owns reconciliation, drains a burst before one pass, and holds the shared catalog lock through discovery and execution. The periodic interval remains the correctness fallback.
+
+Watcher setup and callback failures are operator-visible. Failure of one watcher does not remove the other watcher.
+
+## Consequences
+
+- `st2 agent publish`, catalog apply, and in-place st2 authoring receive a commit-aligned wake after durable publication.
+- Authorized direct atomic Agent Spec replacement retains declaration-event wakeups without a publisher-specific API.
+- The generation watcher has a separate backend queue, so declaration subscription volume or overflow cannot consume its events.
+- The declaration watcher still has cost proportional to declaration-space directory count. This cost is accepted to preserve direct-author behavior; Resource payload depth remains excluded.
+- A public kick command, Unix socket, or caller-side Axe workaround is not part of the contract.
+- The exact reason that the production declaration event was missed remains unproved unless a production-shaped test reproduces it. The independent commit channel removes that unknown from transactional publication latency without mislabeling it as the root cause.
+
+## Evidence required for acceptance
+
+- exact-revision source evidence that the issue revision already had a catalog watcher;
+- a deterministic loop test that direct publication launches an added agent before a long timer;
+- atomic bundle publication coverage at the reported catalog scale;
+- atomic catalog-generation replacement coverage through the independent channel;
+- root `catalog.kdl`, mutation-only, disconnected-channel, and idle no-spin coverage;
+- no new-record diagnostics under strict VRS validation, and the repository Nix check.

docs/vrs/.experiments/2026-09-03-catalog-reconcile-wakeup.md

@@ -0,0 +1,57 @@
+# Catalog reconcile wakeup investigation
+
+Date: 2026-09-03
+
+## Question
+
+Why did three successful `axe agent new` publications wait 17–21 seconds for the resident st2 supervisor, and which wake contract removes that latency without weakening direct Agent Spec authoring?
+
+## Method
+
+1. Read issue #430 and split each canary duration at the published Agent Spec mtime.
+2. Inspected `src/main.rs`, `src/run.rs`, `src/watch.rs`, `src/agent_publish.rs`, and the catalog generation fence.
+3. Inspected `src/run.rs` at the exact reported revision `fbd1ff6` rather than inferring behavior from the current branch.
+4. Traced the dotfiles `axe agent new` caller through bundle publication and its PTY wait loop.
+5. Added a Linux watcher regression that atomically renames a complete Agent bundle into a catalog containing 750 existing agents.
+6. Added a deterministic resident-loop regression that creates a new direct Agent Spec after the first pass and requires its launch before a 60-second fallback interval.
+7. Prototyped a second watcher instance that accepts only atomic replacement of `.st2/catalog-generation`.
+
+## Evidence
+
+- Issue #430 records three supervisor-side waits of 21.0, 18.4, and 17.4 seconds against a 30-second interval.
+- `git show fbd1ff6:src/run.rs` contains `best_effort_catalog_watcher`, `wait_for_reconcile`, and the mutation channel used by `up_loop`. The `timer-only; no fs-watch` comment cited by the issue belongs to `up_loop_specs`, the static single-spec mode, not the catalog mode used by the dev3 service.
+- The dev3 service command was `st2 up --catalog ... --host dev3`, so it selected the catalog loop rather than static-spec mode.
+- The pre-change atomic bundle regression passed against the existing declaration watcher. The atomic rename shape alone does not reproduce the production miss.
+- Source tracing shows `axe agent new` invokes the pinned `st2 agent publish --bundle ...` and then polls `pty list` every 200 ms. Axe sends no wake request.
+- Every successful built-in declaration transaction advances `catalog-generation` after durable publication while it still owns the exclusive catalog lock.
+- The production catalog contained approximately 750 agents. Its full tree contained 20,579 directories, although declaration watcher pruning excludes runtime and Resource payload descendants. Retained supervisor logs showed no watcher setup warning. These readings do not prove watcher exhaustion or overflow.
+
+## Result
+
+The issue's timer-only source diagnosis is falsified. The observed latency is real, but the exact production declaration-event loss is not reproduced by the supported atomic bundle publication shape in an isolated catalog. Watcher setup failure, partial subscription failure, queue overflow, and lost backend events remain possible failure classes; available evidence does not select one.
+
+The selected solution does not disguise that uncertainty. A dedicated constant-cost catalog-generation watcher makes transactional publication latency independent of the declaration watcher. The existing declaration watcher remains for authorized direct atomic KDL edits. Both queue work into the same serialized supervisor loop, and the timer remains the recovery path.
+
+The Linux wakeup regression set passed 25 tests, including the 750-agent atomic
+bundle case, generation replacement and reinstallation after control-directory
+replacement, declaration subscription refresh, disconnected-channel fallback,
+and no unintended wakeups.
+The deterministic resident-loop proof launched the new declaration in its
+second pass with a 60-second timer, and the idle-supervisor proof completed
+without a spin.
+
+## Conclusion
+
+The experiment selects independent generation and declaration wake channels.
+The source-level change can guarantee prompt transactional publication without
+pretending to have identified the lost production event. A post-deployment
+canary remains useful fleet evidence, but it is not part of the source-level
+mechanism.
+
+## VRS Impact
+
+Root requirement R40 now requires prompt catalog convergence for cooperative
+transactions and authorized direct atomic publication. The root specification
+defines the two watcher instances, accepted paths, serialized coalescing, and
+timer fallback. Decision 0015 records the rejected public-request and
+single-channel alternatives.

docs/vrs/requirements.md

@@ -57,6 +57,16 @@ accepted.
   deterministic st2 reconciler keeps declared local processes converged; the
   root observes host-local runtime health, diagnoses failures, performs bounded
   recovery, and escalates what it cannot resolve.
+- **R40 Prompt catalog convergence:** A resident catalog supervisor begins a
+  new serialized reconciliation pass promptly after a supported declaration
+  publication becomes durably visible; it does not normally wait for the
+  periodic full-audit interval. Transactional st2 writers and authorized direct
+  atomic declaration replacement remain independent wake sources, so
+  degradation of one source does not disable the other. Accepted events
+  coalesce without concurrent passes. The periodic timer remains the
+  correctness fallback for event loss, unavailable platform watchers, and
+  unsupported direct filesystem mutation, and each degraded wake source emits
+  an operator-visible diagnostic.
 - **R31 Reachable restart bounds:** Within one supervisor run, restart
   accounting is per task and only successful launches spend the declared
   budget. `delay` is the minimum launch spacing in either mode. In delay mode,