diff --git a/INVARIANTS.md b/INVARIANTS.md index 69dbcc0a..3b2a6580 100644 --- a/INVARIANTS.md +++ b/INVARIANTS.md @@ -6,7 +6,7 @@ materialization, messaging, DING, or presence must preserve them. | Invariant | Guarantee | Proof | |---|---|---| | **R11 control-plane replacement safety** | Normal stop or SIGKILL of `st2 up`, atomic binary replacement, and control-plane restart leave live PTY and exec agents usable with the same PID and creation identity. The replacement adopts each survivor exactly once, launches only genuinely missing work, and never duplicates a survivor; only explicit teardown kills an agent. | `tests/nomad_survival.rs::normal_stop_and_binary_replacement_adopt_exec_unchanged_without_duplicate`; `tests/nomad_survival.rs::forced_kill_and_binary_replacement_adopt_exec_unchanged_without_duplicate`; `tests/nomad_survival.rs::normal_stop_and_binary_replacement_adopt_pty_unchanged_without_duplicate`; `tests/nomad_survival.rs::forced_kill_and_binary_replacement_adopt_pty_unchanged_without_duplicate`; `tests/nomad_survival.rs::explicit_teardown_kills_exec_but_plain_stop_does_not`; `tests/nomad_survival.rs::explicit_teardown_kills_pty_but_plain_stop_does_not` | -| **Runner-owned task identity** | Every launched PTY and exec task receives `ST_AGENT=.` from reconciliation. Missing values are injected, exact authored matches are accepted, conflicts refuse before workspace or runner mutation, and restart/replay preserves the value. Every managed PTY carries the actor path, while only the canonical agent PTY carries the agent compatibility role. | `tests/run.rs::runner_owned_identity_injects_compact_and_explicit_task_omissions_and_accepts_a_match`; `tests/run.rs::runner_owned_identity_conflict_refuses_before_materialization_or_runner_access`; `tests/run.rs::runner_owned_identity_metadata_is_form_equivalent_and_role_scoped`; `tests/run.rs::runner_owned_identity_is_rederived_for_dead_task_replay`; `src/run.rs::tests::build_run_command_persists_the_complete_managed_environment_before_the_command` | +| **Runner-owned task identity** | Every launched PTY and exec task receives `ST_AGENT=.` from reconciliation. Missing values are injected, exact authored matches are accepted, conflicts refuse before workspace or runner mutation, and restart/replay preserves the value. Every managed PTY carries the actor path, while only the canonical agent PTY carries the agent compatibility role. | `tests/run.rs::runner_owned_identity_injects_compact_and_explicit_task_omissions_and_accepts_a_match`; `tests/run.rs::runner_owned_identity_conflict_refuses_before_materialization_or_runner_access`; `tests/run.rs::runner_owned_identity_metadata_is_form_equivalent_and_role_scoped`; `tests/run.rs::runner_owned_identity_is_rederived_for_dead_task_replay`; `src/run/tests.rs::build_run_command_persists_the_complete_managed_environment_before_the_command` | | **R23 fail-closed diagnostic inventory** | `st2 tasks --json` joins a semantically stable desired-catalog observation to exact PTY and conservatively proven exec generations without writes or reconciliation. Task absence intent, typed agent desired state/rationale, and the owning supervisor's park decision remain distinct: a believable park and structured recovery argv carrying the exact canonical catalog and selected host accompany the unmodified runtime observation as a complete known fault, while an unbelievable marker is indeterminate. Missing roots are positively absent without creation; declaration drift, malformed, unreadable, duplicate, PID-reused, or incomplete evidence makes the typed envelope incomplete and exits non-zero. The observational boundary does not serialize catalog writers or authorize cutover. | `src/task_inventory.rs::stable_wire_shape_maps_pty_exec_explicit_default_and_ignores_foreign_host`; `src/task_inventory.rs::complete_missing_runtime_is_absent_but_incomplete_is_indeterminate`; `src/exec_backend.rs::legacy_plain_pid_is_observed_without_rewriting_it`; `src/exec_backend.rs::record_older_than_current_process_generation_cannot_be_promoted`; `src/task_inventory.rs::a_parked_task_reports_its_fault_alongside_a_truthful_runtime_state`; `src/task_inventory.rs::an_unbelievable_park_marker_makes_the_envelope_incomplete`; `tests/task_inventory_cli.rs::projected_recovery_targets_its_exact_catalog_and_host_despite_ambient_defaults`; `tests/task_inventory_cli.rs::suspended_agent_projects_task_absence_and_agent_rationale_separately`; `tests/task_inventory_cli.rs` | | **Transport-decoupled lifecycle** | Each task is isolated from a supervisor/transport process-group or cgroup cascade. | `tests/transport_isolation.rs`; `tests/transport_isolation_macos.rs` | | **Clean exec teardown** | Killing an exec task reaps its whole process group. | `tests/exec_backend.rs::exec_kill_reaps_the_whole_process_group_not_just_the_leader` | @@ -16,20 +16,20 @@ materialization, messaging, DING, or presence must preserve them. | **Durable native delivery ownership** | Each native harness keeps one monotone evidence chain per inbox filename in `delivery-ledger.json`. Ownership is persisted before the transport call, so a crash between attempt and evidence reopens as an ambiguous attempt that holds instead of re-sending. Evidence only advances, and only to a phase the harness can actually observe: Codex proves transport acceptance and consumption, OpenCode transport acceptance and durable read-back, and a record claiming any other phase is refused. Only an authoritative negative receipt re-authorizes a retry. Settlement stays with the recipient: an entry is released only on a phase its profile releases, and is otherwise reconciled away when the message leaves the unread set. A foreign, malformed, or self-inconsistent record quarantines delivery without rewriting it, and the persisted ledger is owner-only with no temp residue. A delivery attempt an earlier release left behind is carried forward rather than re-sent: the pre-ledger `delivery-state.json` record is translated exactly once, when no ledger file exists, into an asserted phase that holds the delivery and authorizes no transport until this build observes evidence of its own. | `src/delivery_ledger.rs::tests::begin_persists_attempted_before_transport`; `src/delivery_ledger.rs::tests::positive_evidence_never_downgrades`; `src/delivery_ledger.rs::tests::profiles_accept_only_evidence_their_harness_can_produce`; `src/delivery_ledger.rs::tests::negative_receipt_is_the_only_retry_authority`; `src/delivery_ledger.rs::tests::opencode_persistence_holds_until_archive`; `src/delivery_ledger.rs::tests::foreign_or_malformed_state_quarantines_without_rewriting`; `src/delivery_ledger.rs::tests::every_entry_must_validate_its_own_correlation`; `src/delivery_ledger.rs::tests::the_persisted_ledger_is_owner_only_and_leaves_no_temp_residue`; `src/delivery_ledger.rs::tests::an_asserted_phase_suppresses_a_duplicate_and_authorizes_no_transport`; `src/migrations/delivery_state/mod.rs::tests::a_carried_forward_attempt_is_an_assertion_that_authorizes_no_transport`; `src/migrations/delivery_state/mod.rs::tests::translation_happens_once_and_leaves_the_old_record_in_place`; `src/migrations/delivery_state/mod.rs::tests::deletion_trigger_absent_old_record_makes_this_module_a_no_op` | | **Idempotent service requests** | A declared non-agent service principal publishes one exact JSON request per caller-supplied idempotency key to a canonical Agent Spec inbox. Concurrent or crash-replayed publication reuses the reserved filename; conflicting key reuse fails. The typed reply routes to the principal's canonical inbox without an Agent Spec identity or orphan mailbox. | `tests/request_cli.rs::stable_request_key_atomically_deduplicates_one_canonical_agent_message`; `tests/request_cli.rs::concurrent_replays_publish_exactly_one_request`; `tests/request_cli.rs::typed_reply_routes_to_the_principal_and_status_is_a_tagged_json_union`; `tests/request_cli.rs::request_api_rejects_agent_impersonation_and_unknown_flat_principals` | | **Bounded idempotent stream ingress** | A running agent accepts events only for a declared stream. Within the retained 128-receipt ring, concurrent or crash-replayed `(stream, event-id)` publication reuses one canonical filename, conflicting content fails, and supersession archives only the matching keyed predecessor or the stream-wide head through ordinary archive semantics. State remains bounded and honestly treats an identity evicted from the ring as new without searching inbox or archive history. Events do not write the Sent ledger and DING marks them as stream work. | `tests/event_e2e.rs::stable_event_identity_publishes_exactly_one_canonical_message`; `tests/event_e2e.rs::concurrent_replays_publish_exactly_one_event`; `tests/event_e2e.rs::conflicting_reuse_and_undeclared_or_suspended_ingress_fail_closed`; `tests/event_e2e.rs::supersede_collapses_only_the_matching_key_and_preserves_archive_receipts`; `tests/event_e2e.rs::keyless_supersede_replaces_the_stream_wide_head`; `tests/event_e2e.rs::crash_replay_honors_an_archive_receipt_and_never_restores_the_inbox_copy`; `tests/event_e2e.rs::subject_frontmatter_injection_is_refused_before_any_write`; `tests/event_e2e.rs::stream_state_is_bounded_and_forgets_only_beyond_its_honest_horizon`; `tests/event_e2e.rs::event_emit_cli_returns_a_stable_json_receipt_and_ding_marks_the_record`; `tests/stream_authoring_cli.rs::a_direct_adapter_launch_executes_the_exact_event_cli_contract`; `tests/run.rs::suspend_and_resume_relaunch_the_agent_and_stream_together` | -| **Fail-closed observed native DING** | Each unread message becomes one normalized `[DING]` frame. Fresh delivery records ownership, then preserves the one combined bracketed-paste, 0.5 second delay, and Return transaction. PTY and Return success are transport only: `Delivered` additionally requires adapter classification of the expected notice text in a submitted-prompt or queued-message pattern while the lowest live composer is empty or an accepted idle placeholder. Retry never re-pastes and may send one bare Return only after two adjacent `RetainedSafe` observations. A maintained adapter's positive `NotRetained` observation releases only an already archived staged head; unread, blocked, timed-out, errored, unknown, and unrecognized states retain ownership and later FIFO work remains blocked. Ownership prevents duplicate paste across command failures, receipt ambiguity, archive races, and restart adoption without letting a vanished archived head block FIFO indefinitely. Startup backlog otherwise becomes one generic recovery DING; new arrivals remain FIFO; `busy` delivers immediately; only fresh `dnd` defers. | `src/ding/mod.rs::poke_text_normalizes_and_bounds_untrusted_fields`; `src/ding/mod.rs::malicious_controls_cannot_escape_the_single_paste_frame`; `src/ding/mod.rs::pty_delivery_uses_face607_delay_order_and_seconds`; `src/ding/mod.rs::maintained_composer_classifiers_require_exact_idle_state`; `src/ding/mod.rs::successful_transport_with_retained_or_unproven_pixels_is_not_delivered`; `src/ding/mod.rs::ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership`; `src/ding/mod.rs::adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt`; `src/ding/mod.rs::staged_retry_submits_only_retained_safe_and_requires_a_receipt`; `src/ding/mod.rs::staged_retry_keeps_unproven_and_retained_blocked_owned`; `src/ding/mod.rs::staged_ownership_survives_archive_and_never_repastes`; `src/ding/mod.rs::archived_not_retained_releases_fifo_without_repasting_owned_notice`; `src/ding/mod.rs::unread_not_retained_keeps_fifo_ownership_without_repasting`; `src/ding/mod.rs::pty_commands_have_a_real_outer_timeout`; `src/ding/mod.rs::session_watch_has_startup_grace_debounce_and_live_reset`; `src/ding/mod.rs::new_arrivals_is_fifo_and_archive_receipts_prevent_reding`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry`; `src/ding/mod.rs::startup_recovery_notice_retries_in_memory`; `src/ding/mod.rs::startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke` | -| **Mutation-only filesystem wakeups** | Supervisor and DING filesystem watchers ignore read/open access events and wake early only for create, modify, rename, or remove events. Their own catalog and inbox reads therefore cannot bypass the bounded timer cadence or form a Linux inotify CPU loop. | `src/watch.rs::only_mutations_wake_watch_loops`; `src/watch.rs::linux_reads_are_silent_but_real_mutations_wake`; `src/ding/mod.rs::idle_ding_does_not_spin_on_its_own_inbox_reads`; `src/run.rs::idle_supervisor_does_not_spin_on_its_own_catalog_reads` | -| **Prompt catalog convergence** | A resident catalog supervisor observes cooperative transaction commits through a constant-cost catalog-generation watcher and authorized direct Agent Spec publication through an independent declaration watcher. Both channels coalesce into the serialized loop without waiting for the periodic audit interval; failure of one leaves the other and the timer fallback available. Runtime/control noise remains excluded. | `src/watch.rs::catalog_generation_commit_wakes_catalog_watch`; `src/watch.rs::atomic_agent_bundle_publication_wakes_production_shaped_catalog`; `src/run.rs::supervisor_wakes_and_launches_a_new_direct_declaration`; `src/run.rs::failed_watch_installation_keeps_supervisor_on_timer_cadence` | -| **Bounded DING PTY probe churn** | An unsafe or active composer retains its FIFO notice but deferred delivery retries use a bounded backoff, so each inbox poll cannot spawn another short-lived PTY probe. | `src/ding/mod.rs::deferred_delivery_backoff_bounds_short_lived_pty_attempts` | -| **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/native_only.rs::clean_path_executes_the_maintained_native_authoring_guide`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` | +| **Fail-closed observed native DING** | Each unread message becomes one normalized `[DING]` frame. Fresh delivery records ownership, then preserves the one combined bracketed-paste, 0.5 second delay, and Return transaction. PTY and Return success are transport only: `Delivered` additionally requires adapter classification of the expected notice text in a submitted-prompt or queued-message pattern while the lowest live composer is empty or an accepted idle placeholder. Retry never re-pastes and may send one bare Return only after two adjacent `RetainedSafe` observations. A maintained adapter's positive `NotRetained` observation releases only an already archived staged head; unread, blocked, timed-out, errored, unknown, and unrecognized states retain ownership and later FIFO work remains blocked. Ownership prevents duplicate paste across command failures, receipt ambiguity, archive races, and restart adoption without letting a vanished archived head block FIFO indefinitely. Startup backlog otherwise becomes one generic recovery DING; new arrivals remain FIFO; `busy` delivers immediately; only fresh `dnd` defers. | `src/ding/tests.rs::poke_text_normalizes_and_bounds_untrusted_fields`; `src/ding/tests.rs::malicious_controls_cannot_escape_the_single_paste_frame`; `src/ding/tests.rs::pty_delivery_uses_face607_delay_order_and_seconds`; `src/ding/tests.rs::maintained_composer_classifiers_require_exact_idle_state`; `src/ding/tests.rs::successful_transport_with_retained_or_unproven_pixels_is_not_delivered`; `src/ding/tests.rs::ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership`; `src/ding/tests.rs::adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt`; `src/ding/tests.rs::staged_retry_submits_only_retained_safe_and_requires_a_receipt`; `src/ding/tests.rs::staged_retry_keeps_unproven_and_retained_blocked_owned`; `src/ding/tests.rs::staged_ownership_survives_archive_and_never_repastes`; `src/ding/tests.rs::archived_not_retained_releases_fifo_without_repasting_owned_notice`; `src/ding/tests.rs::unread_not_retained_keeps_fifo_ownership_without_repasting`; `src/ding/tests.rs::pty_commands_have_a_real_outer_timeout`; `src/ding/tests.rs::session_watch_has_startup_grace_debounce_and_live_reset`; `src/ding/tests.rs::new_arrivals_is_fifo_and_archive_receipts_prevent_reding`; `src/ding/tests.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry`; `src/ding/tests.rs::startup_recovery_notice_retries_in_memory`; `src/ding/tests.rs::startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke` | +| **Mutation-only filesystem wakeups** | Supervisor and DING filesystem watchers ignore read/open access events and wake early only for create, modify, rename, or remove events. Their own catalog and inbox reads therefore cannot bypass the bounded timer cadence or form a Linux inotify CPU loop. | `src/watch.rs::only_mutations_wake_watch_loops`; `src/watch.rs::linux_reads_are_silent_but_real_mutations_wake`; `src/ding/tests.rs::idle_ding_does_not_spin_on_its_own_inbox_reads`; `src/run/tests.rs::idle_supervisor_does_not_spin_on_its_own_catalog_reads` | +| **Prompt catalog convergence** | A resident catalog supervisor observes cooperative transaction commits through a constant-cost catalog-generation watcher and authorized direct Agent Spec publication through an independent declaration watcher. Both channels coalesce into the serialized loop without waiting for the periodic audit interval; failure of one leaves the other and the timer fallback available. Runtime/control noise remains excluded. | `src/watch.rs::catalog_generation_commit_wakes_catalog_watch`; `src/watch.rs::atomic_agent_bundle_publication_wakes_production_shaped_catalog`; `src/run/tests.rs::supervisor_wakes_and_launches_a_new_direct_declaration`; `src/run/tests.rs::failed_watch_installation_keeps_supervisor_on_timer_cadence` | +| **Bounded DING PTY probe churn** | An unsafe or active composer retains its FIFO notice but deferred delivery retries use a bounded backoff, so each inbox poll cannot spawn another short-lived PTY probe. | `src/ding/tests.rs::deferred_delivery_backoff_bounds_short_lived_pty_attempts` | +| **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/native_only.rs::clean_path_executes_the_maintained_native_authoring_guide`; `src/ding/tests.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` | | **Stable roster JSON** | `st2 agents --json [--enrich]` preserves field names, order, null handling, presence, typed desired state and rationale, the retirement compatibility projection, opaque declared Resource descriptors, origin-timed activity, inbox counts, and the appended immutable `id`, mutable `address`, nullable `busAddress`, `observedState`, `driverDiagnostic`, and `context` objects. Declared presence, desired lifecycle, observed harness state, native-driver degradation, and harness context are independent axes: none is derived from another; a missing observed or context record is `null`, while missing diagnostic evidence is explicitly `absent`, never healthy. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `src/agents.rs::observed_state_joins_declared_presence_without_touching_either`; `src/agents.rs::driver_diagnostic_wire_exposes_failure_and_evidence_age_without_identity_payloads`; `src/agents.rs::context_is_a_fourth_axis_that_survives_an_indeterminate_observed_state`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence`; `tests/status_agents.rs::roster_keeps_presence_separate_from_suspended_desired_state`; `tests/status_agents.rs::roster_uses_version_1_origin_time_for_last_activity`; `tests/status_agents.rs::roster_joins_a_real_context_record_independently_of_observed_state` | -| **Agent-declared presence** | Refresh preserves non-DND declared status and advances the version 1 heartbeat. A missing status starts as `available`. Legacy DND migrates without renewing its hold. Version 1 DND is not refreshed. Stale, malformed, or implausibly future heartbeats read as `unknown`. The outer Codex, Claude, and pi session wrappers own a five-minute heartbeat while their provider remains alive. | `src/status.rs::refresh_preserves_value_and_changes_heartbeat_bytes`; `src/status.rs::refresh_upgrades_legacy_dnd_without_renewing_the_hold`; `src/status.rs::refresh_missing_writes_available_default`; `src/status.rs::version_1_staleness_and_future_skew_are_bounded`; `src/status.rs::malformed_versioned_record_is_unknown_without_mtime_fallback`; `src/claude_session.rs::idle_provider_refreshes_presence_without_mcp_input`; `src/pi_session.rs::idle_pi_provider_refreshes_presence_without_channel_input`; `src/codex_app_server.rs::inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat` | +| **Agent-declared presence** | Refresh preserves non-DND declared status and advances the version 1 heartbeat. A missing status starts as `available`. Legacy DND migrates without renewing its hold. Version 1 DND is not refreshed. Stale, malformed, or implausibly future heartbeats read as `unknown`. The outer Codex, Claude, and pi session wrappers own a five-minute heartbeat while their provider remains alive. | `src/status.rs::refresh_preserves_value_and_changes_heartbeat_bytes`; `src/status.rs::refresh_upgrades_legacy_dnd_without_renewing_the_hold`; `src/status.rs::refresh_missing_writes_available_default`; `src/status.rs::version_1_staleness_and_future_skew_are_bounded`; `src/status.rs::malformed_versioned_record_is_unknown_without_mtime_fallback`; `src/claude_session.rs::idle_provider_refreshes_presence_without_mcp_input`; `src/pi_session.rs::idle_pi_provider_refreshes_presence_without_channel_input`; `src/codex_app_server/tests.rs::inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat` | | **Scoped delivery-input wakeups** | Native delivery pumps watch only their inputs: the agent's `resources/inbox` subtree and its `status` file. Runtime records written beside them by the pump's own process group — presence temp siblings, the `harness-state` and `harness-context` records with their locks and staged siblings, stream state — never wake delivery, so a producer that writes on every turn boundary cannot pump its own delivery loop. | `src/watch.rs::delivery_watcher_ignores_runtime_records_but_wakes_on_inbox_and_status` | -| **Observed harness state discipline** | The `harness-state` record is written only by the owning session's driver processes — wrapper, channel, or hooks — serialized by a cross-process lock and coalesced against the on-disk record, atomically and byte-distinct on every write that lands, with freshness from its embedded timestamp and never file mtime; restating an unchanged state touches the record only when the refresh cadence is due. `unknown` is derived, never written: staleness, future skew, malformation, an unsupported schema, and a provably dead pty session each read as `unknown` with a distinct reason; an indeterminate liveness probe downgrades nothing; a missing record is no observation rather than `unknown`; no absence derives a definite state. A writer that loses sight of its harness stops heartbeating instead of refreshing a state it cannot see, and a predecessor session's record is never re-stamped. A reaped provider yields a terminal `ended` record carrying its real exit — written before the stop path's SIGKILL escalation and rewritten from the escalation cover when a grace-window reap observes the real status, proven against the real wrapper binaries of both stop implementations — and never a live state. | `src/harness_state.rs::unknown_state_is_derived_and_cannot_be_written`; `src/harness_state.rs::malformed_record_is_unknown_without_mtime_fallback`; `src/harness_state.rs::staleness_and_future_skew_derive_unknown_with_distinct_reasons`; `src/harness_state.rs::a_dead_session_reads_unknown_even_while_fresh_but_ended_survives`; `src/harness_state.rs::every_landed_write_is_byte_distinct_and_fresh_restatements_do_not_write`; `src/harness_state.rs::a_chatty_producer_restating_its_state_causes_zero_writes`; `src/harness_state.rs::concurrent_writers_defer_to_the_on_disk_record_not_their_cache`; `src/harness_state.rs::a_predecessor_sessions_record_is_never_heartbeat_eligible`; `src/harness_state.rs::missing_record_reads_as_none_not_unknown`; `src/codex_app_server.rs::pump_publishes_observations_and_stops_heartbeating_on_evidence_loss`; `src/claude_session.rs::a_provider_killed_mid_turn_reads_ended_rather_than_active`; `src/claude_session.rs::a_clean_provider_exit_writes_the_terminal_record`; `tests/harness_state_teardown.rs::stop_escalation_writes_the_terminal_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_stop_escalation_writes_the_cover_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_graceful_stop_records_the_real_reaped_exit` | +| **Observed harness state discipline** | The `harness-state` record is written only by the owning session's driver processes — wrapper, channel, or hooks — serialized by a cross-process lock and coalesced against the on-disk record, atomically and byte-distinct on every write that lands, with freshness from its embedded timestamp and never file mtime; restating an unchanged state touches the record only when the refresh cadence is due. `unknown` is derived, never written: staleness, future skew, malformation, an unsupported schema, and a provably dead pty session each read as `unknown` with a distinct reason; an indeterminate liveness probe downgrades nothing; a missing record is no observation rather than `unknown`; no absence derives a definite state. A writer that loses sight of its harness stops heartbeating instead of refreshing a state it cannot see, and a predecessor session's record is never re-stamped. A reaped provider yields a terminal `ended` record carrying its real exit — written before the stop path's SIGKILL escalation and rewritten from the escalation cover when a grace-window reap observes the real status, proven against the real wrapper binaries of both stop implementations — and never a live state. | `src/harness_state.rs::unknown_state_is_derived_and_cannot_be_written`; `src/harness_state.rs::malformed_record_is_unknown_without_mtime_fallback`; `src/harness_state.rs::staleness_and_future_skew_derive_unknown_with_distinct_reasons`; `src/harness_state.rs::a_dead_session_reads_unknown_even_while_fresh_but_ended_survives`; `src/harness_state.rs::every_landed_write_is_byte_distinct_and_fresh_restatements_do_not_write`; `src/harness_state.rs::a_chatty_producer_restating_its_state_causes_zero_writes`; `src/harness_state.rs::concurrent_writers_defer_to_the_on_disk_record_not_their_cache`; `src/harness_state.rs::a_predecessor_sessions_record_is_never_heartbeat_eligible`; `src/harness_state.rs::missing_record_reads_as_none_not_unknown`; `src/codex_app_server/tests.rs::pump_publishes_observations_and_stops_heartbeating_on_evidence_loss`; `src/claude_session.rs::a_provider_killed_mid_turn_reads_ended_rather_than_active`; `src/claude_session.rs::a_clean_provider_exit_writes_the_terminal_record`; `tests/harness_state_teardown.rs::stop_escalation_writes_the_terminal_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_stop_escalation_writes_the_cover_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_graceful_stop_records_the_real_reaped_exit` | | **Harness context discipline** | The numeric `harness-context` record is a sibling of `harness-state` and shares none of its ownership machinery: `incarnation` is provenance and never a fence, and a straggler's write lands rather than being refused. A reading is written only when it enters a different bucket of `usedPercent`, when a compaction edge arrives, when proven Claude account-window exhaustion changes, or when the record is older than the heartbeat, so a chatty producer cannot inflate the write rate; Codex account-window occupancy does not classify availability without its separate credit metadata. Every landed write is byte-distinct, and freshness comes from the record's own `observedAtMs`, never file mtime. A withheld value is carried as `null` and never fabricated from zero, the previous reading, or a division st2 could have done itself, and a reading above the window is carried unclamped. There is no `unknown` on this axis: past the horizon the reading is returned marked `stale` with its age, surviving every derivation the categorical record makes, while an unparseable record, a foreign schema, an uninterpretable harness, and an untrustworthy clock each read as absent. The relaunch claim removes the record, and Doctor's high-reading and stale-record lines are advisory in both directions and never change its exit status. Where a producer ships, its numerator is the harness's own and is pinned to the build it was measured on: pi's `tokens` is the last assistant message's `totalTokens` and omp's is its prompt figure, so neither harness's arithmetic can be published under the other's tag, and a harness bump that changes a number's meaning without changing its shape fails a fixture rather than shipping. A harness that positively reports it does not know its own occupancy is forwarded as withheld in the same write as the compaction edge that emptied it, a compaction edge whose harness names no reason yields `unknown` rather than an invented word, and a durable count the producer cannot read narrows to st2 counting edges rather than losing them. | `src/harness_context.rs::a_reading_inside_the_written_bucket_does_not_write_and_a_crossing_does`; `src/harness_context.rs::claude_rate_limit_exhaustion_and_reset_crossings_land_inside_one_usage_bucket`; `src/harness_context.rs::codex_account_window_exhaustion_does_not_prove_the_runtime_is_blocked`; `src/harness_context.rs::a_withheld_percent_has_no_bucket_and_only_a_compaction_or_heartbeat_writes`; `src/harness_context.rs::a_record_older_than_the_heartbeat_is_rewritten_and_every_landed_write_is_byte_distinct`; `src/harness_context.rs::a_compaction_always_lands_with_its_trigger_and_may_carry_a_durable_count`; `src/harness_context.rs::withheld_values_are_null_and_are_never_fabricated`; `src/harness_context.rs::a_reading_above_the_window_is_carried_unclamped`; `src/harness_context.rs::a_stale_reading_is_returned_with_its_age_rather_than_derived_away`; `src/harness_context.rs::freshness_comes_from_the_record_bytes_and_never_from_file_mtime`; `src/harness_context.rs::additive_fields_decode_but_foreign_schema_and_harness_read_as_nothing`; `src/harness_context.rs::a_record_beyond_the_future_skew_bound_reads_as_nothing`; `src/harness_context.rs::a_straggler_lands_and_is_visible_as_provenance_rather_than_being_refused`; `src/harness_state.rs::the_relaunch_claim_removes_the_harness_context_record`; `src/agents.rs::context_is_a_fourth_axis_that_survives_an_indeterminate_observed_state`; `tests/doctor.rs::harness_context_doctor_lines_are_advisory_and_never_change_the_exit_status`; `src/pi_channel.rs::the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/pi_channel.rs::a_pi_compaction_withholds_the_reading_it_emptied_in_the_same_write`; `src/pi_channel.rs::an_omp_compaction_yields_unknown_because_the_event_names_no_reason`; `src/pi_channel.rs::an_unreadable_durable_count_degrades_to_counting_edges_not_to_losing_them`; `src/pi_channel.rs::context_frames_decode_conservatively_or_not_at_all`; `src/pi_channel.rs::the_measured_pi_release_is_the_one_the_extension_gate_pins`; `src/omp_session.rs::the_measured_context_builds_are_admitted_by_this_gate` | | **Replicated-path discipline** | st2 pins the exact driver-record names it expects the replication transport's include list to carry — `harness-state` and `harness-context` — and both readers derive their paths from that list, so a rename cannot silently stop replication in a repository that does not own the list. A harness-context write stages in the catalog control plane, outside the complete `agents` identity namespace, after validating canonical ancestry and same-filesystem atomic rename. Current-catalog identity walkers overlook only an exact legacy `.harness-context.tmp--` regular non-symlink host child and leave it untouched; directories, symlinks, special files, generic dotfiles, near misses, and prepared topology remain strict. `harness-state` still stages beside itself, pending its own change; the shared write helper takes the staging directory as an argument precisely because the two answer this differently. | `src/harness_context.rs::the_replicated_driver_record_names_are_pinned`; `src/harness_context.rs::writes_stage_in_catalog_control_and_clean_up_after_success_or_failure`; `src/catalog_transaction.rs::current_projection_ignores_and_preserves_an_exact_legacy_staging_file`; `src/catalog_transaction.rs::current_projection_rejects_legacy_shape_type_confusion_and_near_misses`; `src/message.rs::transition_addressability_ignores_and_preserves_exact_legacy_staging_files`; `src/message.rs::transition_addressability_rejects_legacy_type_confusion_and_near_misses`; `tests/catalog_apply.rs::snapshot_ignores_and_preserves_an_exact_legacy_harness_context_staging_file` | | **Status-line slot chaining** | Claude's `statusLine` is a single slot whose winning declaration replaces the others outright, and `.claude/settings.local.json` — the file st2 materializes — wins over every other settings file. So st2's entry there is a tee: it records the reading and then invokes the operator's own renderer, resolved from `$ST_CLAUDE_STATUSLINE_RENDERER` and then `~/.claude/statusline-renderer.json`, first hit wins and never both. Where no renderer resolves, and wherever a resolved renderer fails, the tee writes NOTHING to stdout and puts its diagnostic on stderr: the payload is machine JSON, so echoing it into the slot is strictly worse for the operator than a blank row. Recording is unaffected by which arm runs. | `tests/claude_statusline.rs::the_tee_records_the_reading_and_hands_the_same_payload_to_the_env_renderer`; `tests/claude_statusline.rs::the_operator_file_supplies_the_renderer_when_no_variable_does`; `tests/claude_statusline.rs::the_variable_wins_over_the_file_and_the_file_is_never_also_run`; `tests/claude_statusline.rs::with_no_renderer_the_tee_renders_nothing_rather_than_the_raw_payload`; `tests/claude_statusline.rs::a_recording_failure_still_renders_the_status_line`; `tests/claude_statusline.rs::a_renderer_that_exits_non_zero_leaves_stdout_empty`; `tests/claude_statusline.rs::a_renderer_file_that_is_not_executable_leaves_stdout_empty`; `tests/claude_statusline.rs::without_st2_on_path_the_script_drains_stdin_and_renders_nothing`; `tests/claude_statusline.rs::the_rendered_registration_names_the_chaining_tee_and_carries_the_whole_slot` | -| **Version-pinned producer arithmetic** | Every harness-context producer publishes the number its own harness means by that number, and each one is pinned to the exact build the meaning was measured on: Claude's integer percent over `total_input_tokens` and `context_window_size` (2.1.250), Codex's window with the 12,000 baseline subtracted from both sides (codex-cli 0.151.0), pi's last assistant `totalTokens` and omp's prompt-only `input` (0.84.2 and 18.0.9), and OpenCode's last non-summary assistant total over the providers window (1.18.25). The failure this bounds is a harness bump that changes what a key means without changing its shape, which no type gate and no round-trip assertion can see, so each fixture decodes a verbatim capture, asserts the version literally, and asserts the wrong numerator the same capture would yield. | `src/claude_session.rs::a_mid_session_statusline_payload_yields_claudes_own_triple`; `src/claude_session.rs::a_pre_turn_statusline_payload_withholds_rather_than_reporting_zero`; `src/codex_app_server.rs::codex_context_recomputes_the_captured_reading_and_pins_its_verified_version`; `src/pi_channel.rs::the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/opencode_session.rs::captured_opencode_turns_publish_the_assistant_total_over_the_providers_window` | -| **Native-driver diagnostics are bounded and recoverable** | One additive-tolerant `st2.driver-diagnostic.v1` snapshot exposes the earliest failing native-driver boundary through closed driver/stage/reason/source/support vocabulary and origin-timed evidence age. Absent, malformed, foreign, unknown, and wrongly paired records never read healthy. Stage recovery clears only its own failure and removes the record after final recovery without changing prompt submission, retry, read-back, or archive semantics. A rejected provider credential is published on its own `providerAuth` stage from the harness's own typed turn result — never from provider prose, and never from an auth-class flag a capacity, policy, or retry class also sets — outranking the delivery and read-back symptoms it causes while the four producer gates outrank it; only a turn that reached its ordinary end clears it, and no credential knowledge or remedy text enters st2. Roster and Doctor share the typed reader/repair policy — Doctor advising on absence only for a driver that publishes a boundary result on every launch; telemetry labels contain only bounded driver/stage/reason/source/support/outcome values, never versions, identities, prompts, messages, or paths. | `src/driver_diagnostic.rs::every_stage_reason_and_source_has_bounded_wire_vocabulary`; `src/driver_diagnostic.rs::additive_fields_decode_but_malformed_foreign_and_unknown_records_are_indeterminate`; `src/driver_diagnostic.rs::recovery_clears_only_its_stage_and_reveals_the_next_failure`; `src/driver_diagnostic.rs::a_credential_rejection_is_evidence_only_from_a_typed_turn_result`; `src/driver_diagnostic.rs::a_rejected_credential_outranks_its_symptoms_but_not_the_producer_gates`; `src/claude_session.rs::stop_failure_classifies_only_the_credential_class_as_provider_auth`; `src/claude_session.rs::a_rejected_claude_credential_stands_until_a_turn_reaches_its_ordinary_end`; `src/codex_app_server.rs::a_rejected_codex_credential_reads_provider_auth_while_a_quota_failure_does_not`; `src/codex_app_server.rs::protocol_schema_gate_requires_the_distinct_credential_and_quota_error_words`; `src/opencode_session.rs::delivery_and_read_back_boundaries_publish_and_clear_diagnostics_without_changing_retry`; `src/metrics.rs::driver_diagnostic_metric_attributes_are_exactly_the_bounded_axes`; `tests/doctor.rs::native_driver_diagnostic_roster_and_doctor_agree_and_recovery_clears`; `src/pi_channel.rs::only_omps_own_credential_class_becomes_provider_auth`; `src/pi_channel.rs::a_rejected_omp_credential_stands_until_a_turn_reaches_its_ordinary_end`; `src/pi_channel.rs::the_pi_channel_publishes_no_credential_verdict`; `tests/doctor.rs::a_rejected_provider_credential_advises_while_absence_stays_silent` | +| **Version-pinned producer arithmetic** | Every harness-context producer publishes the number its own harness means by that number, and each one is pinned to the exact build the meaning was measured on: Claude's integer percent over `total_input_tokens` and `context_window_size` (2.1.250), Codex's window with the 12,000 baseline subtracted from both sides (codex-cli 0.151.0), pi's last assistant `totalTokens` and omp's prompt-only `input` (0.84.2 and 18.0.9), and OpenCode's last non-summary assistant total over the providers window (1.18.25). The failure this bounds is a harness bump that changes what a key means without changing its shape, which no type gate and no round-trip assertion can see, so each fixture decodes a verbatim capture, asserts the version literally, and asserts the wrong numerator the same capture would yield. | `src/claude_session.rs::a_mid_session_statusline_payload_yields_claudes_own_triple`; `src/claude_session.rs::a_pre_turn_statusline_payload_withholds_rather_than_reporting_zero`; `src/codex_app_server/tests.rs::codex_context_recomputes_the_captured_reading_and_pins_its_verified_version`; `src/pi_channel.rs::the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/opencode_session.rs::captured_opencode_turns_publish_the_assistant_total_over_the_providers_window` | +| **Native-driver diagnostics are bounded and recoverable** | One additive-tolerant `st2.driver-diagnostic.v1` snapshot exposes the earliest failing native-driver boundary through closed driver/stage/reason/source/support vocabulary and origin-timed evidence age. Absent, malformed, foreign, unknown, and wrongly paired records never read healthy. Stage recovery clears only its own failure and removes the record after final recovery without changing prompt submission, retry, read-back, or archive semantics. A rejected provider credential is published on its own `providerAuth` stage from the harness's own typed turn result — never from provider prose, and never from an auth-class flag a capacity, policy, or retry class also sets — outranking the delivery and read-back symptoms it causes while the four producer gates outrank it; only a turn that reached its ordinary end clears it, and no credential knowledge or remedy text enters st2. Roster and Doctor share the typed reader/repair policy — Doctor advising on absence only for a driver that publishes a boundary result on every launch; telemetry labels contain only bounded driver/stage/reason/source/support/outcome values, never versions, identities, prompts, messages, or paths. | `src/driver_diagnostic.rs::every_stage_reason_and_source_has_bounded_wire_vocabulary`; `src/driver_diagnostic.rs::additive_fields_decode_but_malformed_foreign_and_unknown_records_are_indeterminate`; `src/driver_diagnostic.rs::recovery_clears_only_its_stage_and_reveals_the_next_failure`; `src/driver_diagnostic.rs::a_credential_rejection_is_evidence_only_from_a_typed_turn_result`; `src/driver_diagnostic.rs::a_rejected_credential_outranks_its_symptoms_but_not_the_producer_gates`; `src/claude_session.rs::stop_failure_classifies_only_the_credential_class_as_provider_auth`; `src/claude_session.rs::a_rejected_claude_credential_stands_until_a_turn_reaches_its_ordinary_end`; `src/codex_app_server/tests.rs::a_rejected_codex_credential_reads_provider_auth_while_a_quota_failure_does_not`; `src/codex_app_server/tests.rs::protocol_schema_gate_requires_the_distinct_credential_and_quota_error_words`; `src/opencode_session.rs::delivery_and_read_back_boundaries_publish_and_clear_diagnostics_without_changing_retry`; `src/metrics.rs::driver_diagnostic_metric_attributes_are_exactly_the_bounded_axes`; `tests/doctor.rs::native_driver_diagnostic_roster_and_doctor_agree_and_recovery_clears`; `src/pi_channel.rs::only_omps_own_credential_class_becomes_provider_auth`; `src/pi_channel.rs::a_rejected_omp_credential_stands_until_a_turn_reaches_its_ordinary_end`; `src/pi_channel.rs::the_pi_channel_publishes_no_credential_verdict`; `tests/doctor.rs::a_rejected_provider_credential_advises_while_absence_stays_silent` | | **Retirement health** | A retired declaration is healthy only after every declared task ID is absent. Any live or dead declared task record reports incomplete retirement; retired declarations do not require presence. Live declarations retain their existing task and presence checks. | `tests/doctor.rs::retired_declaration_is_healthy_when_tasks_and_presence_are_absent`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_declared_task_is_alive`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_dead_task_record_remains` | | **Suspension health** | A suspended declaration is healthy when no declared task is live and every retained dead record is explicitly keep-pinned. It requires no presence, but this weaker result never proves retirement. Resume preserves ordinary keep and adopt-only policy. | `tests/doctor.rs::suspended_declaration_is_healthy_when_tasks_are_absent_without_presence`; `tests/doctor.rs::suspended_declaration_distinguishes_live_dead_keep_and_dead_nonkeep`; `tests/reconcile.rs::resuming_uses_ordinary_reconcile_and_does_not_override_keep` | | **Archival leaves the live catalog** | `st2 catalog archive` moves a retired identity's whole directory — declaration and `resources/` byte-identical — under the catalog control plane's `.st2/archive`, where discovery structurally cannot reach it, and records one tombstone row in `st2 catalog graph --json`. Eligibility fails closed: a running or suspended declaration, any surviving live or dead declared task record, and any remaining declaration that still names the identity as `supervisor` each refuse before anything moves, and `--dry-run` proves the plan without creating the archive root. An archived directory with no readable tombstone is unexplained control-plane state that makes the graph incomplete. `st2 catalog unarchive` is the exact reverse move. | `tests/catalog_archive.rs::archive_moves_a_retired_seat_out_of_discovery_with_its_resources_byte_identical`; `tests/catalog_archive.rs::archive_refuses_a_running_or_suspended_declaration`; `tests/catalog_archive.rs::archive_refuses_while_any_declared_task_record_survives`; `tests/catalog_archive.rs::archive_refuses_an_identity_another_declaration_still_names_as_supervisor`; `tests/catalog_archive.rs::dry_run_reports_the_plan_and_changes_nothing`; `tests/catalog_archive.rs::an_archived_directory_without_a_tombstone_makes_the_graph_incomplete`; `tests/catalog_archive.rs::unarchive_restores_the_identity_byte_identically_and_clears_its_tombstone` | @@ -40,8 +40,8 @@ materialization, messaging, DING, or presence must preserve them. | **Native flat root** | Without an authored override, catalog tasks, eval messaging, shell helpers, and DING all use the catalog itself as `ST_ROOT`; no nested bus directory is synthesized. | `src/eval_run.rs::bus_root_expands_st_root_else_defaults`; `tests/eval_run_e2e.rs::st2_eval_runs_a_benign_folder_to_a_pass_verdict`; `tests/pty.rs` | | **Resource observation is state-first, atomic, and fenced** | ABI-3 periodic publication and demanded `Published` results reuse one bounded `Publication` payload and one host acceptance, digest, relevance, typed-fact, and catch-up core; the host never trusts a runtime digest or observation timestamp. Demand reaches only a resident runtime that explicitly declares `capability "demand"`. Every `Observe` carries a positive watermark and the exact owner, binding, and registration, and exactly one matching `Unchanged`, `Failed`, or `Published` atomic result closes it. One outstanding dispatch plus one latest trailing watermark coalesces bursts without losing in-flight arrivals. Backpressure retains queued demand, replacement fences stale output, restart and provider failure settle honestly, and client disconnect or wait expiry never cancels accepted work. | `tests/resource_profile_supervisor_e2e.rs::production_demand_jobs_coalesce_queue_disconnect_and_fence_generation`; `tests/resource_profile_supervisor_e2e.rs::durable_admission_retains_the_256_request_boundary_without_a_runtime`; `tests/resource_profile_supervisor_e2e.rs::production_component_preserves_resync_filter_catch_up_and_scope_isolation`; `tests/resource_profile_supervisor_e2e.rs::supervisor_spawns_vista_capability_and_preserves_stable_snapshot`; `tests/agent_resource.rs::refresh_cli_reports_exact_receipts_and_wait_expiry_keeps_the_request`; `src/resource_observe.rs::tests::receipt_evidence_shape_matches_atomic_results` | | **Atomic resource proposal publication** | Every changed resource publication is one host-owned compare-and-swap fenced by binding generation, state revision, and prior carrier digest. A persistent cross-process lock admits at most one proposal from the same prior. The content-derived proposal ID binds the accepted carrier digest and semantic outbox envelope; the durable intent becomes eligible only with the exact canonical carrier, then folds into one authoritative catch-up state. A pre-carrier crash exposes old state, a post-carrier crash catches up on restart, and retry after a lost acknowledgement returns the durable receipt without another transition. Ordinary reconciliation fails closed on out-of-band divergence after a committed intent; only an explicit generation-advance recovery may re-adopt the canonical carrier or its absence while invalidating the old intent and fence. | `src/resource_profile.rs::tests::atomic_publication_fences_races_and_survives_crash_restarts`; `src/resource_profile.rs::tests::generation_advance_explicitly_recovers_diverged_or_missing_carrier` | -| **Pass progress is independent of publication** | A reconcile pass completes whether or not resync publication is making progress. Publication runs on its own thread, so a pass's per-seat watch-set handshakes never wait for a catalog lock, a stream lock, or a refused publication. One publication is outstanding per subscription at a time, and a queued publication is dropped when its subscription is deactivated or removed by a refresh, so nothing is published to a seat the pass has already decided receives no events. | `src/run.rs::reconcile_pass_completes_while_a_resync_publication_is_blocked`; `src/resync.rs::a_flush_never_hands_off_a_subscription_whose_publication_is_outstanding`; `src/resync.rs::deactivation_drops_only_that_recipients_queued_publication`; `src/resync.rs::a_refresh_drops_a_queued_publication_for_a_subscription_it_removed` | -| **Terminal refusals are classified, not retried** | A refused resync publication is classified by what could admit it later. A recipient that is declared but not running parks its reservation: it is attempted once, captures and schedules nothing while parked, and re-arms with its exact reserved bytes when a refresh carries that recipient again. An ambiguous recipient, a foreign owning host, and an undeclared stream are permanently refused: the reservation is dropped and the carrier baseline advances so the same transition is not re-captured. Everything else, including an absent declaration, stays retryable. | `src/resync.rs::a_not_running_recipient_parks_its_reservation_and_is_attempted_once`; `src/resync.rs::a_parked_reservation_re_arms_and_replays_when_its_recipient_runs_again`; `src/resync.rs::a_permanently_refused_reservation_is_dropped_and_not_recaptured`; `src/resync.rs::refusals_are_classified_by_what_could_admit_them_later` | -| **Unbindable session sockets fail at admission** | Host-scoped validation rejects a pty task whose session socket path would exceed the portable 104-byte `sun_path` bound, because `pty` refuses that bind and the task can then never spawn. The bound is derived from the pty root resolved for the selected host rather than a fixed identity length, only that host's pty tasks are judged against it, and the diagnostic names the resolved path and the byte overage. A park whose cause is that same unbindable path does not advise `st2 unpark`, which would relaunch into the identical failure. | `tests/validate.rs::an_unbindable_session_socket_path_is_rejected_at_admission`; `tests/validate.rs::another_hosts_long_identity_is_not_judged_against_this_hosts_pty_root`; `tests/validate.rs::a_long_exec_task_id_is_not_a_socket_path_issue`; `src/run.rs::session_socket_overage_is_derived_from_the_resolved_root`; `tests/run.rs::a_structurally_unrecoverable_park_does_not_advise_unpark` | +| **Pass progress is independent of publication** | A reconcile pass completes whether or not resync publication is making progress. Publication runs on its own thread, so a pass's per-seat watch-set handshakes never wait for a catalog lock, a stream lock, or a refused publication. One publication is outstanding per subscription at a time, and a queued publication is dropped when its subscription is deactivated or removed by a refresh, so nothing is published to a seat the pass has already decided receives no events. | `src/run/tests.rs::reconcile_pass_completes_while_a_resync_publication_is_blocked`; `src/resync/tests.rs::a_flush_never_hands_off_a_subscription_whose_publication_is_outstanding`; `src/resync/tests.rs::deactivation_drops_only_that_recipients_queued_publication`; `src/resync/tests.rs::a_refresh_drops_a_queued_publication_for_a_subscription_it_removed` | +| **Terminal refusals are classified, not retried** | A refused resync publication is classified by what could admit it later. A recipient that is declared but not running parks its reservation: it is attempted once, captures and schedules nothing while parked, and re-arms with its exact reserved bytes when a refresh carries that recipient again. An ambiguous recipient, a foreign owning host, and an undeclared stream are permanently refused: the reservation is dropped and the carrier baseline advances so the same transition is not re-captured. Everything else, including an absent declaration, stays retryable. | `src/resync/tests.rs::a_not_running_recipient_parks_its_reservation_and_is_attempted_once`; `src/resync/tests.rs::a_parked_reservation_re_arms_and_replays_when_its_recipient_runs_again`; `src/resync/tests.rs::a_permanently_refused_reservation_is_dropped_and_not_recaptured`; `src/resync/tests.rs::refusals_are_classified_by_what_could_admit_them_later` | +| **Unbindable session sockets fail at admission** | Host-scoped validation rejects a pty task whose session socket path would exceed the portable 104-byte `sun_path` bound, because `pty` refuses that bind and the task can then never spawn. The bound is derived from the pty root resolved for the selected host rather than a fixed identity length, only that host's pty tasks are judged against it, and the diagnostic names the resolved path and the byte overage. A park whose cause is that same unbindable path does not advise `st2 unpark`, which would relaunch into the identical failure. | `tests/validate.rs::an_unbindable_session_socket_path_is_rejected_at_admission`; `tests/validate.rs::another_hosts_long_identity_is_not_judged_against_this_hosts_pty_root`; `tests/validate.rs::a_long_exec_task_id_is_not_a_socket_path_issue`; `src/run/tests.rs::session_socket_overage_is_derived_from_the_resolved_root`; `tests/run.rs::a_structurally_unrecoverable_park_does_not_advise_unpark` | | **Marker-matched declaration authority** | Every st2 write path that rewrites a declaration carrying `meta { managed-by "nix" }` admits it only from a caller asserting exactly the marker that declaration carries: the lifecycle verb and the compare-and-swap `agent publish` both refuse a mismatched marker, an unmarked subject, an unresolvable multi-marker declaration, and an empty or padded assertion before any write, while presentation, address, stream, and Resource authoring refuse a Nix-owned declaration outright. Publication reads that authority from the incumbent, so creating a declaration and republishing byte-identical bytes stay open — neither replaces bytes another writer owns. A marker-matched edit rewrites nothing but the lifecycle line and is admitted through the same full-catalog gate as a compare-and-swap publication of the same bytes, so it cannot commit a declaration the catalog would reject. | `src/agent_author.rs::marker_matched_lifecycle_authority_is_exact_and_source_preserving`; `src/agent_author.rs::marker_matched_retirement_refuses_a_candidate_admission_would_reject`; `tests/agent_desired_state.rs::cli_managed_by_authority_retires_a_projected_seat_and_refuses_every_inexact_claim`; `tests/agent_publish.rs::publication_honours_the_incumbent_ownership_marker`; `tests/agent_publish.rs::publication_refuses_an_unresolvable_owner_and_repairs_unreadable_bytes` | | **Proof references resolve** | Every qualified test named in this table exists in its named source file, so stale invariant claims fail the suite instead of silently surviving a refactor. | `tests/invariants.rs::qualified_proof_references_resolve` | diff --git a/docs/vrs/04-stream/.experiments/2026-08-20-pipes-actor-model-unification.md b/docs/vrs/04-stream/.experiments/2026-08-20-pipes-actor-model-unification.md index ed554e17..5ca9bbad 100644 --- a/docs/vrs/04-stream/.experiments/2026-08-20-pipes-actor-model-unification.md +++ b/docs/vrs/04-stream/.experiments/2026-08-20-pipes-actor-model-unification.md @@ -406,7 +406,7 @@ key still fails per recipient. every ambiguity, so no shipped address can be re-pointed) and `an_undeclared_sender_is_still_refused` (publication authority unchanged). -**DING.** `src/ding/mod.rs::tests::a_declared_service_actor_renders_its_own_marker_rather_than_unknown` +**DING.** `src/ding/tests.rs::a_declared_service_actor_renders_its_own_marker_rather_than_unknown` renders `[DING] » h.gh-ci: CI failure on PR #42 [id:abc123]` while an undeclared sender still renders `?`. `an_invalid_catalog_cannot_promote_a_service_sender_out_of_unknown` is the fail-closed gate: a broken declaration anywhere in the catalog forces `?` even for a genuinely declared service actor, so diff --git a/docs/vrs/04-stream/.experiments/2026-08-20-pipes-ingress-prototype.md b/docs/vrs/04-stream/.experiments/2026-08-20-pipes-ingress-prototype.md index 787f80a8..7f411331 100644 --- a/docs/vrs/04-stream/.experiments/2026-08-20-pipes-ingress-prototype.md +++ b/docs/vrs/04-stream/.experiments/2026-08-20-pipes-ingress-prototype.md @@ -451,7 +451,7 @@ this spike: - `tests/eval_run_e2e.rs::canonical_agents_freeze_the_admitted_route_across_post_boot_catalog_mutation` — fails identically with the spike stashed. -- `src/codex_app_server.rs::tests::runtime_owner_lock_is_nonblocking_and_released_on_close` +- `src/codex_app_server/tests.rs::runtime_owner_lock_is_nonblocking_and_released_on_close` — flaky under parallel `cargo test --lib` when the machine is also compiling; green in 4/4 subsequent full runs and green in isolation. - `tests/native_only.rs::tracked_product_surface_contains_only_native_names` — diff --git a/src/agent_author.rs b/src/agent_author.rs index 749b8d72..46723435 100644 --- a/src/agent_author.rs +++ b/src/agent_author.rs @@ -23,7 +23,21 @@ use kdl::{KdlDocument, KdlNode}; use serde::Serialize; use crate::catalog_lock::CatalogLock; -use crate::run::Runner as _; + +mod declared_field; +mod desired_state; +mod markers; +mod resource; +mod stream; + +// Re-exported at each item's own visibility so every existing path — `st2::agent_author::*` for +// the binary and the integration tests, `crate::agent_author::*` for `agent_publish`, and the +// inline test module's `use super::*` — resolves unchanged. +pub use declared_field::*; +pub use desired_state::*; +pub(crate) use markers::*; +pub use resource::*; +pub use stream::*; #[derive(Debug, Clone, Copy, PartialEq, Eq)] struct SourceVersion { @@ -50,55 +64,6 @@ impl SourceVersion { } } -/// A mutable presentation field with no routing or lifecycle authority. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] -#[serde(rename_all = "lowercase")] -pub enum PresentationField { - Name, - Description, -} - -impl PresentationField { - pub fn as_str(self) -> &'static str { - match self { - Self::Name => "name", - Self::Description => "description", - } - } - - fn max_chars(self) -> usize { - match self { - Self::Name => AGENT_NAME_MAX_CHARS, - Self::Description => AGENT_DESCRIPTION_MAX_CHARS, - } - } -} - -/// One single-positional-string child node these source-preserving edits may rewrite. -/// -/// Address is not presentation — it is the mutable route (R24/R25), and it carries authority -/// presentation never has — but it is edited by exactly the same span-bounded machinery: find, -/// replace, insert, or remove one child node while every other byte of the declaration survives. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum DeclaredField { - Presentation(PresentationField), - Address, -} - -impl DeclaredField { - fn as_str(self) -> &'static str { - match self { - Self::Presentation(field) => field.as_str(), - Self::Address => "address", - } - } -} - -impl From for DeclaredField { - fn from(field: PresentationField) -> Self { - Self::Presentation(field) - } -} /// Whether a request changed declaration bytes. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] @@ -108,111 +73,6 @@ pub enum AuthorOutcome { Unchanged, } -/// Stable machine-readable receipt from one presentation edit. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct PresentationReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub field: PresentationField, - pub value: Option, - pub retired: bool, -} - -/// Stable machine-readable receipt from one agent-address cutover. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -#[serde(rename_all = "camelCase")] -pub struct AddressReceipt { - pub result: AuthorOutcome, - /// The subject's immutable agent ID (R24) — the value an address cutover must not touch. - pub id: String, - /// The positional declaration key, also unchanged: it stays the legacy address fallback. - pub identity: String, - /// The declared `address` after the edit. `None` means the positional fallback is effective. - pub address: Option, - /// `.` after the cutover. `None` for a retired subject, which is - /// non-routable and released its address. - pub bus_address: Option, - pub retired: bool, -} - -/// Stable authored desired-state selector. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] -#[serde(rename_all = "lowercase")] -pub enum DesiredStateValue { - Running, - Suspended, - Retired, -} - -impl DesiredStateValue { - pub fn as_str(self) -> &'static str { - match self { - Self::Running => "running", - Self::Suspended => "suspended", - Self::Retired => "retired", - } - } -} - -/// Stable machine-readable receipt from one desired-state edit. -/// -/// `managed_by` is the ownership marker the caller asserted and the declaration confirmed, so the -/// receipt records which authority admitted the edit; `null` is the ordinary unmarked path. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct DesiredStateReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub desired_state: DesiredStateValue, - pub reason: Option, - pub managed_by: Option, -} - -/// Stable machine-readable receipt from adding one agent-owned stream. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct StreamAddReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub name: String, - pub launch: Option, -} - -/// Stable machine-readable receipt from removing one agent-owned stream. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct StreamRemoveReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub name: String, -} - -/// Stable machine-readable receipt from adding or updating one Resource binding. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct ResourceAddReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub name: String, - pub uri: String, - pub reason: String, - pub inactive_reason: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub selector: Option, -} - -/// Stable machine-readable receipt from removing one Resource binding. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct ResourceRemoveReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub name: String, -} - -/// Stable machine-readable receipt from relabelling one Resource binding. -#[derive(Debug, Clone, PartialEq, Eq, Serialize)] -pub struct ResourceRenameReceipt { - pub result: AuthorOutcome, - pub identity: String, - pub old: String, - pub new: String, -} /// A classified authoring refusal. `code` is stable for machine consumers. #[derive(Debug)] @@ -254,633 +114,6 @@ struct AgentTarget { retired: bool, } -/// Add an agent-owned stream, or prove that the identical declaration already exists. -pub fn add_stream( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - name: &str, - launch: Option, -) -> Result { - author_stream( - catalog_root, - selector, - this_host, - actor, - name, - launch.as_ref(), - false, - ) - .map(|(result, identity)| StreamAddReceipt { - result, - identity, - name: name.to_owned(), - launch, - }) -} - -/// Remove one agent-owned stream. An already absent stream is an idempotent success. -pub fn remove_stream( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - name: &str, -) -> Result { - author_stream(catalog_root, selector, this_host, actor, name, None, true).map( - |(result, identity)| StreamRemoveReceipt { - result, - identity, - name: name.to_owned(), - }, - ) -} - -fn author_stream( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - name: &str, - launch: Option<&StreamLaunch>, - remove: bool, -) -> Result<(AuthorOutcome, String), AuthorError> { - let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { - AuthorError::new( - "catalog-lock-failed", - format!("acquire catalog-authoring lock: {error:#}"), - ) - })?; - let found = crate::discover_strict(catalog_root); - if let Some(error) = found.errors.first() { - return Err(AuthorError::new( - "catalog-malformed", - format!( - "cannot prove an exact stream target while {} is malformed: {}", - error.path.display(), - error.message - ), - )); - } - let target = resolve_target(&found.specs, selector, this_host)?; - let actor = actor - .map(|actor| resolve_target(&found.specs, actor, this_host).map(|target| target.identity)) - .transpose()?; - authorize_actor( - &found.specs, - &target.identity, - this_host, - actor.as_deref(), - "stream-not-authorized", - )?; - if remove { - let spec = found - .specs - .iter() - .find(|spec| spec.path == target.declaration) - .ok_or_else(|| { - AuthorError::new("stream-target-lost", "resolved stream target disappeared") - })?; - if spec - .streams - .iter() - .find(|stream| stream.name == name) - .is_some_and(|stream| stream.launch.is_some()) - { - let task_name = format!("{}{}", agent_spec::STREAM_TASK_PREFIX, name); - let task = spec - .tasks - .iter() - .find(|task| task.name == task_name) - .ok_or_else(|| { - AuthorError::new("stream-task-missing", "launched stream has no derived task") - })?; - let runtime_id = task - .id - .clone() - .unwrap_or_else(|| format!("{}.{}", spec.bus_id(this_host), task.name)); - let runner = crate::run::SystemRunner::new( - catalog_root.to_path_buf(), - crate::run::exec_state_dir(this_host), - ); - let live = runner - .list_sessions() - .map_err(|error| { - AuthorError::new("stream-runtime-observation-failed", error.to_string()) - })? - .into_iter() - .any(|session| session.alive && session.pty_id == runtime_id); - if live { - runner.retire(&runtime_id).map_err(|error| { - AuthorError::new( - "stream-runtime-retirement-failed", - format!("retire launched stream runtime {runtime_id}: {error:#}"), - ) - })?; - } - } - } - let result = edit_stream_declaration( - &catalog_lock, - catalog_root, - &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) - .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, - &target.declaration, - &target.identity, - &target.source_host, - &target.source_identity, - name, - launch, - remove, - || {}, - )?; - Ok((result, target.identity)) -} - -/// Declare one Resource binding, or update the binding that already carries `name`. -/// -/// st2 preserves the binding for readers; it resolves nothing and grants nothing. `uri` is the -/// exact absolute identity and is stored byte for byte with no normalization. -#[allow(clippy::too_many_arguments)] -pub fn add_resource( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - name: &str, - uri: &str, - reason: &str, - inactive_reason: Option<&str>, -) -> Result { - add_resource_with_selector( - catalog_root, - selector, - this_host, - actor, - name, - uri, - reason, - inactive_reason, - None, - ) -} - -#[allow(clippy::too_many_arguments)] -pub fn add_resource_with_selector( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - name: &str, - uri: &str, - reason: &str, - inactive_reason: Option<&str>, - resource_selector: Option<&serde_json::Value>, -) -> Result { - author_resource( - catalog_root, - selector, - this_host, - actor, - ResourceIntent::Upsert { - name, - uri, - reason, - inactive_reason, - selector: resource_selector, - }, - ) - .map(|(result, identity)| ResourceAddReceipt { - result, - identity, - name: name.to_owned(), - uri: uri.to_owned(), - reason: reason.to_owned(), - inactive_reason: inactive_reason.map(str::to_owned), - selector: resource_selector.cloned(), - }) -} - -/// Remove one Resource binding. An already absent binding is an idempotent success. -pub fn remove_resource( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - name: &str, -) -> Result { - author_resource( - catalog_root, - selector, - this_host, - actor, - ResourceIntent::Remove { name }, - ) - .map(|(result, identity)| ResourceRemoveReceipt { - result, - identity, - name: name.to_owned(), - }) -} - -/// Relabel one Resource binding, carrying its `uri`, `reason`, and `inactive-reason` unchanged. -/// -/// An absent `old` and an already declared `new` both refuse: binding names are unique within one -/// agent, so neither request has an outcome that preserves the caller's intent. -pub fn rename_resource( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - old: &str, - new: &str, -) -> Result { - author_resource( - catalog_root, - selector, - this_host, - actor, - ResourceIntent::Rename { old, new }, - ) - .map(|(result, identity)| ResourceRenameReceipt { - result, - identity, - old: old.to_owned(), - new: new.to_owned(), - }) -} - -/// One requested Resource-binding mutation, resolved against the declaration under the lock. -#[derive(Debug, Clone, Copy)] -enum ResourceIntent<'a> { - Upsert { - name: &'a str, - uri: &'a str, - reason: &'a str, - inactive_reason: Option<&'a str>, - selector: Option<&'a serde_json::Value>, - }, - Remove { - name: &'a str, - }, - Rename { - old: &'a str, - new: &'a str, - }, -} - -/// The binding state a candidate must read back as before it may be committed. -#[derive(Debug)] -struct ResourceExpectation { - absent: Option, - present: Option, -} - -fn author_resource( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - intent: ResourceIntent<'_>, -) -> Result<(AuthorOutcome, String), AuthorError> { - let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { - AuthorError::new( - "catalog-lock-failed", - format!("acquire catalog-authoring lock: {error:#}"), - ) - })?; - let found = crate::discover_strict(catalog_root); - if let Some(error) = found.errors.first() { - return Err(AuthorError::new( - "catalog-malformed", - format!( - "cannot prove an exact resource target while {} is malformed: {}", - error.path.display(), - error.message - ), - )); - } - let target = resolve_target(&found.specs, selector, this_host)?; - let actor = actor - .map(|actor| resolve_target(&found.specs, actor, this_host).map(|target| target.identity)) - .transpose()?; - authorize_actor( - &found.specs, - &target.identity, - this_host, - actor.as_deref(), - "resource-not-authorized", - )?; - let result = edit_resource_declaration( - &catalog_lock, - catalog_root, - &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) - .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, - &target.declaration, - &target.identity, - &target.source_host, - &target.source_identity, - intent, - || {}, - )?; - Ok((result, target.identity)) -} - -/// Author one whole-agent desired state without claiming runtime convergence. -/// -/// `managed_by` is the ownership marker the caller asserts owns the declaration. `None` is the -/// ordinary path and refuses a Nix-owned declaration, exactly as presentation, address, stream, and -/// Resource authoring do. `Some(marker)` is a generator saying "I am the writer of these bytes", -/// and is admitted only when the declaration's own `meta { managed-by "..." }` names exactly that -/// marker: it is the projection's typed route to the one transition its own source can no longer -/// express, because the source edit that has to be projected is the seat's removal (#473). -pub fn set_desired_state( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - state: DesiredStateValue, - reason: Option<&str>, - managed_by: Option<&str>, -) -> Result { - match state { - DesiredStateValue::Running if reason.is_some() => { - return Err(AuthorError::new( - "invalid-desired-state", - "running desired state forbids --reason", - )); - } - DesiredStateValue::Suspended | DesiredStateValue::Retired if reason.is_none() => { - return Err(AuthorError::new( - "invalid-desired-state", - format!("{} desired state requires --reason", state.as_str()), - )); - } - _ => {} - } - if let Some(reason) = reason { - validate_desired_state_reason(reason) - .map_err(|error| AuthorError::new("invalid-desired-state", error.to_string()))?; - } - validate_marker_assertion(managed_by)?; - let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { - AuthorError::new( - "catalog-lock-failed", - format!("acquire catalog-authoring lock: {error:#}"), - ) - })?; - let found = crate::discover(catalog_root); - if let Some(error) = found.errors.first() { - return Err(AuthorError::new( - "catalog-malformed", - format!( - "cannot prove an exact desired-state target while {} is malformed: {}", - error.path.display(), - error.message - ), - )); - } - let target = resolve_target(&found.specs, selector, this_host)?; - authorize_actor( - &found.specs, - &target.identity, - this_host, - actor, - "desired-state-not-authorized", - )?; - let result = edit_desired_state_declaration( - &catalog_lock, - catalog_root, - &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) - .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, - &target.declaration, - &target.identity, - &target.source_host, - &target.source_identity, - state, - reason, - managed_by, - || {}, - )?; - Ok(DesiredStateReceipt { - result, - identity: target.identity, - desired_state: state, - reason: reason.map(str::to_owned), - managed_by: managed_by.map(str::to_owned), - }) -} - -/// Set or clear one presentation field for one stable Agent Spec identity. -/// -/// `actor` is the caller-supplied `ST_AGENT` identity. An absent actor is the explicit operator -/// path. Within the trusted-fleet model, the guardrail limits a catalog-managed caller to itself or -/// a descendant reached through declared supervisor edges; no presentation field expands it. -pub fn set_presentation( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - field: PresentationField, - requested: Option<&str>, -) -> Result { - let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { - AuthorError::new( - "catalog-lock-failed", - format!("acquire catalog-authoring lock: {error:#}"), - ) - })?; - let found = crate::discover(catalog_root); - if let Some(error) = found.errors.first() { - return Err(AuthorError::new( - "catalog-malformed", - format!( - "cannot prove an exact presentation target while {} is malformed: {}", - error.path.display(), - error.message - ), - )); - } - let target = resolve_target(&found.specs, selector, this_host)?; - authorize_actor( - &found.specs, - &target.identity, - this_host, - actor, - "presentation-not-authorized", - )?; - let requested = requested - .map(|value| { - validate_presentation(field.as_str(), Some(value), field.max_chars()) - .map(|()| value.to_owned()) - .map_err(|error| AuthorError::new("invalid-presentation", error.to_string())) - }) - .transpose()?; - let result = edit_declaration( - &catalog_lock, - catalog_root, - &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) - .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, - &target.declaration, - &target.identity, - &target.source_host, - &target.source_identity, - field.into(), - requested.as_deref(), - || {}, - )?; - Ok(PresentationReceipt { - result, - identity: target.identity, - field, - value: requested, - retired: target.retired, - }) -} - -/// Assign or clear one subject's mutable agent address — one atomic address-book cutover (R25). -/// -/// The old address stops resolving as soon as the new catalog generation is visible; st2 stores no -/// rename history, redirect, implicit alias, or time-bounded compatibility route, so a stale -/// caller fails loudly and refreshes the roster. The edit rewrites exactly the `address` child -/// node, which is what makes the cutover nondisruptive by construction: the declaration-parent -/// state anchor, ID-keyed supervisor edges, task IDs, launch fingerprints, workspace, inbox, -/// archive, context, Resource state, and runtime ownership are all keyed off values this edit -/// never touches. `None` restores the positional `identity` fallback and is admitted only while -/// that fallback address is itself still unique on the resolved host. -pub fn set_address( - catalog_root: &Path, - selector: &str, - this_host: &str, - actor: Option<&str>, - requested: Option<&str>, -) -> Result { - let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { - AuthorError::new( - "catalog-lock-failed", - format!("acquire catalog-authoring lock: {error:#}"), - ) - })?; - let found = crate::discover(catalog_root); - if let Some(error) = found.errors.first() { - return Err(AuthorError::new( - "catalog-malformed", - format!( - "cannot prove an exact address target while {} is malformed: {}", - error.path.display(), - error.message - ), - )); - } - let target = resolve_target(&found.specs, selector, this_host)?; - authorize_actor( - &found.specs, - &target.identity, - this_host, - actor, - "address-not-authorized", - )?; - if let Some(value) = requested { - agent_spec::validate_agent_address(value) - .map_err(|error| AuthorError::new("invalid-address", error.to_string()))?; - } - refuse_address_collision(catalog_root, &found.specs, this_host, &target, requested)?; - let result = edit_declaration( - &catalog_lock, - catalog_root, - &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) - .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, - &target.declaration, - &target.identity, - &target.source_host, - &target.source_identity, - DeclaredField::Address, - requested, - || {}, - )?; - let effective = requested.unwrap_or(&target.source_identity); - Ok(AddressReceipt { - result, - // A retired subject is non-routable and released its address, so null is the honest bus - // address here — exactly what the roster projects for the same subject. - bus_address: (!target.retired).then(|| format!("{}.{effective}", target.source_host)), - address: requested.map(str::to_owned), - id: target.agent_id, - identity: target.identity, - retired: target.retired, - }) -} - -/// Refuse an effective address that would not be unique on the target's resolved logical host. -/// -/// The prospective catalog is the discovery this command already holds with exactly this subject's -/// `address` replaced, so `validate.rs`'s `dup-address` rule — the same rule whole-catalog -/// validation enforces — decides explicit/explicit and explicit/identity-fallback collisions -/// alike, including the `--clear` case where the restored fallback is the candidate address. Any -/// duplicate address in the prospective catalog refuses: an address book with two claims on one -/// route cannot answer an ordinary reference, so there is no cutover to admit. -fn refuse_address_collision( - catalog_root: &Path, - specs: &[crate::AgentSpec], - this_host: &str, - target: &AgentTarget, - requested: Option<&str>, -) -> Result<(), AuthorError> { - let mut prospective = crate::Discovered { - specs: specs.to_vec(), - ..Default::default() - }; - for spec in &mut prospective.specs { - if spec.bus_id(this_host) == target.identity { - spec.address = requested.map(str::to_owned); - } - } - let report = crate::validate::validate_discovered(catalog_root, Some(this_host), &prospective); - if !report - .issues - .iter() - .any(|issue| issue.code == "dup-address") - { - return Ok(()); - } - // Name the incumbent, not the first declaration in path order: the forwarded diagnostic often - // pointed at the candidate's own file, because that is where `dup-address` first saw the - // address. The claimant is the *other* subject reading the same effective address on this - // host. - let candidate = requested.unwrap_or(&target.source_identity); - let claimant = prospective - .specs - .iter() - .find(|spec| { - spec.bus_id(this_host) != target.identity - && !spec.desired_state.is_retired() - && spec.resolved_host(this_host) == target.source_host - && spec.effective_address() == candidate - }) - .map(|spec| { - format!( - "{} declared in {}", - spec.bus_id(this_host), - spec.path - .strip_prefix(catalog_root) - .unwrap_or(&spec.path) - .display() - ) - }); - Err(AuthorError::new( - "address-conflict", - format!( - "{} is not unique on host {:?}: already claimed by {}", - requested.map_or_else( - || format!("identity fallback address {:?}", target.source_identity), - |value| format!("address {value:?}") - ), - target.source_host, - claimant.unwrap_or_else(|| "another declaration in this catalog".to_owned()) - ), - )) -} fn resolve_target( specs: &[crate::AgentSpec], @@ -961,989 +194,82 @@ fn authorize_actor( let same_host = format!("{}.{}", spec.resolved_host(this_host), supervisor); let qualified = if by_identity.contains_key(supervisor) { supervisor.to_owned() - } else if by_identity.contains_key(&same_host) { - same_host - } else { - supervisor.to_owned() - }; - if qualified == actor { - return Ok(()); - } - current = qualified; - } - Err(AuthorError::new( - refusal_code, - format!("agent {actor:?} may edit only itself or a declared descendant, not {target:?}"), - )) -} - -#[cfg(test)] -fn edit_declaration_for_test( - path: &Path, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - field: PresentationField, - requested: Option<&str>, - before_commit: impl FnOnce(), -) -> Result { - let control = path - .parent() - .expect("test declaration has a parent") - .join(crate::catalog_lock::CONTROL_DIR); - fs::create_dir_all(&control).expect("create test catalog control directory"); - let catalog_lock = CatalogLock::exclusive(path.parent().expect("test catalog has a parent")) - .expect("acquire test catalog lock"); - edit_declaration( - &catalog_lock, - path.parent().expect("test catalog has a parent"), - &control, - path, - expected_identity, - expected_host, - expected_agent, - field.into(), - requested, - before_commit, - ) -} - -#[cfg(test)] -fn edit_desired_state_for_test( - path: &Path, - state: DesiredStateValue, - reason: Option<&str>, - before_commit: impl FnOnce(), -) -> Result { - let control = path - .parent() - .expect("test declaration has a parent") - .join(crate::catalog_lock::CONTROL_DIR); - fs::create_dir_all(&control).expect("create test catalog control directory"); - let catalog_lock = CatalogLock::exclusive(path.parent().expect("test catalog has a parent")) - .expect("acquire test catalog lock"); - edit_desired_state_declaration( - &catalog_lock, - path.parent().expect("test catalog has a parent"), - &control, - path, - "h.worker", - "h", - "worker", - state, - reason, - None, - before_commit, - ) -} - -#[allow(clippy::too_many_arguments)] -fn edit_stream_declaration( - catalog_lock: &CatalogLock, - catalog: &Path, - control: &Path, - path: &Path, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - name: &str, - launch: Option<&StreamLaunch>, - remove: bool, - before_commit: impl FnOnce(), -) -> Result { - if path.extension().and_then(|value| value.to_str()) != Some("kdl") { - return Err(AuthorError::new( - "unsupported-declaration-format", - format!( - "stream authoring requires canonical KDL, found {}", - path.display() - ), - )); - } - let metadata = fs::symlink_metadata(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - if !metadata.file_type().is_file() { - return Err(AuthorError::new( - "unsafe-declaration-path", - format!("refusing non-regular declaration path {}", path.display()), - )); - } - let original = fs::read(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - let original_version = SourceVersion::from_metadata(&metadata); - let text = std::str::from_utf8(&original).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("declaration {} is not UTF-8: {error}", path.display()), - ) - })?; - let document = KdlDocument::parse(text).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("parsing declaration {}: {error}", path.display()), - ) - })?; - let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; - if is_nix_managed(target) { - return Err(AuthorError::new( - "nix-managed-declaration", - format!( - "agent {expected_identity:?} is Nix-owned; edit its Nix source instead of {}", - path.display() - ), - )); - } - let replacement = stream_edit(text, target, name, launch, remove)?; - let Some(replacement) = replacement else { - return Ok(AuthorOutcome::Unchanged); - }; - verify_stream_candidate( - catalog, - path, - &replacement, - expected_identity, - expected_host, - expected_agent, - name, - launch, - remove, - )?; - atomic_replace_checked( - catalog_lock, - catalog, - control, - path, - &original, - original_version, - replacement.as_bytes(), - metadata.permissions().mode() & 0o7777, - before_commit, - )?; - Ok(AuthorOutcome::Changed) -} - -fn stream_edit( - text: &str, - target: &KdlNode, - name: &str, - launch: Option<&StreamLaunch>, - remove: bool, -) -> Result, AuthorError> { - let streams = target - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| { - child.name().value() == "stream" - && child.get(0).and_then(|entry| entry.as_string()) == Some(name) - }) - .collect::>(); - if streams.len() > 1 { - return Err(AuthorError::new( - "duplicate-stream", - format!("target declares stream {name:?} more than once"), - )); - } - if remove { - return streams - .first() - .map(|node| remove_field(text, node).map(Some)) - .unwrap_or(Ok(None)); - } - if let Some(existing) = streams.first() { - if parsed_stream_launch(existing)? == launch.cloned() { - return Ok(None); - } - return Err(AuthorError::new( - "stream-already-exists", - format!( - "stream {name:?} already exists with a different launch; remove it before adding a replacement" - ), - )); - } - let authored = match launch { - None => format!("stream {} {{}}", quoted(name)?), - Some(StreamLaunch::Command(command)) => format!( - "stream {} {{ command {} }}", - quoted(name)?, - quoted(command)? - ), - Some(StreamLaunch::Argv(argv)) => { - let values = argv - .iter() - .map(|value| quoted(value)) - .collect::, _>>()?; - format!("stream {} {{ argv {} }}", quoted(name)?, values.join(" ")) - } - }; - insert_node(text, target, &authored).map(Some) -} - -fn parsed_stream_launch(node: &KdlNode) -> Result, AuthorError> { - let children = node - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .collect::>(); - match children.as_slice() { - [] => Ok(None), - [child] if child.name().value() == "command" => child - .get(0) - .and_then(|entry| entry.as_string()) - .map(|value| Some(StreamLaunch::Command(value.to_owned()))) - .ok_or_else(|| { - AuthorError::new("malformed-stream", "stream command must contain one string") - }), - [child] if child.name().value() == "argv" => { - let argv = child - .entries() - .iter() - .map(|entry| entry.value().as_string().map(str::to_owned)) - .collect::>>() - .ok_or_else(|| { - AuthorError::new("malformed-stream", "stream argv values must be strings") - })?; - Ok(Some(StreamLaunch::Argv(argv))) - } - _ => Err(AuthorError::new( - "malformed-stream", - "stream must contain exactly one command or argv node, or be empty", - )), - } -} - -fn verify_stream_candidate( - catalog: &Path, - path: &Path, - candidate: &str, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - name: &str, - launch: Option<&StreamLaunch>, - removed: bool, -) -> Result<(), AuthorError> { - let temporary = tempfile::tempdir() - .map_err(|error| AuthorError::new("unsafe-source-edit", error.to_string()))?; - let relative = path.strip_prefix(catalog).map_err(|_| { - AuthorError::new( - "unsafe-declaration-path", - format!( - "declaration {} is outside catalog {}", - path.display(), - catalog.display() - ), - ) - })?; - let candidate_path = temporary.path().join(relative); - fs::create_dir_all( - candidate_path - .parent() - .expect("candidate declaration has a parent"), - ) - .and_then(|()| fs::write(&candidate_path, candidate)) - .map_err(|error| { - AuthorError::new( - "unsafe-source-edit", - format!("stage stream validation: {error}"), - ) - })?; - let (specs, _) = agent_spec::discover_file(temporary.path(), &candidate_path) - .map_err(|error| AuthorError::new("invalid-stream", error.to_string()))?; - let spec = specs - .iter() - .find(|spec| { - spec.identity == expected_agent && spec.bus_id(expected_host) == expected_identity - }) - .ok_or_else(|| { - AuthorError::new( - "unsafe-source-edit", - "stream candidate lost the authored agent", - ) - })?; - let observed = spec.streams.iter().find(|stream| stream.name == name); - if removed && observed.is_none() - || !removed && observed.is_some_and(|stream| stream.launch.as_ref() == launch) - { - Ok(()) - } else { - Err(AuthorError::new( - "unsafe-source-edit", - "stream candidate did not read back as the authored intent", - )) - } -} - -#[allow(clippy::too_many_arguments)] -fn edit_resource_declaration( - catalog_lock: &CatalogLock, - catalog: &Path, - control: &Path, - path: &Path, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - intent: ResourceIntent<'_>, - before_commit: impl FnOnce(), -) -> Result { - if path.extension().and_then(|value| value.to_str()) != Some("kdl") { - return Err(AuthorError::new( - "unsupported-declaration-format", - format!( - "resource authoring requires canonical KDL, found {}", - path.display() - ), - )); - } - let metadata = fs::symlink_metadata(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - if !metadata.file_type().is_file() { - return Err(AuthorError::new( - "unsafe-declaration-path", - format!("refusing non-regular declaration path {}", path.display()), - )); - } - let original = fs::read(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - let original_version = SourceVersion::from_metadata(&metadata); - let text = std::str::from_utf8(&original).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("declaration {} is not UTF-8: {error}", path.display()), - ) - })?; - let document = KdlDocument::parse(text).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("parsing declaration {}: {error}", path.display()), - ) - })?; - let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; - if is_nix_managed(target) { - return Err(AuthorError::new( - "nix-managed-declaration", - format!( - "agent {expected_identity:?} is Nix-owned; edit its Nix source instead of {}", - path.display() - ), - )); - } - let Some((replacement, expectation)) = resource_edit(text, target, intent)? else { - return Ok(AuthorOutcome::Unchanged); - }; - verify_resource_candidate( - catalog, - path, - &replacement, - expected_identity, - expected_host, - expected_agent, - &expectation, - )?; - atomic_replace_checked( - catalog_lock, - catalog, - control, - path, - &original, - original_version, - replacement.as_bytes(), - metadata.permissions().mode() & 0o7777, - before_commit, - )?; - Ok(AuthorOutcome::Changed) -} - -/// Resolve one intent against the declared bindings, preserving every unrelated byte. -/// -/// `Ok(None)` is the proven no-op: an unchanged upsert, an absent removal, or a self-rename. A -/// changed upsert rewrites exactly the one binding node in place, so its position, its leading -/// trivia, and every sibling binding survive. -fn resource_edit( - text: &str, - target: &KdlNode, - intent: ResourceIntent<'_>, -) -> Result, AuthorError> { - let declared = target - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| child.name().value() == "resource") - .collect::>(); - let declaring = |name: &str| -> Result, AuthorError> { - let matches = declared - .iter() - .copied() - .filter(|child| child.get(0).and_then(|entry| entry.as_string()) == Some(name)) - .collect::>(); - if matches.len() > 1 { - return Err(AuthorError::new( - "duplicate-resource", - format!("target declares resource {name:?} more than once"), - )); - } - Ok(matches.first().copied()) - }; - match intent { - ResourceIntent::Upsert { - name, - uri, - reason, - inactive_reason, - selector, - } => { - let authored = declared_resource(name, uri, reason, inactive_reason, selector)?; - let replacement = match declaring(name)? { - Some(node) if parsed_resource(node)? == authored => return Ok(None), - Some(node) => replace_node(text, node, &render_resource(&authored)?)?, - None => insert_node(text, target, &render_resource(&authored)?)?, - }; - Ok(Some(( - replacement, - ResourceExpectation { - absent: None, - present: Some(authored), - }, - ))) - } - ResourceIntent::Remove { name } => { - let Some(node) = declaring(name)? else { - return Ok(None); - }; - Ok(Some(( - remove_field(text, node)?, - ResourceExpectation { - absent: Some(name.to_owned()), - present: None, - }, - ))) - } - ResourceIntent::Rename { old, new } => { - let Some(node) = declaring(old)? else { - return Err(AuthorError::new( - "resource-not-found", - format!("target declares no resource {old:?}"), - )); - }; - if old == new { - return Ok(None); - } - if declaring(new)?.is_some() { - return Err(AuthorError::new( - "resource-already-exists", - format!( - "target already declares resource {new:?}; binding names are unique within one agent" - ), - )); - } - let carried = parsed_resource(node)?; - let renamed = declared_resource( - new, - carried.uri(), - carried.reason(), - carried.inactive_reason(), - carried.selector(), - )?; - Ok(Some(( - replace_node(text, node, &render_resource(&renamed)?)?, - ResourceExpectation { - absent: Some(old.to_owned()), - present: Some(renamed), - }, - ))) - } - } -} - -/// Enforce the canonical binding invariants — `agent_spec` owns them; this mints no new rule. -fn declared_resource( - name: &str, - uri: &str, - reason: &str, - inactive_reason: Option<&str>, - selector: Option<&serde_json::Value>, -) -> Result { - let resource = match inactive_reason { - None => Resource::new(name.to_owned(), uri.to_owned(), reason.to_owned()), - Some(inactive_reason) => Resource::new_inactive( - name.to_owned(), - uri.to_owned(), - reason.to_owned(), - inactive_reason.to_owned(), - ), - } - .map_err(|error| AuthorError::new("invalid-resource", error))?; - Ok(match selector { - Some(selector) => resource.with_selector(selector.clone()), - None => resource, - }) -} - -fn parsed_resource(node: &KdlNode) -> Result { - let malformed = - |detail: &str| AuthorError::new("malformed-resource", format!("resource binding {detail}")); - if node.children().is_some() { - return Err(malformed("cannot have children")); - } - let mut name = None; - let mut uri = None; - let mut reason = None; - let mut inactive_reason = None; - let mut selector = None; - for entry in node.entries() { - let value = entry - .value() - .as_string() - .ok_or_else(|| malformed("accepts only string values"))?; - match entry.name().map(|name| name.value()) { - None => { - if name.replace(value).is_some() { - return Err(malformed("declares one of its fields more than once")); - } - } - Some("uri") => { - if uri.replace(value).is_some() { - return Err(malformed("declares one of its fields more than once")); - } - } - Some("reason") => { - if reason.replace(value).is_some() { - return Err(malformed("declares one of its fields more than once")); - } - } - Some("inactive-reason") => { - if inactive_reason.replace(value).is_some() { - return Err(malformed("declares one of its fields more than once")); - } - } - Some("selector") => { - if selector.is_some() { - return Err(malformed("declares one of its fields more than once")); - } - selector = Some(serde_json::from_str(value).map_err(|error| { - malformed(&format!("has invalid JSON `selector`: {error}")) - })?); - } - Some(other) => return Err(malformed(&format!("has unsupported property `{other}`"))), - } - } - let (Some(name), Some(uri), Some(reason)) = (name, uri, reason) else { - return Err(malformed("needs a name, a `uri`, and a `reason`")); - }; - declared_resource(name, uri, reason, inactive_reason, selector.as_ref()) -} - -fn render_resource(resource: &Resource) -> Result { - let mut authored = format!( - "resource {} uri={} reason={}", - quoted(resource.name())?, - quoted(resource.uri())?, - quoted(resource.reason())? - ); - if let Some(inactive_reason) = resource.inactive_reason() { - authored.push_str(&format!(" inactive-reason={}", quoted(inactive_reason)?)); - } - if let Some(selector) = resource.selector() { - authored.push_str(" selector="); - authored.push_str(&raw_json(selector)?); - } - Ok(authored) -} - -fn raw_json(value: &serde_json::Value) -> Result { - let json = serde_json::to_string(value).map_err(|error| { - AuthorError::new( - "invalid-resource", - format!("serialize Resource selector as canonical JSON: {error}"), - ) - })?; - for hashes in 1..=json.len() + 1 { - let fence = "#".repeat(hashes); - if !json.contains(&format!("\"{fence}")) { - return Ok(format!("{fence}\"{json}\"{fence}")); - } - } - unreachable!("a delimiter longer than the JSON payload cannot occur in the payload") -} - -/// Replace exactly one node's source span. A KDL node span carries neither the leading trivia nor -/// the trailing terminator, so the surrounding line survives untouched. -fn replace_node(text: &str, node: &KdlNode, authored: &str) -> Result { - let span = node.span(); - let range = span.offset()..span.offset() + span.len(); - text.get(range.clone()).ok_or_else(|| { - AuthorError::new( - "malformed-declaration", - "resource binding span falls outside the declaration", - ) - })?; - // The span can run to the start of trailing trivia, so replacing it verbatim would glue the - // rendered node onto a following `// comment`. Leave that separator in the source. - let kept = text[range.clone()].trim_end_matches([' ', '\t']).len(); - let mut replacement = text.to_owned(); - replacement.replace_range(range.start..range.start + kept, authored); - Ok(replacement) -} - -fn verify_resource_candidate( - catalog: &Path, - path: &Path, - candidate: &str, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - expectation: &ResourceExpectation, -) -> Result<(), AuthorError> { - let temporary = tempfile::tempdir() - .map_err(|error| AuthorError::new("unsafe-source-edit", error.to_string()))?; - let relative = path.strip_prefix(catalog).map_err(|_| { - AuthorError::new( - "unsafe-declaration-path", - format!( - "declaration {} is outside catalog {}", - path.display(), - catalog.display() - ), - ) - })?; - let candidate_path = temporary.path().join(relative); - fs::create_dir_all( - candidate_path - .parent() - .expect("candidate declaration has a parent"), - ) - .and_then(|()| fs::write(&candidate_path, candidate)) - .map_err(|error| { - AuthorError::new( - "unsafe-source-edit", - format!("stage resource validation: {error}"), - ) - })?; - let (specs, _) = agent_spec::discover_file(temporary.path(), &candidate_path) - .map_err(|error| AuthorError::new("invalid-resource", error.to_string()))?; - let spec = specs - .iter() - .find(|spec| { - spec.identity == expected_agent && spec.bus_id(expected_host) == expected_identity - }) - .ok_or_else(|| { - AuthorError::new( - "unsafe-source-edit", - "resource candidate lost the authored agent", - ) - })?; - let declares = |name: &str| { - spec.resources - .iter() - .find(|resource| resource.name() == name) - }; - if expectation - .absent - .as_deref() - .is_some_and(|name| declares(name).is_some()) - || expectation - .present - .as_ref() - .is_some_and(|expected| declares(expected.name()) != Some(expected)) - { - return Err(AuthorError::new( - "unsafe-source-edit", - "resource candidate did not read back as the authored intent", - )); + } else if by_identity.contains_key(&same_host) { + same_host + } else { + supervisor.to_owned() + }; + if qualified == actor { + return Ok(()); + } + current = qualified; } - Ok(()) + Err(AuthorError::new( + refusal_code, + format!("agent {actor:?} may edit only itself or a declared descendant, not {target:?}"), + )) } -fn edit_declaration( - catalog_lock: &CatalogLock, - catalog: &Path, - control: &Path, +#[cfg(test)] +fn edit_declaration_for_test( path: &Path, expected_identity: &str, expected_host: &str, expected_agent: &str, - field: DeclaredField, + field: PresentationField, requested: Option<&str>, before_commit: impl FnOnce(), ) -> Result { - if path.extension().and_then(|value| value.to_str()) != Some("kdl") { - return Err(AuthorError::new( - "unsupported-declaration-format", - format!( - "presentation authoring requires canonical KDL, found {}", - path.display() - ), - )); - } - let metadata = fs::symlink_metadata(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - if !metadata.file_type().is_file() { - return Err(AuthorError::new( - "unsafe-declaration-path", - format!("refusing non-regular declaration path {}", path.display()), - )); - } - let original = fs::read(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - let original_version = SourceVersion::from_metadata(&metadata); - let text = std::str::from_utf8(&original).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("declaration {} is not UTF-8: {error}", path.display()), - ) - })?; - let document = KdlDocument::parse(text).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("parsing declaration {}: {error}", path.display()), - ) - })?; - let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; - if is_nix_managed(target) { - return Err(AuthorError::new( - "nix-managed-declaration", - format!( - "agent {expected_identity:?} is Nix-owned; edit its Nix source instead of {}", - path.display() - ), - )); - } - // No span-bounded field edit may change the subject's immutable agent ID (R24). Address, name, - // and description are all mutable; `id` is the one value that identifies the subject across - // every one of those changes, so the candidate must read back with the exact same bytes — or - // with none, on a declaration ID migration has not reached yet. - let expected_id = declared_id(target); - let Some(replacement) = presentation_edit(text, target, field, requested)? else { - return Ok(AuthorOutcome::Unchanged); - }; - verify_candidate( - &replacement, + let control = path + .parent() + .expect("test declaration has a parent") + .join(crate::catalog_lock::CONTROL_DIR); + fs::create_dir_all(&control).expect("create test catalog control directory"); + let catalog_lock = CatalogLock::exclusive(path.parent().expect("test catalog has a parent")) + .expect("acquire test catalog lock"); + edit_declaration( + &catalog_lock, + path.parent().expect("test catalog has a parent"), + &control, + path, expected_identity, expected_host, expected_agent, - field, + field.into(), requested, - expected_id.as_deref(), - )?; - atomic_replace_checked( - catalog_lock, - catalog, - control, - path, - &original, - original_version, - replacement.as_bytes(), - metadata.permissions().mode() & 0o7777, before_commit, - )?; - Ok(AuthorOutcome::Changed) + ) } -#[allow(clippy::too_many_arguments)] -fn edit_desired_state_declaration( - catalog_lock: &CatalogLock, - catalog: &Path, - control: &Path, +#[cfg(test)] +fn edit_desired_state_for_test( path: &Path, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, state: DesiredStateValue, reason: Option<&str>, - managed_by: Option<&str>, before_commit: impl FnOnce(), ) -> Result { - if path.extension().and_then(|value| value.to_str()) != Some("kdl") { - return Err(AuthorError::new( - "unsupported-declaration-format", - format!( - "desired-state authoring requires canonical KDL, found {}", - path.display() - ), - )); - } - let metadata = fs::symlink_metadata(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - if !metadata.file_type().is_file() { - return Err(AuthorError::new( - "unsafe-declaration-path", - format!("refusing non-regular declaration path {}", path.display()), - )); - } - let original = fs::read(path).map_err(|error| { - AuthorError::new( - "declaration-read-failed", - format!("reading declaration {}: {error}", path.display()), - ) - })?; - let original_version = SourceVersion::from_metadata(&metadata); - let text = std::str::from_utf8(&original).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("declaration {} is not UTF-8: {error}", path.display()), - ) - })?; - let document = KdlDocument::parse(text).map_err(|error| { - AuthorError::new( - "malformed-declaration", - format!("parsing declaration {}: {error}", path.display()), - ) - })?; - let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; - let marker_matched = authorize_marker(target, expected_identity, path, managed_by)?; - let Some(replacement) = desired_state_edit(text, target, state, reason)? else { - return Ok(AuthorOutcome::Unchanged); - }; - verify_desired_state_candidate( - &replacement, - expected_identity, - expected_host, - expected_agent, + let control = path + .parent() + .expect("test declaration has a parent") + .join(crate::catalog_lock::CONTROL_DIR); + fs::create_dir_all(&control).expect("create test catalog control directory"); + let catalog_lock = CatalogLock::exclusive(path.parent().expect("test catalog has a parent")) + .expect("acquire test catalog lock"); + edit_desired_state_declaration( + &catalog_lock, + path.parent().expect("test catalog has a parent"), + &control, + path, + "h.worker", + "h", + "worker", state, reason, - )?; - // A marker-matched edit stands in for the CAS `agent publish` the projection would otherwise - // have to perform, so it inherits that path's admission gate rather than only the local - // candidate reparse: the whole prospective catalog must still be admissible. That is what - // makes retiring a supervisor with a live descendant refuse (`retired-root`) instead of - // committing bytes the next reconcile pass rejects (#434). - if marker_matched - && let Err(error) = crate::agent_publish::admit_declaration_rewrite( - catalog, - control, - path, - replacement.as_bytes(), - ) - { - return Err(AuthorError::new( - "candidate-not-admissible", - format!("{error:#}"), - )); - } - atomic_replace_checked( - catalog_lock, - catalog, - control, - path, - &original, - original_version, - replacement.as_bytes(), - metadata.permissions().mode() & 0o7777, + None, before_commit, - )?; - Ok(AuthorOutcome::Changed) -} - -fn desired_state_edit( - text: &str, - target: &KdlNode, - state: DesiredStateValue, - reason: Option<&str>, -) -> Result, AuthorError> { - let lifecycle = target - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| matches!(child.name().value(), "desired-state" | "retired")) - .collect::>(); - if lifecycle.len() > 1 { - return Err(AuthorError::new( - "duplicate-lifecycle-field", - "target declares more than one lifecycle field", - )); - } - if state == DesiredStateValue::Running { - return lifecycle - .first() - .map(|node| remove_field(text, node).map(Some)) - .unwrap_or(Ok(None)); - } - let authored = format!( - "desired-state {} reason={}", - quoted(state.as_str())?, - quoted(reason.expect("validated by set_desired_state"))? - ); - match lifecycle.as_slice() { - [] => insert_node(text, target, &authored).map(Some), - [node] => { - let span = node.span(); - let range = span.offset()..span.offset() + span.len(); - if text.get(range.clone()) == Some(authored.as_str()) { - return Ok(None); - } - let mut replacement = text.to_owned(); - replacement.replace_range(range, &authored); - Ok(Some(replacement)) - } - _ => unreachable!(), - } + ) } -fn verify_desired_state_candidate( - candidate: &str, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - state: DesiredStateValue, - reason: Option<&str>, -) -> Result<(), AuthorError> { - let document = KdlDocument::parse(candidate).map_err(|error| { - AuthorError::new( - "unsafe-source-edit", - format!("desired-state edit did not produce valid KDL: {error}"), - ) - })?; - let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; - let lifecycle = target - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| matches!(child.name().value(), "desired-state" | "retired")) - .collect::>(); - if state == DesiredStateValue::Running { - if lifecycle.is_empty() { - return Ok(()); - } - } else if let [node] = lifecycle.as_slice() - && node.name().value() == "desired-state" - && node.get(0).and_then(|entry| entry.as_string()) == Some(state.as_str()) - && node.get("reason").and_then(|entry| entry.as_string()) == reason - { - return Ok(()); - } - Err(AuthorError::new( - "unsafe-source-edit", - "desired-state candidate did not read back as the authored intent", - )) -} fn exact_agent_node<'a>( document: &'a KdlDocument, @@ -2018,189 +344,6 @@ fn agent_identity_parts(node: &KdlNode) -> (Option, Option) { (host, identity) } -/// The declaration's explicit immutable `id`, if it carries one. -fn declared_id(node: &KdlNode) -> Option { - node.children()? - .nodes() - .iter() - .find(|child| child.name().value() == "id") - .and_then(|child| child.get(0)) - .and_then(|value| value.as_string()) - .map(str::to_owned) -} - -/// Every ownership marker this declaration carries, in source order. -/// -/// A well-formed declaration carries at most one. Several is not a resolvable ownership claim, so -/// they are returned as-is and no assertion can match them. -fn declared_markers(node: &KdlNode) -> Vec<&str> { - node.children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| child.name().value() == "meta") - .filter_map(KdlNode::children) - .flat_map(|meta| meta.nodes()) - .filter(|child| child.name().value() == "managed-by") - .filter_map(|child| child.get(0).and_then(|value| value.as_string())) - .collect() -} - -fn is_nix_managed(node: &KdlNode) -> bool { - declared_markers(node).contains(&"nix") -} - -/// Reject an ownership assertion no declaration could carry, before any lock or read. -/// -/// A marker is compared byte-exactly against the declaration's own `meta { managed-by "..." }` -/// value, so an empty or padded assertion can only ever be a caller mistake. -pub(crate) fn validate_marker_assertion(asserted: Option<&str>) -> Result<(), AuthorError> { - if let Some(marker) = asserted - && (marker.is_empty() || marker.trim() != marker) - { - return Err(AuthorError::new( - "invalid-managed-by", - format!("asserted ownership marker {marker:?} is empty or padded"), - )); - } - Ok(()) -} - -/// Every ownership marker the declaration source `bytes` carries, in source order. -/// -/// `agent publish` replaces a whole declaration file rather than one node, so every marker any -/// `agent` node in the incumbent file declares is at stake in that publication and the union is -/// what an assertion has to resolve against (#486). Bytes that are not UTF-8 or do not parse as -/// KDL carry no discoverable claim and read as unmarked: a writer able to leave such bytes in a -/// declaration leaf already holds direct filesystem write authority over the file, which is -/// strictly stronger than any st2 write path this marker governs. -pub(crate) fn declaration_markers(bytes: &[u8]) -> Vec { - let Ok(text) = std::str::from_utf8(bytes) else { - return Vec::new(); - }; - let Ok(document) = text.parse::() else { - return Vec::new(); - }; - document - .nodes() - .iter() - .filter(|node| node.name().value() == "agent") - .flat_map(|node| { - declared_markers(node) - .into_iter() - .map(str::to_owned) - .collect::>() - }) - .collect() -} - -/// Decide whether `asserted` authorizes rewriting bytes carrying the `declared` markers. -/// -/// `meta { managed-by "nix" }` says the Nix projection, not st2, is the writer of these bytes: an -/// edit made behind it is silently reverted on the next activation, which is why an unasserted -/// rewrite refuses (R25, decision 0003). Only that marker refuses; the others are labels on -/// declarations st2's own verbs and publishers are expected to rewrite. -/// -/// The generator itself is the one writer that legitimately authors the declaration, and -/// `--managed-by` is how it says so. An assertion is admitted only when it names exactly the one -/// marker the declaration carries — a caller wrong about who owns the bytes is wrong about the -/// edit, so a mismatched marker, an unmarked declaration, and an unresolvable multi-marker -/// declaration all fail closed. Returns whether an assertion was matched. -pub(crate) fn authorize_asserted_marker( - declared: &[&str], - subject: &str, - path: &Path, - asserted: Option<&str>, -) -> Result { - match (asserted, declared) { - (None, _) if !declared.contains(&"nix") => Ok(false), - (None, _) => Err(AuthorError::new( - "nix-managed-declaration", - format!( - "agent {subject:?} is Nix-owned; edit its Nix source instead of {}, or pass --managed-by \"nix\" if you are that projection", - path.display() - ), - )), - (Some(asserted), [marker]) if *marker == asserted => Ok(true), - (Some(asserted), []) => Err(AuthorError::new( - "managed-by-unmarked", - format!( - "--managed-by {asserted:?} claims agent {subject:?}, whose declaration {} carries no `meta {{ managed-by }}` marker", - path.display() - ), - )), - (Some(asserted), markers) => Err(AuthorError::new( - "managed-by-mismatch", - format!( - "--managed-by {asserted:?} does not own agent {subject:?}: {} declares owner {}", - path.display(), - markers - .iter() - .map(|marker| format!("{marker:?}")) - .collect::>() - .join(" and ") - ), - )), - } -} - -/// The lifecycle verb's marker authority, read from the exact declaration node it edits. -fn authorize_marker( - target: &KdlNode, - expected_identity: &str, - path: &Path, - asserted: Option<&str>, -) -> Result { - authorize_asserted_marker(&declared_markers(target), expected_identity, path, asserted) -} - -fn presentation_edit( - text: &str, - target: &KdlNode, - field: DeclaredField, - requested: Option<&str>, -) -> Result, AuthorError> { - let fields = target - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| child.name().value() == field.as_str()) - .collect::>(); - match fields.as_slice() { - [] => match requested { - Some(value) => insert_field(text, target, field, value).map(Some), - None => Ok(None), - }, - [node] => match requested { - Some(value) => replace_field(text, node, field, value), - None => remove_field(text, node).map(Some), - }, - _ => Err(AuthorError::new( - "duplicate-presentation-field", - format!("target declares `{}` more than once", field.as_str()), - )), - } -} - -fn parse_field_value(node: &KdlNode, field: DeclaredField) -> Result<&str, AuthorError> { - if node.children().is_some() || node.entries().len() != 1 || node.entries()[0].name().is_some() - { - return Err(AuthorError::new( - "malformed-presentation-field", - format!( - "`{}` must contain exactly one positional string", - field.as_str() - ), - )); - } - node.get(0) - .and_then(|value| value.as_string()) - .ok_or_else(|| { - AuthorError::new( - "malformed-presentation-field", - format!("`{}` must contain a string", field.as_str()), - ) - }) -} fn quoted(value: &str) -> Result { serde_json::to_string(value).map_err(|error| { @@ -2211,41 +354,6 @@ fn quoted(value: &str) -> Result { }) } -fn replace_field( - text: &str, - node: &KdlNode, - field: DeclaredField, - value: &str, -) -> Result, AuthorError> { - if parse_field_value(node, field)? == value { - return Ok(None); - } - let entry = &node.entries()[0]; - let span = entry.span(); - let range = span.offset()..span.offset() + span.len(); - text.get(range.clone()).ok_or_else(|| { - AuthorError::new( - "malformed-declaration", - "presentation value span falls outside the declaration", - ) - })?; - let mut replacement = text.to_owned(); - replacement.replace_range(range, "ed(value)?); - Ok(Some(replacement)) -} - -fn insert_field( - text: &str, - target: &KdlNode, - field: DeclaredField, - value: &str, -) -> Result { - insert_node( - text, - target, - &format!("{} {}", field.as_str(), quoted(value)?), - ) -} fn insert_node(text: &str, target: &KdlNode, authored: &str) -> Result { let span = target.span(); @@ -2395,61 +503,6 @@ fn line_indent(text: &str, offset: usize) -> Option { .then(|| indent.to_owned()) } -fn verify_candidate( - candidate: &str, - expected_identity: &str, - expected_host: &str, - expected_agent: &str, - field: DeclaredField, - expected: Option<&str>, - expected_id: Option<&str>, -) -> Result<(), AuthorError> { - let document = KdlDocument::parse(candidate).map_err(|error| { - AuthorError::new( - "unsafe-source-edit", - format!("field edit did not produce valid KDL: {error}"), - ) - })?; - let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; - if declared_id(target).as_deref() != expected_id { - return Err(AuthorError::new( - "agent-id-immutable", - format!( - "edit would change the immutable agent id of {expected_identity:?}; `id` is the \ - one declared value no authoring command may rewrite" - ), - )); - } - let fields = target - .children() - .into_iter() - .flat_map(|children| children.nodes()) - .filter(|child| child.name().value() == field.as_str()) - .collect::>(); - let observed = match fields.as_slice() { - [] => None, - [node] => Some(parse_field_value(node, field)?), - _ => { - return Err(AuthorError::new( - "unsafe-source-edit", - format!( - "presentation edit produced duplicate `{}` fields", - field.as_str() - ), - )); - } - }; - if observed != expected { - return Err(AuthorError::new( - "unsafe-source-edit", - format!( - "presentation edit did not produce the requested `{}`", - field.as_str() - ), - )); - } - Ok(()) -} fn atomic_replace_checked( catalog_lock: &CatalogLock, diff --git a/src/agent_author/declared_field.rs b/src/agent_author/declared_field.rs new file mode 100644 index 00000000..58f7926a --- /dev/null +++ b/src/agent_author/declared_field.rs @@ -0,0 +1,538 @@ +//! Single-positional-string declaration fields: presentation (name, description) and address. +//! +//! Moved verbatim out of `agent_author.rs`; the shared declaration-writer primitives (error +//! vocabulary, target resolution, KDL node location, span edits, atomic commit) stay there. + +use super::*; + +/// A mutable presentation field with no routing or lifecycle authority. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum PresentationField { + Name, + Description, +} + +impl PresentationField { + pub fn as_str(self) -> &'static str { + match self { + Self::Name => "name", + Self::Description => "description", + } + } + + fn max_chars(self) -> usize { + match self { + Self::Name => AGENT_NAME_MAX_CHARS, + Self::Description => AGENT_DESCRIPTION_MAX_CHARS, + } + } +} + +/// One single-positional-string child node these source-preserving edits may rewrite. +/// +/// Address is not presentation — it is the mutable route (R24/R25), and it carries authority +/// presentation never has — but it is edited by exactly the same span-bounded machinery: find, +/// replace, insert, or remove one child node while every other byte of the declaration survives. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum DeclaredField { + Presentation(PresentationField), + Address, +} + +impl DeclaredField { + fn as_str(self) -> &'static str { + match self { + Self::Presentation(field) => field.as_str(), + Self::Address => "address", + } + } +} + +impl From for DeclaredField { + fn from(field: PresentationField) -> Self { + Self::Presentation(field) + } +} + +/// Stable machine-readable receipt from one presentation edit. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct PresentationReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub field: PresentationField, + pub value: Option, + pub retired: bool, +} + +/// Stable machine-readable receipt from one agent-address cutover. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(rename_all = "camelCase")] +pub struct AddressReceipt { + pub result: AuthorOutcome, + /// The subject's immutable agent ID (R24) — the value an address cutover must not touch. + pub id: String, + /// The positional declaration key, also unchanged: it stays the legacy address fallback. + pub identity: String, + /// The declared `address` after the edit. `None` means the positional fallback is effective. + pub address: Option, + /// `.` after the cutover. `None` for a retired subject, which is + /// non-routable and released its address. + pub bus_address: Option, + pub retired: bool, +} + +/// Set or clear one presentation field for one stable Agent Spec identity. +/// +/// `actor` is the caller-supplied `ST_AGENT` identity. An absent actor is the explicit operator +/// path. Within the trusted-fleet model, the guardrail limits a catalog-managed caller to itself or +/// a descendant reached through declared supervisor edges; no presentation field expands it. +pub fn set_presentation( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + field: PresentationField, + requested: Option<&str>, +) -> Result { + let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { + AuthorError::new( + "catalog-lock-failed", + format!("acquire catalog-authoring lock: {error:#}"), + ) + })?; + let found = crate::discover(catalog_root); + if let Some(error) = found.errors.first() { + return Err(AuthorError::new( + "catalog-malformed", + format!( + "cannot prove an exact presentation target while {} is malformed: {}", + error.path.display(), + error.message + ), + )); + } + let target = resolve_target(&found.specs, selector, this_host)?; + authorize_actor( + &found.specs, + &target.identity, + this_host, + actor, + "presentation-not-authorized", + )?; + let requested = requested + .map(|value| { + validate_presentation(field.as_str(), Some(value), field.max_chars()) + .map(|()| value.to_owned()) + .map_err(|error| AuthorError::new("invalid-presentation", error.to_string())) + }) + .transpose()?; + let result = edit_declaration( + &catalog_lock, + catalog_root, + &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) + .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, + &target.declaration, + &target.identity, + &target.source_host, + &target.source_identity, + field.into(), + requested.as_deref(), + || {}, + )?; + Ok(PresentationReceipt { + result, + identity: target.identity, + field, + value: requested, + retired: target.retired, + }) +} + +/// Assign or clear one subject's mutable agent address — one atomic address-book cutover (R25). +/// +/// The old address stops resolving as soon as the new catalog generation is visible; st2 stores no +/// rename history, redirect, implicit alias, or time-bounded compatibility route, so a stale +/// caller fails loudly and refreshes the roster. The edit rewrites exactly the `address` child +/// node, which is what makes the cutover nondisruptive by construction: the declaration-parent +/// state anchor, ID-keyed supervisor edges, task IDs, launch fingerprints, workspace, inbox, +/// archive, context, Resource state, and runtime ownership are all keyed off values this edit +/// never touches. `None` restores the positional `identity` fallback and is admitted only while +/// that fallback address is itself still unique on the resolved host. +pub fn set_address( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + requested: Option<&str>, +) -> Result { + let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { + AuthorError::new( + "catalog-lock-failed", + format!("acquire catalog-authoring lock: {error:#}"), + ) + })?; + let found = crate::discover(catalog_root); + if let Some(error) = found.errors.first() { + return Err(AuthorError::new( + "catalog-malformed", + format!( + "cannot prove an exact address target while {} is malformed: {}", + error.path.display(), + error.message + ), + )); + } + let target = resolve_target(&found.specs, selector, this_host)?; + authorize_actor( + &found.specs, + &target.identity, + this_host, + actor, + "address-not-authorized", + )?; + if let Some(value) = requested { + agent_spec::validate_agent_address(value) + .map_err(|error| AuthorError::new("invalid-address", error.to_string()))?; + } + refuse_address_collision(catalog_root, &found.specs, this_host, &target, requested)?; + let result = edit_declaration( + &catalog_lock, + catalog_root, + &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) + .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, + &target.declaration, + &target.identity, + &target.source_host, + &target.source_identity, + DeclaredField::Address, + requested, + || {}, + )?; + let effective = requested.unwrap_or(&target.source_identity); + Ok(AddressReceipt { + result, + // A retired subject is non-routable and released its address, so null is the honest bus + // address here — exactly what the roster projects for the same subject. + bus_address: (!target.retired).then(|| format!("{}.{effective}", target.source_host)), + address: requested.map(str::to_owned), + id: target.agent_id, + identity: target.identity, + retired: target.retired, + }) +} + +/// Refuse an effective address that would not be unique on the target's resolved logical host. +/// +/// The prospective catalog is the discovery this command already holds with exactly this subject's +/// `address` replaced, so `validate.rs`'s `dup-address` rule — the same rule whole-catalog +/// validation enforces — decides explicit/explicit and explicit/identity-fallback collisions +/// alike, including the `--clear` case where the restored fallback is the candidate address. Any +/// duplicate address in the prospective catalog refuses: an address book with two claims on one +/// route cannot answer an ordinary reference, so there is no cutover to admit. +fn refuse_address_collision( + catalog_root: &Path, + specs: &[crate::AgentSpec], + this_host: &str, + target: &AgentTarget, + requested: Option<&str>, +) -> Result<(), AuthorError> { + let mut prospective = crate::Discovered { + specs: specs.to_vec(), + ..Default::default() + }; + for spec in &mut prospective.specs { + if spec.bus_id(this_host) == target.identity { + spec.address = requested.map(str::to_owned); + } + } + let report = crate::validate::validate_discovered(catalog_root, Some(this_host), &prospective); + if !report + .issues + .iter() + .any(|issue| issue.code == "dup-address") + { + return Ok(()); + } + // Name the incumbent, not the first declaration in path order: the forwarded diagnostic often + // pointed at the candidate's own file, because that is where `dup-address` first saw the + // address. The claimant is the *other* subject reading the same effective address on this + // host. + let candidate = requested.unwrap_or(&target.source_identity); + let claimant = prospective + .specs + .iter() + .find(|spec| { + spec.bus_id(this_host) != target.identity + && !spec.desired_state.is_retired() + && spec.resolved_host(this_host) == target.source_host + && spec.effective_address() == candidate + }) + .map(|spec| { + format!( + "{} declared in {}", + spec.bus_id(this_host), + spec.path + .strip_prefix(catalog_root) + .unwrap_or(&spec.path) + .display() + ) + }); + Err(AuthorError::new( + "address-conflict", + format!( + "{} is not unique on host {:?}: already claimed by {}", + requested.map_or_else( + || format!("identity fallback address {:?}", target.source_identity), + |value| format!("address {value:?}") + ), + target.source_host, + claimant.unwrap_or_else(|| "another declaration in this catalog".to_owned()) + ), + )) +} + +pub(super) fn edit_declaration( + catalog_lock: &CatalogLock, + catalog: &Path, + control: &Path, + path: &Path, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + field: DeclaredField, + requested: Option<&str>, + before_commit: impl FnOnce(), +) -> Result { + if path.extension().and_then(|value| value.to_str()) != Some("kdl") { + return Err(AuthorError::new( + "unsupported-declaration-format", + format!( + "presentation authoring requires canonical KDL, found {}", + path.display() + ), + )); + } + let metadata = fs::symlink_metadata(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + if !metadata.file_type().is_file() { + return Err(AuthorError::new( + "unsafe-declaration-path", + format!("refusing non-regular declaration path {}", path.display()), + )); + } + let original = fs::read(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + let original_version = SourceVersion::from_metadata(&metadata); + let text = std::str::from_utf8(&original).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("declaration {} is not UTF-8: {error}", path.display()), + ) + })?; + let document = KdlDocument::parse(text).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("parsing declaration {}: {error}", path.display()), + ) + })?; + let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; + if is_nix_managed(target) { + return Err(AuthorError::new( + "nix-managed-declaration", + format!( + "agent {expected_identity:?} is Nix-owned; edit its Nix source instead of {}", + path.display() + ), + )); + } + // No span-bounded field edit may change the subject's immutable agent ID (R24). Address, name, + // and description are all mutable; `id` is the one value that identifies the subject across + // every one of those changes, so the candidate must read back with the exact same bytes — or + // with none, on a declaration ID migration has not reached yet. + let expected_id = declared_id(target); + let Some(replacement) = presentation_edit(text, target, field, requested)? else { + return Ok(AuthorOutcome::Unchanged); + }; + verify_candidate( + &replacement, + expected_identity, + expected_host, + expected_agent, + field, + requested, + expected_id.as_deref(), + )?; + atomic_replace_checked( + catalog_lock, + catalog, + control, + path, + &original, + original_version, + replacement.as_bytes(), + metadata.permissions().mode() & 0o7777, + before_commit, + )?; + Ok(AuthorOutcome::Changed) +} + +/// The declaration's explicit immutable `id`, if it carries one. +fn declared_id(node: &KdlNode) -> Option { + node.children()? + .nodes() + .iter() + .find(|child| child.name().value() == "id") + .and_then(|child| child.get(0)) + .and_then(|value| value.as_string()) + .map(str::to_owned) +} + +fn presentation_edit( + text: &str, + target: &KdlNode, + field: DeclaredField, + requested: Option<&str>, +) -> Result, AuthorError> { + let fields = target + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| child.name().value() == field.as_str()) + .collect::>(); + match fields.as_slice() { + [] => match requested { + Some(value) => insert_field(text, target, field, value).map(Some), + None => Ok(None), + }, + [node] => match requested { + Some(value) => replace_field(text, node, field, value), + None => remove_field(text, node).map(Some), + }, + _ => Err(AuthorError::new( + "duplicate-presentation-field", + format!("target declares `{}` more than once", field.as_str()), + )), + } +} + +fn parse_field_value(node: &KdlNode, field: DeclaredField) -> Result<&str, AuthorError> { + if node.children().is_some() || node.entries().len() != 1 || node.entries()[0].name().is_some() + { + return Err(AuthorError::new( + "malformed-presentation-field", + format!( + "`{}` must contain exactly one positional string", + field.as_str() + ), + )); + } + node.get(0) + .and_then(|value| value.as_string()) + .ok_or_else(|| { + AuthorError::new( + "malformed-presentation-field", + format!("`{}` must contain a string", field.as_str()), + ) + }) +} + +fn replace_field( + text: &str, + node: &KdlNode, + field: DeclaredField, + value: &str, +) -> Result, AuthorError> { + if parse_field_value(node, field)? == value { + return Ok(None); + } + let entry = &node.entries()[0]; + let span = entry.span(); + let range = span.offset()..span.offset() + span.len(); + text.get(range.clone()).ok_or_else(|| { + AuthorError::new( + "malformed-declaration", + "presentation value span falls outside the declaration", + ) + })?; + let mut replacement = text.to_owned(); + replacement.replace_range(range, "ed(value)?); + Ok(Some(replacement)) +} + +fn insert_field( + text: &str, + target: &KdlNode, + field: DeclaredField, + value: &str, +) -> Result { + insert_node( + text, + target, + &format!("{} {}", field.as_str(), quoted(value)?), + ) +} + +pub(super) fn verify_candidate( + candidate: &str, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + field: DeclaredField, + expected: Option<&str>, + expected_id: Option<&str>, +) -> Result<(), AuthorError> { + let document = KdlDocument::parse(candidate).map_err(|error| { + AuthorError::new( + "unsafe-source-edit", + format!("field edit did not produce valid KDL: {error}"), + ) + })?; + let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; + if declared_id(target).as_deref() != expected_id { + return Err(AuthorError::new( + "agent-id-immutable", + format!( + "edit would change the immutable agent id of {expected_identity:?}; `id` is the \ + one declared value no authoring command may rewrite" + ), + )); + } + let fields = target + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| child.name().value() == field.as_str()) + .collect::>(); + let observed = match fields.as_slice() { + [] => None, + [node] => Some(parse_field_value(node, field)?), + _ => { + return Err(AuthorError::new( + "unsafe-source-edit", + format!( + "presentation edit produced duplicate `{}` fields", + field.as_str() + ), + )); + } + }; + if observed != expected { + return Err(AuthorError::new( + "unsafe-source-edit", + format!( + "presentation edit did not produce the requested `{}`", + field.as_str() + ), + )); + } + Ok(()) +} diff --git a/src/agent_author/desired_state.rs b/src/agent_author/desired_state.rs new file mode 100644 index 00000000..8f0ec69e --- /dev/null +++ b/src/agent_author/desired_state.rs @@ -0,0 +1,315 @@ +//! Whole-agent lifecycle desired state. +//! +//! Moved verbatim out of `agent_author.rs`; the shared declaration-writer primitives (error +//! vocabulary, target resolution, KDL node location, span edits, atomic commit) stay there. + +use super::*; + +/// Stable authored desired-state selector. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "lowercase")] +pub enum DesiredStateValue { + Running, + Suspended, + Retired, +} + +impl DesiredStateValue { + pub fn as_str(self) -> &'static str { + match self { + Self::Running => "running", + Self::Suspended => "suspended", + Self::Retired => "retired", + } + } +} + +/// Stable machine-readable receipt from one desired-state edit. +/// +/// `managed_by` is the ownership marker the caller asserted and the declaration confirmed, so the +/// receipt records which authority admitted the edit; `null` is the ordinary unmarked path. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct DesiredStateReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub desired_state: DesiredStateValue, + pub reason: Option, + pub managed_by: Option, +} + +/// Author one whole-agent desired state without claiming runtime convergence. +/// +/// `managed_by` is the ownership marker the caller asserts owns the declaration. `None` is the +/// ordinary path and refuses a Nix-owned declaration, exactly as presentation, address, stream, and +/// Resource authoring do. `Some(marker)` is a generator saying "I am the writer of these bytes", +/// and is admitted only when the declaration's own `meta { managed-by "..." }` names exactly that +/// marker: it is the projection's typed route to the one transition its own source can no longer +/// express, because the source edit that has to be projected is the seat's removal (#473). +pub fn set_desired_state( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + state: DesiredStateValue, + reason: Option<&str>, + managed_by: Option<&str>, +) -> Result { + match state { + DesiredStateValue::Running if reason.is_some() => { + return Err(AuthorError::new( + "invalid-desired-state", + "running desired state forbids --reason", + )); + } + DesiredStateValue::Suspended | DesiredStateValue::Retired if reason.is_none() => { + return Err(AuthorError::new( + "invalid-desired-state", + format!("{} desired state requires --reason", state.as_str()), + )); + } + _ => {} + } + if let Some(reason) = reason { + validate_desired_state_reason(reason) + .map_err(|error| AuthorError::new("invalid-desired-state", error.to_string()))?; + } + validate_marker_assertion(managed_by)?; + let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { + AuthorError::new( + "catalog-lock-failed", + format!("acquire catalog-authoring lock: {error:#}"), + ) + })?; + let found = crate::discover(catalog_root); + if let Some(error) = found.errors.first() { + return Err(AuthorError::new( + "catalog-malformed", + format!( + "cannot prove an exact desired-state target while {} is malformed: {}", + error.path.display(), + error.message + ), + )); + } + let target = resolve_target(&found.specs, selector, this_host)?; + authorize_actor( + &found.specs, + &target.identity, + this_host, + actor, + "desired-state-not-authorized", + )?; + let result = edit_desired_state_declaration( + &catalog_lock, + catalog_root, + &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) + .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, + &target.declaration, + &target.identity, + &target.source_host, + &target.source_identity, + state, + reason, + managed_by, + || {}, + )?; + Ok(DesiredStateReceipt { + result, + identity: target.identity, + desired_state: state, + reason: reason.map(str::to_owned), + managed_by: managed_by.map(str::to_owned), + }) +} + +#[allow(clippy::too_many_arguments)] +pub(super) fn edit_desired_state_declaration( + catalog_lock: &CatalogLock, + catalog: &Path, + control: &Path, + path: &Path, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + state: DesiredStateValue, + reason: Option<&str>, + managed_by: Option<&str>, + before_commit: impl FnOnce(), +) -> Result { + if path.extension().and_then(|value| value.to_str()) != Some("kdl") { + return Err(AuthorError::new( + "unsupported-declaration-format", + format!( + "desired-state authoring requires canonical KDL, found {}", + path.display() + ), + )); + } + let metadata = fs::symlink_metadata(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + if !metadata.file_type().is_file() { + return Err(AuthorError::new( + "unsafe-declaration-path", + format!("refusing non-regular declaration path {}", path.display()), + )); + } + let original = fs::read(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + let original_version = SourceVersion::from_metadata(&metadata); + let text = std::str::from_utf8(&original).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("declaration {} is not UTF-8: {error}", path.display()), + ) + })?; + let document = KdlDocument::parse(text).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("parsing declaration {}: {error}", path.display()), + ) + })?; + let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; + let marker_matched = authorize_marker(target, expected_identity, path, managed_by)?; + let Some(replacement) = desired_state_edit(text, target, state, reason)? else { + return Ok(AuthorOutcome::Unchanged); + }; + verify_desired_state_candidate( + &replacement, + expected_identity, + expected_host, + expected_agent, + state, + reason, + )?; + // A marker-matched edit stands in for the CAS `agent publish` the projection would otherwise + // have to perform, so it inherits that path's admission gate rather than only the local + // candidate reparse: the whole prospective catalog must still be admissible. That is what + // makes retiring a supervisor with a live descendant refuse (`retired-root`) instead of + // committing bytes the next reconcile pass rejects (#434). + if marker_matched + && let Err(error) = crate::agent_publish::admit_declaration_rewrite( + catalog, + control, + path, + replacement.as_bytes(), + ) + { + return Err(AuthorError::new( + "candidate-not-admissible", + format!("{error:#}"), + )); + } + atomic_replace_checked( + catalog_lock, + catalog, + control, + path, + &original, + original_version, + replacement.as_bytes(), + metadata.permissions().mode() & 0o7777, + before_commit, + )?; + Ok(AuthorOutcome::Changed) +} + +fn desired_state_edit( + text: &str, + target: &KdlNode, + state: DesiredStateValue, + reason: Option<&str>, +) -> Result, AuthorError> { + let lifecycle = target + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| matches!(child.name().value(), "desired-state" | "retired")) + .collect::>(); + if lifecycle.len() > 1 { + return Err(AuthorError::new( + "duplicate-lifecycle-field", + "target declares more than one lifecycle field", + )); + } + if state == DesiredStateValue::Running { + return lifecycle + .first() + .map(|node| remove_field(text, node).map(Some)) + .unwrap_or(Ok(None)); + } + let authored = format!( + "desired-state {} reason={}", + quoted(state.as_str())?, + quoted(reason.expect("validated by set_desired_state"))? + ); + match lifecycle.as_slice() { + [] => insert_node(text, target, &authored).map(Some), + [node] => { + let span = node.span(); + let range = span.offset()..span.offset() + span.len(); + if text.get(range.clone()) == Some(authored.as_str()) { + return Ok(None); + } + let mut replacement = text.to_owned(); + replacement.replace_range(range, &authored); + Ok(Some(replacement)) + } + _ => unreachable!(), + } +} + +fn verify_desired_state_candidate( + candidate: &str, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + state: DesiredStateValue, + reason: Option<&str>, +) -> Result<(), AuthorError> { + let document = KdlDocument::parse(candidate).map_err(|error| { + AuthorError::new( + "unsafe-source-edit", + format!("desired-state edit did not produce valid KDL: {error}"), + ) + })?; + let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; + let lifecycle = target + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| matches!(child.name().value(), "desired-state" | "retired")) + .collect::>(); + if state == DesiredStateValue::Running { + if lifecycle.is_empty() { + return Ok(()); + } + } else if let [node] = lifecycle.as_slice() + && node.name().value() == "desired-state" + && node.get(0).and_then(|entry| entry.as_string()) == Some(state.as_str()) + && node.get("reason").and_then(|entry| entry.as_string()) == reason + { + return Ok(()); + } + Err(AuthorError::new( + "unsafe-source-edit", + "desired-state candidate did not read back as the authored intent", + )) +} + +/// The lifecycle verb's marker authority, read from the exact declaration node it edits. +fn authorize_marker( + target: &KdlNode, + expected_identity: &str, + path: &Path, + asserted: Option<&str>, +) -> Result { + authorize_asserted_marker(&declared_markers(target), expected_identity, path, asserted) +} diff --git a/src/agent_author/markers.rs b/src/agent_author/markers.rs new file mode 100644 index 00000000..f5cfd424 --- /dev/null +++ b/src/agent_author/markers.rs @@ -0,0 +1,120 @@ +//! Ownership-marker authority: who may rewrite a declaration carrying `meta { managed-by }`. +//! +//! Moved verbatim out of `agent_author.rs`; the shared declaration-writer primitives (error +//! vocabulary, target resolution, KDL node location, span edits, atomic commit) stay there. + +use super::*; + +/// Every ownership marker this declaration carries, in source order. +/// +/// A well-formed declaration carries at most one. Several is not a resolvable ownership claim, so +/// they are returned as-is and no assertion can match them. +pub(super) fn declared_markers(node: &KdlNode) -> Vec<&str> { + node.children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| child.name().value() == "meta") + .filter_map(KdlNode::children) + .flat_map(|meta| meta.nodes()) + .filter(|child| child.name().value() == "managed-by") + .filter_map(|child| child.get(0).and_then(|value| value.as_string())) + .collect() +} + +pub(super) fn is_nix_managed(node: &KdlNode) -> bool { + declared_markers(node).contains(&"nix") +} + +/// Reject an ownership assertion no declaration could carry, before any lock or read. +/// +/// A marker is compared byte-exactly against the declaration's own `meta { managed-by "..." }` +/// value, so an empty or padded assertion can only ever be a caller mistake. +pub(crate) fn validate_marker_assertion(asserted: Option<&str>) -> Result<(), AuthorError> { + if let Some(marker) = asserted + && (marker.is_empty() || marker.trim() != marker) + { + return Err(AuthorError::new( + "invalid-managed-by", + format!("asserted ownership marker {marker:?} is empty or padded"), + )); + } + Ok(()) +} + +/// Every ownership marker the declaration source `bytes` carries, in source order. +/// +/// `agent publish` replaces a whole declaration file rather than one node, so every marker any +/// `agent` node in the incumbent file declares is at stake in that publication and the union is +/// what an assertion has to resolve against (#486). Bytes that are not UTF-8 or do not parse as +/// KDL carry no discoverable claim and read as unmarked: a writer able to leave such bytes in a +/// declaration leaf already holds direct filesystem write authority over the file, which is +/// strictly stronger than any st2 write path this marker governs. +pub(crate) fn declaration_markers(bytes: &[u8]) -> Vec { + let Ok(text) = std::str::from_utf8(bytes) else { + return Vec::new(); + }; + let Ok(document) = text.parse::() else { + return Vec::new(); + }; + document + .nodes() + .iter() + .filter(|node| node.name().value() == "agent") + .flat_map(|node| { + declared_markers(node) + .into_iter() + .map(str::to_owned) + .collect::>() + }) + .collect() +} + +/// Decide whether `asserted` authorizes rewriting bytes carrying the `declared` markers. +/// +/// `meta { managed-by "nix" }` says the Nix projection, not st2, is the writer of these bytes: an +/// edit made behind it is silently reverted on the next activation, which is why an unasserted +/// rewrite refuses (R25, decision 0003). Only that marker refuses; the others are labels on +/// declarations st2's own verbs and publishers are expected to rewrite. +/// +/// The generator itself is the one writer that legitimately authors the declaration, and +/// `--managed-by` is how it says so. An assertion is admitted only when it names exactly the one +/// marker the declaration carries — a caller wrong about who owns the bytes is wrong about the +/// edit, so a mismatched marker, an unmarked declaration, and an unresolvable multi-marker +/// declaration all fail closed. Returns whether an assertion was matched. +pub(crate) fn authorize_asserted_marker( + declared: &[&str], + subject: &str, + path: &Path, + asserted: Option<&str>, +) -> Result { + match (asserted, declared) { + (None, _) if !declared.contains(&"nix") => Ok(false), + (None, _) => Err(AuthorError::new( + "nix-managed-declaration", + format!( + "agent {subject:?} is Nix-owned; edit its Nix source instead of {}, or pass --managed-by \"nix\" if you are that projection", + path.display() + ), + )), + (Some(asserted), [marker]) if *marker == asserted => Ok(true), + (Some(asserted), []) => Err(AuthorError::new( + "managed-by-unmarked", + format!( + "--managed-by {asserted:?} claims agent {subject:?}, whose declaration {} carries no `meta {{ managed-by }}` marker", + path.display() + ), + )), + (Some(asserted), markers) => Err(AuthorError::new( + "managed-by-mismatch", + format!( + "--managed-by {asserted:?} does not own agent {subject:?}: {} declares owner {}", + path.display(), + markers + .iter() + .map(|marker| format!("{marker:?}")) + .collect::>() + .join(" and ") + ), + )), + } +} diff --git a/src/agent_author/resource.rs b/src/agent_author/resource.rs new file mode 100644 index 00000000..c644633c --- /dev/null +++ b/src/agent_author/resource.rs @@ -0,0 +1,613 @@ +//! Typed Resource bindings: add, remove, rename. +//! +//! Moved verbatim out of `agent_author.rs`; the shared declaration-writer primitives (error +//! vocabulary, target resolution, KDL node location, span edits, atomic commit) stay there. + +use super::*; + +/// Stable machine-readable receipt from adding or updating one Resource binding. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct ResourceAddReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub name: String, + pub uri: String, + pub reason: String, + pub inactive_reason: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub selector: Option, +} + +/// Stable machine-readable receipt from removing one Resource binding. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct ResourceRemoveReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub name: String, +} + +/// Stable machine-readable receipt from relabelling one Resource binding. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct ResourceRenameReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub old: String, + pub new: String, +} + +/// Declare one Resource binding, or update the binding that already carries `name`. +/// +/// st2 preserves the binding for readers; it resolves nothing and grants nothing. `uri` is the +/// exact absolute identity and is stored byte for byte with no normalization. +#[allow(clippy::too_many_arguments)] +pub fn add_resource( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + name: &str, + uri: &str, + reason: &str, + inactive_reason: Option<&str>, +) -> Result { + add_resource_with_selector( + catalog_root, + selector, + this_host, + actor, + name, + uri, + reason, + inactive_reason, + None, + ) +} + +#[allow(clippy::too_many_arguments)] +pub fn add_resource_with_selector( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + name: &str, + uri: &str, + reason: &str, + inactive_reason: Option<&str>, + resource_selector: Option<&serde_json::Value>, +) -> Result { + author_resource( + catalog_root, + selector, + this_host, + actor, + ResourceIntent::Upsert { + name, + uri, + reason, + inactive_reason, + selector: resource_selector, + }, + ) + .map(|(result, identity)| ResourceAddReceipt { + result, + identity, + name: name.to_owned(), + uri: uri.to_owned(), + reason: reason.to_owned(), + inactive_reason: inactive_reason.map(str::to_owned), + selector: resource_selector.cloned(), + }) +} + +/// Remove one Resource binding. An already absent binding is an idempotent success. +pub fn remove_resource( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + name: &str, +) -> Result { + author_resource( + catalog_root, + selector, + this_host, + actor, + ResourceIntent::Remove { name }, + ) + .map(|(result, identity)| ResourceRemoveReceipt { + result, + identity, + name: name.to_owned(), + }) +} + +/// Relabel one Resource binding, carrying its `uri`, `reason`, and `inactive-reason` unchanged. +/// +/// An absent `old` and an already declared `new` both refuse: binding names are unique within one +/// agent, so neither request has an outcome that preserves the caller's intent. +pub fn rename_resource( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + old: &str, + new: &str, +) -> Result { + author_resource( + catalog_root, + selector, + this_host, + actor, + ResourceIntent::Rename { old, new }, + ) + .map(|(result, identity)| ResourceRenameReceipt { + result, + identity, + old: old.to_owned(), + new: new.to_owned(), + }) +} + +/// One requested Resource-binding mutation, resolved against the declaration under the lock. +#[derive(Debug, Clone, Copy)] +enum ResourceIntent<'a> { + Upsert { + name: &'a str, + uri: &'a str, + reason: &'a str, + inactive_reason: Option<&'a str>, + selector: Option<&'a serde_json::Value>, + }, + Remove { + name: &'a str, + }, + Rename { + old: &'a str, + new: &'a str, + }, +} + +/// The binding state a candidate must read back as before it may be committed. +#[derive(Debug)] +struct ResourceExpectation { + absent: Option, + present: Option, +} + +fn author_resource( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + intent: ResourceIntent<'_>, +) -> Result<(AuthorOutcome, String), AuthorError> { + let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { + AuthorError::new( + "catalog-lock-failed", + format!("acquire catalog-authoring lock: {error:#}"), + ) + })?; + let found = crate::discover_strict(catalog_root); + if let Some(error) = found.errors.first() { + return Err(AuthorError::new( + "catalog-malformed", + format!( + "cannot prove an exact resource target while {} is malformed: {}", + error.path.display(), + error.message + ), + )); + } + let target = resolve_target(&found.specs, selector, this_host)?; + let actor = actor + .map(|actor| resolve_target(&found.specs, actor, this_host).map(|target| target.identity)) + .transpose()?; + authorize_actor( + &found.specs, + &target.identity, + this_host, + actor.as_deref(), + "resource-not-authorized", + )?; + let result = edit_resource_declaration( + &catalog_lock, + catalog_root, + &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) + .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, + &target.declaration, + &target.identity, + &target.source_host, + &target.source_identity, + intent, + || {}, + )?; + Ok((result, target.identity)) +} + +#[allow(clippy::too_many_arguments)] +fn edit_resource_declaration( + catalog_lock: &CatalogLock, + catalog: &Path, + control: &Path, + path: &Path, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + intent: ResourceIntent<'_>, + before_commit: impl FnOnce(), +) -> Result { + if path.extension().and_then(|value| value.to_str()) != Some("kdl") { + return Err(AuthorError::new( + "unsupported-declaration-format", + format!( + "resource authoring requires canonical KDL, found {}", + path.display() + ), + )); + } + let metadata = fs::symlink_metadata(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + if !metadata.file_type().is_file() { + return Err(AuthorError::new( + "unsafe-declaration-path", + format!("refusing non-regular declaration path {}", path.display()), + )); + } + let original = fs::read(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + let original_version = SourceVersion::from_metadata(&metadata); + let text = std::str::from_utf8(&original).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("declaration {} is not UTF-8: {error}", path.display()), + ) + })?; + let document = KdlDocument::parse(text).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("parsing declaration {}: {error}", path.display()), + ) + })?; + let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; + if is_nix_managed(target) { + return Err(AuthorError::new( + "nix-managed-declaration", + format!( + "agent {expected_identity:?} is Nix-owned; edit its Nix source instead of {}", + path.display() + ), + )); + } + let Some((replacement, expectation)) = resource_edit(text, target, intent)? else { + return Ok(AuthorOutcome::Unchanged); + }; + verify_resource_candidate( + catalog, + path, + &replacement, + expected_identity, + expected_host, + expected_agent, + &expectation, + )?; + atomic_replace_checked( + catalog_lock, + catalog, + control, + path, + &original, + original_version, + replacement.as_bytes(), + metadata.permissions().mode() & 0o7777, + before_commit, + )?; + Ok(AuthorOutcome::Changed) +} + +/// Resolve one intent against the declared bindings, preserving every unrelated byte. +/// +/// `Ok(None)` is the proven no-op: an unchanged upsert, an absent removal, or a self-rename. A +/// changed upsert rewrites exactly the one binding node in place, so its position, its leading +/// trivia, and every sibling binding survive. +fn resource_edit( + text: &str, + target: &KdlNode, + intent: ResourceIntent<'_>, +) -> Result, AuthorError> { + let declared = target + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| child.name().value() == "resource") + .collect::>(); + let declaring = |name: &str| -> Result, AuthorError> { + let matches = declared + .iter() + .copied() + .filter(|child| child.get(0).and_then(|entry| entry.as_string()) == Some(name)) + .collect::>(); + if matches.len() > 1 { + return Err(AuthorError::new( + "duplicate-resource", + format!("target declares resource {name:?} more than once"), + )); + } + Ok(matches.first().copied()) + }; + match intent { + ResourceIntent::Upsert { + name, + uri, + reason, + inactive_reason, + selector, + } => { + let authored = declared_resource(name, uri, reason, inactive_reason, selector)?; + let replacement = match declaring(name)? { + Some(node) if parsed_resource(node)? == authored => return Ok(None), + Some(node) => replace_node(text, node, &render_resource(&authored)?)?, + None => insert_node(text, target, &render_resource(&authored)?)?, + }; + Ok(Some(( + replacement, + ResourceExpectation { + absent: None, + present: Some(authored), + }, + ))) + } + ResourceIntent::Remove { name } => { + let Some(node) = declaring(name)? else { + return Ok(None); + }; + Ok(Some(( + remove_field(text, node)?, + ResourceExpectation { + absent: Some(name.to_owned()), + present: None, + }, + ))) + } + ResourceIntent::Rename { old, new } => { + let Some(node) = declaring(old)? else { + return Err(AuthorError::new( + "resource-not-found", + format!("target declares no resource {old:?}"), + )); + }; + if old == new { + return Ok(None); + } + if declaring(new)?.is_some() { + return Err(AuthorError::new( + "resource-already-exists", + format!( + "target already declares resource {new:?}; binding names are unique within one agent" + ), + )); + } + let carried = parsed_resource(node)?; + let renamed = declared_resource( + new, + carried.uri(), + carried.reason(), + carried.inactive_reason(), + carried.selector(), + )?; + Ok(Some(( + replace_node(text, node, &render_resource(&renamed)?)?, + ResourceExpectation { + absent: Some(old.to_owned()), + present: Some(renamed), + }, + ))) + } + } +} + +/// Enforce the canonical binding invariants — `agent_spec` owns them; this mints no new rule. +fn declared_resource( + name: &str, + uri: &str, + reason: &str, + inactive_reason: Option<&str>, + selector: Option<&serde_json::Value>, +) -> Result { + let resource = match inactive_reason { + None => Resource::new(name.to_owned(), uri.to_owned(), reason.to_owned()), + Some(inactive_reason) => Resource::new_inactive( + name.to_owned(), + uri.to_owned(), + reason.to_owned(), + inactive_reason.to_owned(), + ), + } + .map_err(|error| AuthorError::new("invalid-resource", error))?; + Ok(match selector { + Some(selector) => resource.with_selector(selector.clone()), + None => resource, + }) +} + +fn parsed_resource(node: &KdlNode) -> Result { + let malformed = + |detail: &str| AuthorError::new("malformed-resource", format!("resource binding {detail}")); + if node.children().is_some() { + return Err(malformed("cannot have children")); + } + let mut name = None; + let mut uri = None; + let mut reason = None; + let mut inactive_reason = None; + let mut selector = None; + for entry in node.entries() { + let value = entry + .value() + .as_string() + .ok_or_else(|| malformed("accepts only string values"))?; + match entry.name().map(|name| name.value()) { + None => { + if name.replace(value).is_some() { + return Err(malformed("declares one of its fields more than once")); + } + } + Some("uri") => { + if uri.replace(value).is_some() { + return Err(malformed("declares one of its fields more than once")); + } + } + Some("reason") => { + if reason.replace(value).is_some() { + return Err(malformed("declares one of its fields more than once")); + } + } + Some("inactive-reason") => { + if inactive_reason.replace(value).is_some() { + return Err(malformed("declares one of its fields more than once")); + } + } + Some("selector") => { + if selector.is_some() { + return Err(malformed("declares one of its fields more than once")); + } + selector = Some(serde_json::from_str(value).map_err(|error| { + malformed(&format!("has invalid JSON `selector`: {error}")) + })?); + } + Some(other) => return Err(malformed(&format!("has unsupported property `{other}`"))), + } + } + let (Some(name), Some(uri), Some(reason)) = (name, uri, reason) else { + return Err(malformed("needs a name, a `uri`, and a `reason`")); + }; + declared_resource(name, uri, reason, inactive_reason, selector.as_ref()) +} + +fn render_resource(resource: &Resource) -> Result { + let mut authored = format!( + "resource {} uri={} reason={}", + quoted(resource.name())?, + quoted(resource.uri())?, + quoted(resource.reason())? + ); + if let Some(inactive_reason) = resource.inactive_reason() { + authored.push_str(&format!(" inactive-reason={}", quoted(inactive_reason)?)); + } + if let Some(selector) = resource.selector() { + authored.push_str(" selector="); + authored.push_str(&raw_json(selector)?); + } + Ok(authored) +} + +fn raw_json(value: &serde_json::Value) -> Result { + let json = serde_json::to_string(value).map_err(|error| { + AuthorError::new( + "invalid-resource", + format!("serialize Resource selector as canonical JSON: {error}"), + ) + })?; + for hashes in 1..=json.len() + 1 { + let fence = "#".repeat(hashes); + if !json.contains(&format!("\"{fence}")) { + return Ok(format!("{fence}\"{json}\"{fence}")); + } + } + unreachable!("a delimiter longer than the JSON payload cannot occur in the payload") +} + +/// Replace exactly one node's source span. A KDL node span carries neither the leading trivia nor +/// the trailing terminator, so the surrounding line survives untouched. +fn replace_node(text: &str, node: &KdlNode, authored: &str) -> Result { + let span = node.span(); + let range = span.offset()..span.offset() + span.len(); + text.get(range.clone()).ok_or_else(|| { + AuthorError::new( + "malformed-declaration", + "resource binding span falls outside the declaration", + ) + })?; + // The span can run to the start of trailing trivia, so replacing it verbatim would glue the + // rendered node onto a following `// comment`. Leave that separator in the source. + let kept = text[range.clone()].trim_end_matches([' ', '\t']).len(); + let mut replacement = text.to_owned(); + replacement.replace_range(range.start..range.start + kept, authored); + Ok(replacement) +} + +fn verify_resource_candidate( + catalog: &Path, + path: &Path, + candidate: &str, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + expectation: &ResourceExpectation, +) -> Result<(), AuthorError> { + let temporary = tempfile::tempdir() + .map_err(|error| AuthorError::new("unsafe-source-edit", error.to_string()))?; + let relative = path.strip_prefix(catalog).map_err(|_| { + AuthorError::new( + "unsafe-declaration-path", + format!( + "declaration {} is outside catalog {}", + path.display(), + catalog.display() + ), + ) + })?; + let candidate_path = temporary.path().join(relative); + fs::create_dir_all( + candidate_path + .parent() + .expect("candidate declaration has a parent"), + ) + .and_then(|()| fs::write(&candidate_path, candidate)) + .map_err(|error| { + AuthorError::new( + "unsafe-source-edit", + format!("stage resource validation: {error}"), + ) + })?; + let (specs, _) = agent_spec::discover_file(temporary.path(), &candidate_path) + .map_err(|error| AuthorError::new("invalid-resource", error.to_string()))?; + let spec = specs + .iter() + .find(|spec| { + spec.identity == expected_agent && spec.bus_id(expected_host) == expected_identity + }) + .ok_or_else(|| { + AuthorError::new( + "unsafe-source-edit", + "resource candidate lost the authored agent", + ) + })?; + let declares = |name: &str| { + spec.resources + .iter() + .find(|resource| resource.name() == name) + }; + if expectation + .absent + .as_deref() + .is_some_and(|name| declares(name).is_some()) + || expectation + .present + .as_ref() + .is_some_and(|expected| declares(expected.name()) != Some(expected)) + { + return Err(AuthorError::new( + "unsafe-source-edit", + "resource candidate did not read back as the authored intent", + )); + } + Ok(()) +} diff --git a/src/agent_author/stream.rs b/src/agent_author/stream.rs new file mode 100644 index 00000000..e8f4341d --- /dev/null +++ b/src/agent_author/stream.rs @@ -0,0 +1,414 @@ +//! Agent-owned stream declarations. +//! +//! Moved verbatim out of `agent_author.rs`; the shared declaration-writer primitives (error +//! vocabulary, target resolution, KDL node location, span edits, atomic commit) stay there. + +use crate::run::Runner as _; + +use super::*; + +/// Stable machine-readable receipt from adding one agent-owned stream. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct StreamAddReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub name: String, + pub launch: Option, +} + +/// Stable machine-readable receipt from removing one agent-owned stream. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct StreamRemoveReceipt { + pub result: AuthorOutcome, + pub identity: String, + pub name: String, +} + +/// Add an agent-owned stream, or prove that the identical declaration already exists. +pub fn add_stream( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + name: &str, + launch: Option, +) -> Result { + author_stream( + catalog_root, + selector, + this_host, + actor, + name, + launch.as_ref(), + false, + ) + .map(|(result, identity)| StreamAddReceipt { + result, + identity, + name: name.to_owned(), + launch, + }) +} + +/// Remove one agent-owned stream. An already absent stream is an idempotent success. +pub fn remove_stream( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + name: &str, +) -> Result { + author_stream(catalog_root, selector, this_host, actor, name, None, true).map( + |(result, identity)| StreamRemoveReceipt { + result, + identity, + name: name.to_owned(), + }, + ) +} + +fn author_stream( + catalog_root: &Path, + selector: &str, + this_host: &str, + actor: Option<&str>, + name: &str, + launch: Option<&StreamLaunch>, + remove: bool, +) -> Result<(AuthorOutcome, String), AuthorError> { + let catalog_lock = CatalogLock::exclusive(catalog_root).map_err(|error| { + AuthorError::new( + "catalog-lock-failed", + format!("acquire catalog-authoring lock: {error:#}"), + ) + })?; + let found = crate::discover_strict(catalog_root); + if let Some(error) = found.errors.first() { + return Err(AuthorError::new( + "catalog-malformed", + format!( + "cannot prove an exact stream target while {} is malformed: {}", + error.path.display(), + error.message + ), + )); + } + let target = resolve_target(&found.specs, selector, this_host)?; + let actor = actor + .map(|actor| resolve_target(&found.specs, actor, this_host).map(|target| target.identity)) + .transpose()?; + authorize_actor( + &found.specs, + &target.identity, + this_host, + actor.as_deref(), + "stream-not-authorized", + )?; + if remove { + let spec = found + .specs + .iter() + .find(|spec| spec.path == target.declaration) + .ok_or_else(|| { + AuthorError::new("stream-target-lost", "resolved stream target disappeared") + })?; + if spec + .streams + .iter() + .find(|stream| stream.name == name) + .is_some_and(|stream| stream.launch.is_some()) + { + let task_name = format!("{}{}", agent_spec::STREAM_TASK_PREFIX, name); + let task = spec + .tasks + .iter() + .find(|task| task.name == task_name) + .ok_or_else(|| { + AuthorError::new("stream-task-missing", "launched stream has no derived task") + })?; + let runtime_id = task + .id + .clone() + .unwrap_or_else(|| format!("{}.{}", spec.bus_id(this_host), task.name)); + let runner = crate::run::SystemRunner::new( + catalog_root.to_path_buf(), + crate::run::exec_state_dir(this_host), + ); + let live = runner + .list_sessions() + .map_err(|error| { + AuthorError::new("stream-runtime-observation-failed", error.to_string()) + })? + .into_iter() + .any(|session| session.alive && session.pty_id == runtime_id); + if live { + runner.retire(&runtime_id).map_err(|error| { + AuthorError::new( + "stream-runtime-retirement-failed", + format!("retire launched stream runtime {runtime_id}: {error:#}"), + ) + })?; + } + } + } + let result = edit_stream_declaration( + &catalog_lock, + catalog_root, + &crate::catalog_transaction::retained_dir_path(catalog_lock.control()) + .map_err(|error| AuthorError::new("declaration-write-failed", error.to_string()))?, + &target.declaration, + &target.identity, + &target.source_host, + &target.source_identity, + name, + launch, + remove, + || {}, + )?; + Ok((result, target.identity)) +} + +#[allow(clippy::too_many_arguments)] +fn edit_stream_declaration( + catalog_lock: &CatalogLock, + catalog: &Path, + control: &Path, + path: &Path, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + name: &str, + launch: Option<&StreamLaunch>, + remove: bool, + before_commit: impl FnOnce(), +) -> Result { + if path.extension().and_then(|value| value.to_str()) != Some("kdl") { + return Err(AuthorError::new( + "unsupported-declaration-format", + format!( + "stream authoring requires canonical KDL, found {}", + path.display() + ), + )); + } + let metadata = fs::symlink_metadata(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + if !metadata.file_type().is_file() { + return Err(AuthorError::new( + "unsafe-declaration-path", + format!("refusing non-regular declaration path {}", path.display()), + )); + } + let original = fs::read(path).map_err(|error| { + AuthorError::new( + "declaration-read-failed", + format!("reading declaration {}: {error}", path.display()), + ) + })?; + let original_version = SourceVersion::from_metadata(&metadata); + let text = std::str::from_utf8(&original).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("declaration {} is not UTF-8: {error}", path.display()), + ) + })?; + let document = KdlDocument::parse(text).map_err(|error| { + AuthorError::new( + "malformed-declaration", + format!("parsing declaration {}: {error}", path.display()), + ) + })?; + let target = exact_agent_node(&document, expected_identity, expected_host, expected_agent)?; + if is_nix_managed(target) { + return Err(AuthorError::new( + "nix-managed-declaration", + format!( + "agent {expected_identity:?} is Nix-owned; edit its Nix source instead of {}", + path.display() + ), + )); + } + let replacement = stream_edit(text, target, name, launch, remove)?; + let Some(replacement) = replacement else { + return Ok(AuthorOutcome::Unchanged); + }; + verify_stream_candidate( + catalog, + path, + &replacement, + expected_identity, + expected_host, + expected_agent, + name, + launch, + remove, + )?; + atomic_replace_checked( + catalog_lock, + catalog, + control, + path, + &original, + original_version, + replacement.as_bytes(), + metadata.permissions().mode() & 0o7777, + before_commit, + )?; + Ok(AuthorOutcome::Changed) +} + +fn stream_edit( + text: &str, + target: &KdlNode, + name: &str, + launch: Option<&StreamLaunch>, + remove: bool, +) -> Result, AuthorError> { + let streams = target + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .filter(|child| { + child.name().value() == "stream" + && child.get(0).and_then(|entry| entry.as_string()) == Some(name) + }) + .collect::>(); + if streams.len() > 1 { + return Err(AuthorError::new( + "duplicate-stream", + format!("target declares stream {name:?} more than once"), + )); + } + if remove { + return streams + .first() + .map(|node| remove_field(text, node).map(Some)) + .unwrap_or(Ok(None)); + } + if let Some(existing) = streams.first() { + if parsed_stream_launch(existing)? == launch.cloned() { + return Ok(None); + } + return Err(AuthorError::new( + "stream-already-exists", + format!( + "stream {name:?} already exists with a different launch; remove it before adding a replacement" + ), + )); + } + let authored = match launch { + None => format!("stream {} {{}}", quoted(name)?), + Some(StreamLaunch::Command(command)) => format!( + "stream {} {{ command {} }}", + quoted(name)?, + quoted(command)? + ), + Some(StreamLaunch::Argv(argv)) => { + let values = argv + .iter() + .map(|value| quoted(value)) + .collect::, _>>()?; + format!("stream {} {{ argv {} }}", quoted(name)?, values.join(" ")) + } + }; + insert_node(text, target, &authored).map(Some) +} + +fn parsed_stream_launch(node: &KdlNode) -> Result, AuthorError> { + let children = node + .children() + .into_iter() + .flat_map(|children| children.nodes()) + .collect::>(); + match children.as_slice() { + [] => Ok(None), + [child] if child.name().value() == "command" => child + .get(0) + .and_then(|entry| entry.as_string()) + .map(|value| Some(StreamLaunch::Command(value.to_owned()))) + .ok_or_else(|| { + AuthorError::new("malformed-stream", "stream command must contain one string") + }), + [child] if child.name().value() == "argv" => { + let argv = child + .entries() + .iter() + .map(|entry| entry.value().as_string().map(str::to_owned)) + .collect::>>() + .ok_or_else(|| { + AuthorError::new("malformed-stream", "stream argv values must be strings") + })?; + Ok(Some(StreamLaunch::Argv(argv))) + } + _ => Err(AuthorError::new( + "malformed-stream", + "stream must contain exactly one command or argv node, or be empty", + )), + } +} + +fn verify_stream_candidate( + catalog: &Path, + path: &Path, + candidate: &str, + expected_identity: &str, + expected_host: &str, + expected_agent: &str, + name: &str, + launch: Option<&StreamLaunch>, + removed: bool, +) -> Result<(), AuthorError> { + let temporary = tempfile::tempdir() + .map_err(|error| AuthorError::new("unsafe-source-edit", error.to_string()))?; + let relative = path.strip_prefix(catalog).map_err(|_| { + AuthorError::new( + "unsafe-declaration-path", + format!( + "declaration {} is outside catalog {}", + path.display(), + catalog.display() + ), + ) + })?; + let candidate_path = temporary.path().join(relative); + fs::create_dir_all( + candidate_path + .parent() + .expect("candidate declaration has a parent"), + ) + .and_then(|()| fs::write(&candidate_path, candidate)) + .map_err(|error| { + AuthorError::new( + "unsafe-source-edit", + format!("stage stream validation: {error}"), + ) + })?; + let (specs, _) = agent_spec::discover_file(temporary.path(), &candidate_path) + .map_err(|error| AuthorError::new("invalid-stream", error.to_string()))?; + let spec = specs + .iter() + .find(|spec| { + spec.identity == expected_agent && spec.bus_id(expected_host) == expected_identity + }) + .ok_or_else(|| { + AuthorError::new( + "unsafe-source-edit", + "stream candidate lost the authored agent", + ) + })?; + let observed = spec.streams.iter().find(|stream| stream.name == name); + if removed && observed.is_none() + || !removed && observed.is_some_and(|stream| stream.launch.as_ref() == launch) + { + Ok(()) + } else { + Err(AuthorError::new( + "unsafe-source-edit", + "stream candidate did not read back as the authored intent", + )) + } +} diff --git a/src/catalog_transaction.rs b/src/catalog_transaction.rs index 3b8c6439..1f85d3b9 100644 --- a/src/catalog_transaction.rs +++ b/src/catalog_transaction.rs @@ -6,7 +6,7 @@ use std::collections::{BTreeMap, BTreeSet}; use std::fs::{self, File, OpenOptions}; use std::io::{Read as _, Write as _}; -use std::os::fd::{AsRawFd as _, FromRawFd as _}; +use std::os::fd::AsRawFd as _; use std::os::unix::fs::{MetadataExt as _, OpenOptionsExt as _, PermissionsExt as _}; use std::path::{Component, Path, PathBuf}; @@ -3344,316 +3344,8 @@ fn is_canonical_workspace_fact(path: &Path) -> bool { && matches!(components[3], Component::Normal(name) if name == ".workspace") } -fn openat_nofollow(parent: &File, name: &std::ffi::OsStr) -> Result { - use std::ffi::CString; - use std::os::unix::ffi::OsStrExt as _; - - let name = CString::new(name.as_bytes()).context("source entry name contains NUL")?; - let fd = unsafe { - libc::openat( - parent.as_raw_fd(), - name.as_ptr(), - libc::O_RDONLY | libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK, - ) - }; - if fd < 0 { - return Err(std::io::Error::last_os_error()).context("open retained source entry"); - } - Ok(unsafe { File::from_raw_fd(fd) }) -} - -fn capability_dir_entries(dir: &File) -> Result> { - let path = retained_dir_path(dir)?; - let mut names = fs::read_dir(&path)? - .map(|entry| entry.map(|entry| entry.file_name())) - .collect::>>()?; - names.sort(); - Ok(names) -} - -pub(crate) fn sync_dir(path: &Path) -> Result<()> { - File::open(path) - .with_context(|| format!("open directory {}", path.display()))? - .sync_all() - .with_context(|| format!("sync directory {}", path.display())) -} - -fn open_dir_nofollow(path: &Path) -> std::io::Result { - OpenOptions::new() - .read(true) - .custom_flags(libc::O_CLOEXEC | libc::O_DIRECTORY | libc::O_NOFOLLOW) - .open(path) -} - -pub(crate) fn open_dir_beneath(catalog: &Path, target: &Path) -> std::io::Result { - let relative = target.strip_prefix(catalog).map_err(|_| { - std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "directory escapes catalog", - ) - })?; - let mut current = open_dir_nofollow(catalog)?; - for component in relative.components() { - let Component::Normal(name) = component else { - return Err(std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "directory has an unsafe component", - )); - }; - current = openat_dir_nofollow(¤t, name)?; - } - Ok(current) -} - -pub(crate) fn openat_dir_nofollow(parent: &File, name: &std::ffi::OsStr) -> std::io::Result { - use std::ffi::CString; - use std::os::unix::ffi::OsStrExt as _; - let name = CString::new(name.as_bytes()).map_err(|_| { - std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "directory name contains NUL", - ) - })?; - let fd = unsafe { - libc::openat( - parent.as_raw_fd(), - name.as_ptr(), - libc::O_RDONLY | libc::O_CLOEXEC | libc::O_DIRECTORY | libc::O_NOFOLLOW, - ) - }; - if fd < 0 { - Err(std::io::Error::last_os_error()) - } else { - Ok(unsafe { File::from_raw_fd(fd) }) - } -} - -fn control_plane_rename_error(error: std::io::Error) -> anyhow::Error { - if error.raw_os_error() == Some(libc::EXDEV) { - anyhow::anyhow!( - "catalog control and declaration planes must share one filesystem for atomic publication" - ) - } else { - error.into() - } -} - -pub(crate) fn persist_tempfile_from_control( - control: &File, - catalog: &Path, - temp: tempfile::NamedTempFile, - target: &Path, -) -> std::io::Result<()> { - let source = temp.path(); - let target_parent = open_dir_beneath( - catalog, - target.parent().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no parent") - })?, - )?; - let source_name = source.file_name().ok_or_else(|| { - std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "temporary file has no name", - ) - })?; - let target_name = target.file_name().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no name") - })?; - renameat(control, source_name, &target_parent, target_name) -} - -pub(crate) fn link_tempfile_from_control( - control: &File, - catalog: &Path, - temp: &tempfile::NamedTempFile, - target: &Path, -) -> std::io::Result<()> { - let source = temp.path(); - let target_parent = open_dir_beneath( - catalog, - target.parent().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no parent") - })?, - )?; - let source_name = source.file_name().ok_or_else(|| { - std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "temporary file has no name", - ) - })?; - let target_name = target.file_name().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no name") - })?; - use std::ffi::CString; - use std::os::unix::ffi::OsStrExt as _; - let source_name = CString::new(source_name.as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "source name contains NUL") - })?; - let target_name = CString::new(target_name.as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target name contains NUL") - })?; - let result = unsafe { - libc::linkat( - control.as_raw_fd(), - source_name.as_ptr(), - target_parent.as_raw_fd(), - target_name.as_ptr(), - 0, - ) - }; - if result == 0 { - Ok(()) - } else { - Err(std::io::Error::last_os_error()) - } -} - -pub(crate) fn rename_noreplace_between_dirs( - control: &File, - catalog: &Path, - source: &Path, - target: &Path, -) -> std::io::Result<()> { - let target_parent = open_dir_beneath( - catalog, - target.parent().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no parent") - })?, - )?; - let source_name = source.file_name().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "source has no name") - })?; - let target_name = target.file_name().ok_or_else(|| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no name") - })?; - renameat_noreplace(control, source_name, &target_parent, target_name) -} - -fn renameat_noreplace( - source_parent: &File, - source: &std::ffi::OsStr, - target_parent: &File, - target: &std::ffi::OsStr, -) -> std::io::Result<()> { - use std::ffi::CString; - use std::os::unix::ffi::OsStrExt as _; - - let source = CString::new(source.as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "source name contains NUL") - })?; - let target = CString::new(target.as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target name contains NUL") - })?; - #[cfg(any(target_os = "linux", target_os = "android"))] - let result = unsafe { - libc::renameat2( - source_parent.as_raw_fd(), - source.as_ptr(), - target_parent.as_raw_fd(), - target.as_ptr(), - libc::RENAME_NOREPLACE, - ) - }; - #[cfg(target_os = "macos")] - let result = unsafe { - libc::renameatx_np( - source_parent.as_raw_fd(), - source.as_ptr(), - target_parent.as_raw_fd(), - target.as_ptr(), - libc::RENAME_EXCL, - ) - }; - #[cfg(not(any(target_os = "linux", target_os = "android", target_os = "macos")))] - let result = { - let _ = (source_parent, source, target_parent, target); - return Err(std::io::Error::new( - std::io::ErrorKind::Unsupported, - "atomic no-replace directory rename is unsupported on this platform", - )); - }; - if result == 0 { - Ok(()) - } else { - Err(std::io::Error::last_os_error()) - } -} - -fn renameat( - source_parent: &File, - source: &std::ffi::OsStr, - target_parent: &File, - target: &std::ffi::OsStr, -) -> std::io::Result<()> { - use std::ffi::CString; - use std::os::unix::ffi::OsStrExt as _; - - let source = CString::new(source.as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "source name contains NUL") - })?; - let target = CString::new(target.as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target name contains NUL") - })?; - let result = unsafe { - libc::renameat( - source_parent.as_raw_fd(), - source.as_ptr(), - target_parent.as_raw_fd(), - target.as_ptr(), - ) - }; - if result == 0 { - Ok(()) - } else { - Err(std::io::Error::last_os_error()) - } -} - -pub(crate) fn rename_noreplace(source: &Path, target: &Path) -> std::io::Result<()> { - use std::ffi::CString; - use std::os::unix::ffi::OsStrExt as _; - - let source = CString::new(source.as_os_str().as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "source contains NUL") - })?; - let target = CString::new(target.as_os_str().as_bytes()).map_err(|_| { - std::io::Error::new(std::io::ErrorKind::InvalidInput, "target contains NUL") - })?; - #[cfg(any(target_os = "linux", target_os = "android"))] - let result = unsafe { - libc::renameat2( - libc::AT_FDCWD, - source.as_ptr(), - libc::AT_FDCWD, - target.as_ptr(), - libc::RENAME_NOREPLACE, - ) - }; - #[cfg(target_os = "macos")] - let result = unsafe { - libc::renameatx_np( - libc::AT_FDCWD, - source.as_ptr(), - libc::AT_FDCWD, - target.as_ptr(), - libc::RENAME_EXCL, - ) - }; - #[cfg(not(any(target_os = "linux", target_os = "android", target_os = "macos")))] - let result = { - let _ = (source, target); - return Err(std::io::Error::new( - std::io::ErrorKind::Unsupported, - "atomic no-replace directory rename is unsupported on this platform", - )); - }; - if result == 0 { - Ok(()) - } else { - Err(std::io::Error::last_os_error()) - } -} +mod capability; +pub(crate) use capability::*; #[cfg(debug_assertions)] fn bootstrap_checkpoint(point: &str) { diff --git a/src/catalog_transaction/capability.rs b/src/catalog_transaction/capability.rs new file mode 100644 index 00000000..15aec3f9 --- /dev/null +++ b/src/catalog_transaction/capability.rs @@ -0,0 +1,325 @@ +//! Capability-style filesystem primitives for the catalog control plane. +//! +//! Moved verbatim out of the parent module: every `openat`/`renameat` call the transaction makes +//! through a retained directory descriptor, so no path is re-resolved between check and use. +//! Named `capability` rather than `fs` because the parent imports `std::fs`. + +use std::fs::{self, File, OpenOptions}; +use std::os::fd::{AsRawFd as _, FromRawFd as _}; +use std::os::unix::fs::OpenOptionsExt as _; +use std::path::Path; + +use anyhow::{Context as _, Result}; + +use super::*; + +pub(super) fn openat_nofollow(parent: &File, name: &std::ffi::OsStr) -> Result { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt as _; + + let name = CString::new(name.as_bytes()).context("source entry name contains NUL")?; + let fd = unsafe { + libc::openat( + parent.as_raw_fd(), + name.as_ptr(), + libc::O_RDONLY | libc::O_CLOEXEC | libc::O_NOFOLLOW | libc::O_NONBLOCK, + ) + }; + if fd < 0 { + return Err(std::io::Error::last_os_error()).context("open retained source entry"); + } + Ok(unsafe { File::from_raw_fd(fd) }) +} + +pub(super) fn capability_dir_entries(dir: &File) -> Result> { + let path = retained_dir_path(dir)?; + let mut names = fs::read_dir(&path)? + .map(|entry| entry.map(|entry| entry.file_name())) + .collect::>>()?; + names.sort(); + Ok(names) +} + +pub(crate) fn sync_dir(path: &Path) -> Result<()> { + File::open(path) + .with_context(|| format!("open directory {}", path.display()))? + .sync_all() + .with_context(|| format!("sync directory {}", path.display())) +} + +fn open_dir_nofollow(path: &Path) -> std::io::Result { + OpenOptions::new() + .read(true) + .custom_flags(libc::O_CLOEXEC | libc::O_DIRECTORY | libc::O_NOFOLLOW) + .open(path) +} + +pub(crate) fn open_dir_beneath(catalog: &Path, target: &Path) -> std::io::Result { + let relative = target.strip_prefix(catalog).map_err(|_| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "directory escapes catalog", + ) + })?; + let mut current = open_dir_nofollow(catalog)?; + for component in relative.components() { + let Component::Normal(name) = component else { + return Err(std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "directory has an unsafe component", + )); + }; + current = openat_dir_nofollow(¤t, name)?; + } + Ok(current) +} + +pub(crate) fn openat_dir_nofollow(parent: &File, name: &std::ffi::OsStr) -> std::io::Result { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt as _; + let name = CString::new(name.as_bytes()).map_err(|_| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "directory name contains NUL", + ) + })?; + let fd = unsafe { + libc::openat( + parent.as_raw_fd(), + name.as_ptr(), + libc::O_RDONLY | libc::O_CLOEXEC | libc::O_DIRECTORY | libc::O_NOFOLLOW, + ) + }; + if fd < 0 { + Err(std::io::Error::last_os_error()) + } else { + Ok(unsafe { File::from_raw_fd(fd) }) + } +} + +pub(super) fn control_plane_rename_error(error: std::io::Error) -> anyhow::Error { + if error.raw_os_error() == Some(libc::EXDEV) { + anyhow::anyhow!( + "catalog control and declaration planes must share one filesystem for atomic publication" + ) + } else { + error.into() + } +} + +pub(crate) fn persist_tempfile_from_control( + control: &File, + catalog: &Path, + temp: tempfile::NamedTempFile, + target: &Path, +) -> std::io::Result<()> { + let source = temp.path(); + let target_parent = open_dir_beneath( + catalog, + target.parent().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no parent") + })?, + )?; + let source_name = source.file_name().ok_or_else(|| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "temporary file has no name", + ) + })?; + let target_name = target.file_name().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no name") + })?; + renameat(control, source_name, &target_parent, target_name) +} + +pub(crate) fn link_tempfile_from_control( + control: &File, + catalog: &Path, + temp: &tempfile::NamedTempFile, + target: &Path, +) -> std::io::Result<()> { + let source = temp.path(); + let target_parent = open_dir_beneath( + catalog, + target.parent().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no parent") + })?, + )?; + let source_name = source.file_name().ok_or_else(|| { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "temporary file has no name", + ) + })?; + let target_name = target.file_name().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no name") + })?; + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt as _; + let source_name = CString::new(source_name.as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "source name contains NUL") + })?; + let target_name = CString::new(target_name.as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target name contains NUL") + })?; + let result = unsafe { + libc::linkat( + control.as_raw_fd(), + source_name.as_ptr(), + target_parent.as_raw_fd(), + target_name.as_ptr(), + 0, + ) + }; + if result == 0 { + Ok(()) + } else { + Err(std::io::Error::last_os_error()) + } +} + +pub(crate) fn rename_noreplace_between_dirs( + control: &File, + catalog: &Path, + source: &Path, + target: &Path, +) -> std::io::Result<()> { + let target_parent = open_dir_beneath( + catalog, + target.parent().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no parent") + })?, + )?; + let source_name = source.file_name().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "source has no name") + })?; + let target_name = target.file_name().ok_or_else(|| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target has no name") + })?; + renameat_noreplace(control, source_name, &target_parent, target_name) +} + +pub(super) fn renameat_noreplace( + source_parent: &File, + source: &std::ffi::OsStr, + target_parent: &File, + target: &std::ffi::OsStr, +) -> std::io::Result<()> { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt as _; + + let source = CString::new(source.as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "source name contains NUL") + })?; + let target = CString::new(target.as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target name contains NUL") + })?; + #[cfg(any(target_os = "linux", target_os = "android"))] + let result = unsafe { + libc::renameat2( + source_parent.as_raw_fd(), + source.as_ptr(), + target_parent.as_raw_fd(), + target.as_ptr(), + libc::RENAME_NOREPLACE, + ) + }; + #[cfg(target_os = "macos")] + let result = unsafe { + libc::renameatx_np( + source_parent.as_raw_fd(), + source.as_ptr(), + target_parent.as_raw_fd(), + target.as_ptr(), + libc::RENAME_EXCL, + ) + }; + #[cfg(not(any(target_os = "linux", target_os = "android", target_os = "macos")))] + let result = { + let _ = (source_parent, source, target_parent, target); + return Err(std::io::Error::new( + std::io::ErrorKind::Unsupported, + "atomic no-replace directory rename is unsupported on this platform", + )); + }; + if result == 0 { + Ok(()) + } else { + Err(std::io::Error::last_os_error()) + } +} + +pub(super) fn renameat( + source_parent: &File, + source: &std::ffi::OsStr, + target_parent: &File, + target: &std::ffi::OsStr, +) -> std::io::Result<()> { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt as _; + + let source = CString::new(source.as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "source name contains NUL") + })?; + let target = CString::new(target.as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target name contains NUL") + })?; + let result = unsafe { + libc::renameat( + source_parent.as_raw_fd(), + source.as_ptr(), + target_parent.as_raw_fd(), + target.as_ptr(), + ) + }; + if result == 0 { + Ok(()) + } else { + Err(std::io::Error::last_os_error()) + } +} + +pub(crate) fn rename_noreplace(source: &Path, target: &Path) -> std::io::Result<()> { + use std::ffi::CString; + use std::os::unix::ffi::OsStrExt as _; + + let source = CString::new(source.as_os_str().as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "source contains NUL") + })?; + let target = CString::new(target.as_os_str().as_bytes()).map_err(|_| { + std::io::Error::new(std::io::ErrorKind::InvalidInput, "target contains NUL") + })?; + #[cfg(any(target_os = "linux", target_os = "android"))] + let result = unsafe { + libc::renameat2( + libc::AT_FDCWD, + source.as_ptr(), + libc::AT_FDCWD, + target.as_ptr(), + libc::RENAME_NOREPLACE, + ) + }; + #[cfg(target_os = "macos")] + let result = unsafe { + libc::renameatx_np( + libc::AT_FDCWD, + source.as_ptr(), + libc::AT_FDCWD, + target.as_ptr(), + libc::RENAME_EXCL, + ) + }; + #[cfg(not(any(target_os = "linux", target_os = "android", target_os = "macos")))] + let result = { + let _ = (source, target); + return Err(std::io::Error::new( + std::io::ErrorKind::Unsupported, + "atomic no-replace directory rename is unsupported on this platform", + )); + }; + if result == 0 { + Ok(()) + } else { + Err(std::io::Error::last_os_error()) + } +} diff --git a/src/cli.rs b/src/cli.rs new file mode 100644 index 00000000..1f3978e6 --- /dev/null +++ b/src/cli.rs @@ -0,0 +1,1250 @@ +//! The `st2` command tree: every clap declaration and nothing else. +//! +//! Split out of `main.rs` verbatim so the verb implementations and the declaration of the +//! surface they implement can be read separately. Items are `pub(crate)` because this is a +//! binary crate: nothing here is part of any public API. + +use std::path::PathBuf; + +use anyhow::{Context, Result}; +use clap::{Args, Parser, Subcommand}; + +#[derive(Parser)] +#[command( + name = "st2", + version = st2::version::display_version(), + about = "Harness-agnostic runner over a unified catalog+inbox folder" +)] +pub(crate) struct Cli { + /// Catalog (or single-file fleet spec) to use. Defaults to $CATALOG, then + /// ${XDG_STATE_HOME:-$HOME/.local/state}/st2/default/catalog. + #[arg(long = "catalog", global = true, value_name = "PATH")] + pub(crate) catalog_path: Option, + #[command(subcommand)] + pub(crate) command: Command, +} + +#[derive(Subcommand)] +pub(crate) enum Command { + /// Discover and print every agent spec under a catalog+inbox folder. + Ls { + /// Legacy positional catalog/spec path. Prefer --catalog; defaults to $CATALOG, then the + /// default st2 catalog. + #[arg(conflicts_with = "catalog_path")] + root: Option, + }, + /// Supervise a catalog+inbox folder: reconcile on a folder-watch + timer, keeping each agent's + /// ptys running. With --once, do a single pass and exit. + Up { + /// Legacy positional catalog/spec path. Prefer --catalog; defaults to $CATALOG, then the + /// default st2 catalog. + #[arg(conflicts_with = "catalog_path")] + root: Option, + /// Host to filter on (which agents this machine runs). Defaults to the local hostname. + #[arg(long)] + host: Option, + /// Do a single reconcile pass and exit, instead of looping. + #[arg(long)] + once: bool, + /// Materialize every local agent's render block and exit without reconciling or spawning. + #[arg(long, conflicts_with = "once")] + materialize_only: bool, + /// Limit materialization to one declared agent identity. + #[arg(long)] + agent: Option, + /// Select one exact local task. Use with --materialize-only to render only its owner, or + /// with --once to render its owner and reconcile only that task. + #[arg(long, conflicts_with = "agent")] + task: Option, + /// Seconds between timer-driven reconcile passes when looping (folder changes reconcile + /// immediately regardless). + #[arg(long, default_value_t = 30)] + interval: u64, + }, + /// Native message bus: send/list/read/archive/reply over agents' `resources/inbox`. + /// The stable wire format is a `-.md` Markdown file. + #[command(subcommand)] + Message(MessageCmd), + /// Declared event streams: durable, bounded, idempotent ingress into an agent inbox. + #[command(subcommand)] + Event(EventCmd), + /// Self-author declared event streams through the serialized catalog path. + #[command(subcommand)] + Stream(StreamCmd), + /// Idempotent JSON request/reply transport for declared non-agent service principals. + #[command(subcommand)] + Request(RequestCmd), + /// An agent's working-state context for lossless restart: read/write/append. + #[command(subcommand)] + Context(ContextCmd), + /// An agent's declared Resource bindings (a named, exact URI a peer can resolve): + /// ls/read/add/remove/rename. + #[command(subcommand)] + Resource(ResourceCmd), + /// Install `st2 up` as a systemd-user service on headless Linux. macOS stays manual (TCC). + /// Subcommands: install / status / uninstall. + #[command(subcommand)] + Service(ServiceCmd), + /// Install and approve the embedded Claude Code channel plugin. + #[command(subcommand)] + ClaudeChannel(ClaudeChannelCmd), + /// Explicit lifecycle-hook management. `up` and materialization only verify; they never install + /// or refresh hooks. + #[command(subcommand)] + Hooks(HooksCmd), + /// Provider-native harness drivers and read-only typed-block expansion. + #[command(subcommand)] + Driver(DriverCmd), + /// The ding sidecar: watch an agent's `resources/inbox` and poke its pty (`[DING] …`) on each new + /// message. Busy does not suppress delivery; only fresh dnd defers FIFO. A startup backlog is + /// coalesced into one recovery notice. Long-running — st2 keeps it alive as a task alongside the + /// agent. Exits when the target pty session is gone. + /// `st2 ping` is an alias (the maintainer is renaming ding → ping, since dinging is the runner's + /// job now); it is the exact same command. + #[command(visible_alias = "ping")] + Ding { + /// The target pty session to poke (a `pty` session ref). Optional — defaults to `--identity` + /// (an agent IS its pty, so the session to poke is the identity), so `st2 ding --identity X` + /// is the common form. + session: Option, + /// Whose inbox to watch — bus id or identity. Defaults to `$ST_AGENT`. Also the default poke + /// target when no positional session is given. + #[arg(long)] + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--identity`. + #[arg(long = "agent-id", conflicts_with = "identity")] + agent_id: Option, + /// Catalog root. Defaults to `$CATALOG`. + #[arg(long, conflicts_with = "catalog_path")] + root: Option, + /// Host used to resolve `.` bus ids. Defaults to the local hostname. + #[arg(long)] + host: Option, + /// Poll/liveness cadence in milliseconds (folder changes poke immediately regardless). + #[arg(long, default_value_t = 1000)] + interval: u64, + }, + /// Internal controlled Codex launch. Generated only for `deliver "app-server"` tasks. + #[command(hide = true)] + CodexAppServer { + /// Exact agent bus identity that owns the controlled thread. + #[arg(long)] + identity: String, + /// Exact reconciled PTY task identity for this runtime. + #[arg(long)] + runtime_id: String, + /// Original structured Codex invocation, including its provider executable. + #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] + codex_argv: Vec, + }, + /// Internal Claude MCP channel server started by Claude from its rendered project declaration. + #[command(hide = true)] + ClaudeMcp { + #[arg(long)] + identity: String, + }, + /// Get or set an agent's presence status. No `--set` prints the status; no identity means yours + /// (`$ST_AGENT`). Settable: offline | available | busy | away | dnd (`unknown` is derived). + Status { + /// Whose status — bus id or identity. Defaults to you (`--as` / `$ST_AGENT`). + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// Set your status to this state instead of printing it. + #[arg(long = "set")] + set: Option, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Set or clear an agent's human-facing name without changing stable identity. + Rename(PresentationArgs), + /// Set or clear an agent's enduring responsibility description. + Describe(PresentationArgs), + /// Transactionally publish one canonical Agent Spec into the live catalog. + #[command(subcommand)] + Agent(AgentCmd), + /// Canonical declaration snapshots and crash-recoverable whole-catalog application. + #[command(subcommand)] + Catalog(CatalogCmd), + /// Explicit teardown: kill every live task of this host's catalog agents. The ONLY thing that ends + /// tasks (stopping/crashing st2 never does). Idempotent. + Down { + /// Optional positional catalog/spec path. Prefer --catalog; defaults to $CATALOG, then the + /// standard st2 catalog. + #[arg(conflicts_with = "catalog_path")] + root: Option, + /// Host to tear down. Defaults to the local hostname. + #[arg(long)] + host: Option, + }, + /// Print shell exports for a catalog's bus — `eval "$(st2 env --catalog )"` sets `CATALOG`/ + /// `ST_ROOT`/`PTY_ROOT` so native bus-aware tools target the catalog. + Env { + /// Optional positional catalog path. Prefer --catalog; defaults to $CATALOG, then the + /// standard st2 catalog. + #[arg(conflicts_with = "catalog_path")] + root: Option, + }, + /// Explicitly pre-trust workspaces in the ambient Claude and Codex configs. This is an operator + /// utility for harnesses that use those ambient configs; `st2 up` never calls it automatically. + /// Account-selecting commands should instead declare trust in the selected harness invocation. + Pretrust { + /// Workspace directories to mark trusted. + #[arg(required = true)] + dirs: Vec, + }, + /// Run an st2-spec eval end to end: copy the fixture, boot the team + judges, deliver the + /// kickoff, wait for the sup's confirmation, run the judges → verdict. `st2 eval ./cells//`. + Eval { + /// The eval folder (or its `.kdl` spec file). + folder: PathBuf, + /// Host. Defaults to the local hostname. + #[arg(long)] + host: Option, + /// Preserve the run's temp catalog instead of deleting it — for inspecting the worker repo + /// (`base..HEAD`), the judge outputs, and the bus after the run (e.g. a gate reproduction). + /// Seats are still torn down (no leaks). Also honored via `ST2_EVAL_KEEP`. + #[arg(long)] + keep: bool, + /// Emit the existing eval report as JSON without changing exit semantics. + #[arg(long)] + json: bool, + }, + /// Run `pty` against this catalog's bus with the env auto-set, so pty subcommands and the + /// interactive UI work without `eval "$(st2 env --catalog )"` first. Catalog selection follows + /// `--catalog`, `$CATALOG`, then the default st2 catalog. `CATALOG`/`ST_ROOT`/`PTY_ROOT` are + /// exported for the child exactly as `st2 env` would. No arguments launches the interactive pty + /// UI. + Pty { + /// Arguments passed through to `pty` verbatim (e.g. `ls`, `peek `). None → the UI. + #[arg(trailing_var_arg = true, allow_hyphen_values = true)] + args: Vec, + }, + /// Drop into `$SHELL` with this catalog's bus env set (`CATALOG`/`ST_ROOT`/`PTY_ROOT`), so `pty`, + /// bus-aware tools target the catalog for the whole session without `eval "$(st2 env …)"`. + /// The general form of `st2 pty`. Catalog selection follows `--catalog`, `$CATALOG`, then the + /// default st2 catalog; extra args go to the shell (e.g. `st2 shell -c "pty ls"`). + Shell { + /// Arguments passed through to `$SHELL` verbatim. None → an interactive shell. + #[arg(trailing_var_arg = true, allow_hyphen_values = true)] + args: Vec, + }, + /// Validate a rendered catalog against the runner contract (spec fields, folder layout, paths) + /// so any renderer can confirm it hit the spec before running. One line per issue with + /// a stable code; exits non-zero on any ERROR (`--strict` also fails on warnings). `--json` emits + /// machine output for a renderer's build gate. + Validate { + /// Legacy positional catalog path. Prefer --catalog; defaults to $CATALOG, then the default + /// st2 catalog. + #[arg(conflicts_with = "catalog_path")] + root: Option, + /// Host whose external workspace/task paths should be checked. Structural checks always + /// cover the whole catalog. Defaults to the local hostname. + #[arg(long)] + host: Option, + /// Validate this one unpublished canonical Agent Spec as an overlay on the live catalog. + #[arg(long, value_name = "FILE")] + candidate: Option, + /// Fail (non-zero exit) on warnings too, not just errors. + #[arg(long)] + strict: bool, + /// Emit the report as JSON instead of human-readable lines. + #[arg(long)] + json: bool, + }, + /// Health check for a catalog: active agents alive, suspended agents not live, and retired + /// agents fully absent. Exits non-zero on problems. + Doctor { + /// Legacy positional catalog path. Prefer --catalog; defaults to $CATALOG, then the default + /// st2 catalog. + #[arg(conflicts_with = "catalog_path")] + root: Option, + /// Host to check. Defaults to the local hostname. + #[arg(long)] + host: Option, + /// Require a live long-running `st2 up` host lock. Omit for manual/--once operation. + #[arg(long)] + require_supervisor: bool, + }, + /// List every agent in the catalog with presence and retirement state. `--json [--enrich]` is + /// the stable machine-readable roster. + Agents { + /// The catalog folder (like `st2 ls`/`up`). Falls back to `--root`/`$CATALOG`. + #[arg(conflicts_with = "catalog_path")] + catalog: Option, + /// Only agents whose effective status matches (offline|available|busy|away|dnd|unknown). + #[arg(long = "status")] + status: Option, + /// Select one exact Agent Spec by its fully qualified `.`. + #[arg(long, value_name = "HOST.IDENTITY")] + identity: Option, + /// Select one exact subject by its immutable agent ID (R24). + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// Machine-readable JSON array, including retirement and declared Resource bindings. + #[arg(long)] + json: bool, + /// With `--json`, add `lastActivity` + `inbox` count per agent. + #[arg(long)] + enrich: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Emit one fail-closed desired-task/runtime diagnostic snapshot. This is + /// read-only observation, not reconciliation or cutover authority. + Tasks { + /// Host whose desired tasks and runtime generations to inspect. Defaults to this host. + #[arg(long)] + host: Option, + /// Emit the versioned machine-readable envelope. Required in v1. + #[arg(long)] + json: bool, + }, + /// Clear one task's park after fixing what crash-looped it. A task parked by its `restart{}` + /// policy (mode=fail) stays parked for the rest of the supervisor run, and this is its per-task + /// exit: the running supervisor relaunches exactly this task on its next pass, leaving every + /// other task on the host untouched. `st2 tasks --json` reports which tasks are parked. + Unpark { + /// The parked task's runtime id, exactly as `st2 tasks --json` reports it. + task: String, + /// Host whose selected-catalog supervisor should grant the request. Defaults to this host. + #[arg(long)] + host: Option, + }, + /// Print a shell completion script for `st2` to stdout (`st2 completions `). + /// Generated from the live command tree, so it never drifts from the actual flags. + Completions { + /// The shell to generate completions for. + shell: clap_complete::Shell, + }, +} + +#[derive(Subcommand)] +pub(crate) enum DriverCmd { + /// Print one typed driver block as plain Agent Spec KDL without running it. + Expand { + /// KDL declaration that contains the typed driver block. + spec: PathBuf, + /// Select one local or fully qualified identity when the file contains multiple agents. + #[arg(long)] + agent: Option, + /// Host fallback when neither the declaration nor its catalog path supplies one. + #[arg(long)] + host: Option, + }, + /// Run the existing controlled Codex app-server path. + Codex { + #[arg(long)] + identity: String, + #[arg(long)] + runtime_id: String, + #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] + argv: Vec, + }, + /// Run the Claude session-owned MCP server over stdio. + ClaudeMcp { + #[arg(long)] + identity: Option, + }, + /// Deprecated name for the Claude MCP server. + // Keep this hidden command until no rendered configuration uses the old name. + #[command(hide = true)] + Claude { + #[arg(long)] + identity: String, + }, + /// Run Claude under the session-owned presence wrapper. + ClaudeSession { + #[arg(long)] + identity: String, + #[arg(long)] + runtime_id: String, + #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] + argv: Vec, + }, + /// Apply one Claude hook event (payload on stdin) to observed harness state. + ClaudeObserve { + #[arg(long)] + identity: String, + /// The wrapper's runtime/task ID; the record's pty session. Defaults to the identity. + #[arg(long)] + runtime_id: Option, + /// The Claude hook event name, e.g. `Stop` or `PermissionRequest`. + #[arg(long)] + event: String, + }, + /// Tee Claude's status-line payload (stdin JSON) into harness context, then chain to the + /// operator's own renderer. + ClaudeStatusline { + #[arg(long)] + identity: String, + }, + /// Run pi under the session-owned presence wrapper. + PiSession { + #[arg(long)] + identity: String, + #[arg(long)] + runtime_id: String, + #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] + argv: Vec, + }, + /// Run the pi native message channel over stdio, owned by the shipped pi extension. + PiChannel { + #[arg(long)] + identity: String, + }, + /// Run omp under the session-owned presence wrapper with a hard version gate. + OmpSession { + #[arg(long)] + identity: String, + #[arg(long)] + runtime_id: String, + #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] + argv: Vec, + }, + /// Run the omp native message channel over stdio, owned by the shipped omp extension. + OmpChannel { + #[arg(long)] + identity: String, + }, + /// Run OpenCode under the session-owned wrapper: presence, observed harness state, and native + /// server delivery over the wrapper-allocated local port. + OpencodeSession { + #[arg(long)] + identity: String, + #[arg(long)] + runtime_id: String, + #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] + argv: Vec, + }, +} + +#[derive(Subcommand)] +pub(crate) enum AgentCmd { + /// Author reversible whole-agent lifecycle intent in one canonical KDL declaration. + DesiredState { + /// Exact bus identity, or a bare stable identity only when unique — or the desired state + /// when `--id` names the subject. Authoring selects the declaration, never the address. + #[arg(value_name = "IDENTITY_OR_STATE")] + first: Option, + /// The desired state, when the first positional is the agent reference. + #[arg(value_name = "STATE")] + second: Option, + /// Exact immutable agent ID (R24). The first positional is then the desired state. + /// + /// Both positionals stay optional so the exact-ID form can shift them; clap refuses a + /// non-required positional ahead of a required one, which is why the state is validated in + /// the handler rather than by a positional `value_parser`. + #[arg(long = "id", conflicts_with = "second")] + agent_id: Option, + /// Required rationale for suspended/retired; forbidden for running. + #[arg(long)] + reason: Option, + /// Assert the ownership marker that owns this declaration, e.g. `nix`. + /// + /// A declaration carrying `meta { managed-by "nix" }` refuses ordinary authoring, + /// because the Nix projection is the writer of those bytes. This is how that + /// projection authors lifecycle on its own declaration — the transition it cannot + /// express in its own source, because the source change being projected is the + /// seat's removal. The assertion is admitted only when it names exactly the one + /// marker the declaration carries. + #[arg(long = "managed-by", value_name = "MARKER")] + managed_by: Option, + /// Host used only to resolve declarations whose host is omitted. + #[arg(long)] + host: Option, + /// Emit a stable JSON authoring receipt. + #[arg(long)] + json: bool, + }, + /// Assign or clear an agent's mutable address — one atomic address-book cutover with no + /// alias, redirect, or rename history. `--clear` restores the positional identity fallback. + Address(PresentationArgs), + /// Compute the authoritative digest bound by `agent publish --input-sha256`. + Digest { + /// A canonical KDL file containing exactly one top-level `agent` node. + #[arg( + long, + value_name = "FILE", + required_unless_present = "bundle", + conflicts_with = "bundle" + )] + spec: Option, + /// A create-only directory whose root contains exactly one canonical `agent.kdl`. + #[arg( + long, + value_name = "DIR", + required_unless_present = "spec", + conflicts_with = "spec" + )] + bundle: Option, + /// Emit the typed source-digest receipt as JSON. + #[arg(long)] + json: bool, + }, + /// Publish exactly one explicit-host, explicit-identity agent under a catalog-wide CAS lock. + Publish { + /// A canonical KDL file containing exactly one top-level `agent` node. + #[arg( + long, + value_name = "FILE", + required_unless_present = "bundle", + conflicts_with = "bundle" + )] + spec: Option, + /// A create-only directory whose root contains exactly one canonical `agent.kdl`. + #[arg( + long, + value_name = "DIR", + required_unless_present = "spec", + conflicts_with = "spec" + )] + bundle: Option, + /// Create only. An identical existing agent.kdl is reported as `unchanged`. + #[arg( + long, + required_unless_present = "expect_sha256", + conflicts_with = "expect_sha256" + )] + expect_absent: bool, + /// Replace only when the current agent.kdl has this lowercase SHA-256. + #[arg( + long, + value_name = "HEX", + required_unless_present = "expect_absent", + conflicts_with = "expect_absent" + )] + expect_sha256: Option, + /// SHA-256 returned by `st2 agent digest` for the exact source capability. + #[arg(long, value_name = "HEX")] + input_sha256: String, + /// Assert the ownership marker that owns the declaration being replaced, e.g. `nix`. + /// + /// A declaration carrying `meta { managed-by "nix" }` refuses an unasserted + /// replacement, because the Nix projection is the writer of those bytes. The + /// assertion is admitted only when it names exactly the one marker the incumbent + /// carries. Create-only publication has no incumbent and needs no assertion. + #[arg(long = "managed-by", value_name = "MARKER")] + managed_by: Option, + /// Emit the typed publication result as JSON. + #[arg(long)] + json: bool, + }, +} + +#[derive(Subcommand)] +pub(crate) enum CatalogCmd { + /// Emit one fail-closed declaration graph plus runtime observation envelope. + Graph { + /// Host used to resolve declarations with no host and host-local runtime facts. + #[arg(long)] + host: Option, + /// Emit the versioned machine-readable envelope. Required in v1. + #[arg(long)] + json: bool, + }, + /// Compute the authoritative digest bound by `catalog apply --input-sha256`. + Digest { + /// Complete prepared declaration directory. Runtime state and control paths are rejected. + #[arg(long, value_name = "DIR")] + prepared: PathBuf, + /// Emit the typed source-digest receipt as JSON. + #[arg(long)] + json: bool, + }, + /// Compare one prepared declaration directory with the coherent live catalog without writing. + Diff { + /// Complete prepared declaration directory. Runtime state and control paths are rejected. + #[arg(long, value_name = "DIR")] + prepared: PathBuf, + /// Expected canonical declaration-root SHA-256 of the live catalog. + #[arg(long, value_name = "HEX")] + expect_sha256: String, + /// Emit the versioned semantic-diff receipt. Required in v1. + #[arg(long)] + json: bool, + }, + /// Publish a complete prepared declaration directory as one absent catalog. + Bootstrap { + /// Complete prepared declaration directory. Runtime state and control paths are rejected. + #[arg(long, value_name = "DIR")] + prepared: PathBuf, + /// Root SHA-256 of the exact prepared projection being published. + #[arg(long, value_name = "HEX")] + input_sha256: String, + /// Emit the typed bootstrap receipt as JSON. + #[arg(long)] + json: bool, + }, + /// Capture the coherent declaration plane into a create-only canonical directory. + Snapshot { + /// Destination directory. It must be outside the live catalog. + #[arg(long, value_name = "DIR")] + output: PathBuf, + /// Hash and capture the declaration plane without parsing it. The captured directory + /// remains unvalidated and is suitable only as an exact-byte CAS preimage. + #[arg(long)] + raw_preimage: bool, + /// Emit the typed snapshot receipt as JSON. + #[arg(long)] + json: bool, + }, + /// Apply a complete canonical declaration directory under declaration-root CAS. + Apply { + /// Complete prepared declaration directory. Runtime state and control paths are rejected. + #[arg( + long, + value_name = "DIR", + required_unless_present = "resume", + conflicts_with = "resume" + )] + prepared: Option, + /// Root SHA-256 of the exact prepared projection being applied. + #[arg( + long, + value_name = "HEX", + required_unless_present = "resume", + conflicts_with = "resume" + )] + input_sha256: Option, + /// Expected canonical declaration-root SHA-256 of the live catalog. + #[arg( + long, + value_name = "HEX", + required_unless_present = "resume", + conflicts_with = "resume" + )] + expect_sha256: Option, + /// Match the current declaration plane without parsing it. The prepared catalog is still + /// fully validated; use this mode only when the current parser cannot admit the preimage. + #[arg(long, conflicts_with = "resume")] + raw_preimage: bool, + /// Resume the durable incomplete marker and internal stage without the original source. + #[arg(long, conflicts_with_all = ["prepared", "input_sha256", "expect_sha256"])] + resume: bool, + /// Emit the typed application receipt as JSON. + #[arg(long)] + json: bool, + }, + /// Move retired, runtime-free identities out of the live catalog into `.st2/archive`, leaving a + /// tombstone row in `st2 catalog graph --json`. An archived spec is not discoverable. + Archive { + /// Identity to archive, repeatable. Every named identity must be eligible or nothing moves. + #[arg( + long, + value_name = "IDENTITY", + required_unless_present = "all_retired", + conflicts_with = "all_retired" + )] + identity: Vec, + /// Archive every eligible retired identity of the selected host. Ineligible ones are + /// reported and skipped. + #[arg(long)] + all_retired: bool, + /// Host whose identities are archived. Defaults to this host; another host's runtime + /// records are not observable from here, so only the local host is eligible. + #[arg(long)] + host: Option, + /// Decide eligibility and print the plan without moving anything. + #[arg(long)] + dry_run: bool, + /// Emit the typed archive receipt as JSON. + #[arg(long)] + json: bool, + }, + /// Move one archived identity back into the live catalog. The exact reverse of `archive`. + Unarchive { + /// Archived identity to restore. + identity: String, + /// Host the identity was archived under. Defaults to this host. + #[arg(long)] + host: Option, + /// Emit the typed restoration receipt as JSON. + #[arg(long)] + json: bool, + }, +} + +/// Shared context for message subcommands: where the catalog is, who "I" am, and the local host. +/// Defaults come from the same env st2 sets on every task it spawns (`$CATALOG`, `$ST_AGENT`), so a +/// running agent needs no flags. +#[derive(Args)] +pub(crate) struct MsgCtx { + /// Legacy catalog/bus root override. Prefer global `--catalog`; defaults to `$CATALOG`, then the + /// default st2 catalog. + #[arg(long, conflicts_with = "catalog_path")] + pub(crate) root: Option, + /// The acting identity — who the message is `from` / whose inbox is "mine". An ordinary + /// address reference, unlike `$ST_AGENT`, which carries the exact agent ID; the two are + /// different strings once a subject declares an explicit `address`. Defaults to `$ST_AGENT`. + #[arg(long = "as")] + pub(crate) as_id: Option, + /// Host used to resolve `.` bus ids. Defaults to the local hostname. + #[arg(long)] + pub(crate) host: Option, +} + +/// `[] ` plus the mutually exclusive exact-ID form. +/// +/// `--id` takes the agent off the positional list, so the first positional is then the value — +/// the same `[identity] ` convention `st2 message read` and `st2 resource read` already +/// use, and the reason clap's exclusion is expressed against the second positional. +#[derive(Args)] +pub(crate) struct PresentationArgs { + /// Exact bus identity, or a bare stable identity only when unique in the selected catalog — + /// or the new value when `--id` names the subject. Authoring selects the declaration, never + /// the address. + #[arg(value_name = "IDENTITY_OR_TEXT")] + pub(crate) first: Option, + /// The new value, when the first positional is the agent reference. + #[arg(value_name = "TEXT")] + pub(crate) second: Option, + /// Exact immutable agent ID (R24). The first positional is then the new value. + #[arg(long = "id", conflicts_with = "second")] + pub(crate) agent_id: Option, + /// Remove the optional field. + #[arg(long)] + pub(crate) clear: bool, + /// Emit a stable JSON receipt or classified refusal. + #[arg(long)] + pub(crate) json: bool, + /// Host used only to resolve declarations whose host is omitted. + #[arg(long)] + pub(crate) host: Option, +} + +impl PresentationArgs { + /// The selected subject and the requested value, where `None` is the cleared representation. + pub(crate) fn selection(self) -> Result<(st2::identity::AgentSelector, Option, bool, Option)> + { + let (selector, value) = match self.agent_id { + Some(id) => (st2::identity::AgentSelector::Id(id), self.first), + None => ( + st2::identity::AgentSelector::Address( + self.first + .context("no agent selected: pass an agent reference or the `--id` form")?, + ), + self.second, + ), + }; + anyhow::ensure!( + !(self.clear && value.is_some()), + "--clear removes the field and takes no value" + ); + anyhow::ensure!( + self.clear || value.is_some(), + "a value is required unless --clear" + ); + Ok(( + selector, + if self.clear { None } else { value }, + self.json, + self.host, + )) + } +} + +#[derive(Subcommand)] +pub(crate) enum ServiceCmd { + /// Write the `st2.service` systemd-user unit, enable it (start on boot), and start it now. + /// Idempotent — safe to re-run. The unit runs `st2 up --catalog `; agents spawn in sibling + /// scopes, so a service restart never cascades to them. + Install { + /// Legacy positional catalog/spec path for `st2 up`. Prefer --catalog; defaults to + /// `$CATALOG`, then the default st2 catalog. It must exist at install time. + #[arg(conflicts_with = "catalog_path")] + catalog: Option, + /// Bake `--host ` into the unit. Omit to let `st2 up` auto-detect the hostname at runtime. + #[arg(long)] + host: Option, + /// Machine-local pty registry to export as PTY_ROOT in the unit. Omit to use + /// `/pty`. Useful when adopting live sessions from a legacy runner. + #[arg(long)] + pty_root: Option, + /// Supervisor memory ceiling (MiB). The agents live in sibling scopes and are NOT bounded. + #[arg(long = "memory-max-mb", default_value_t = st2::service::DEFAULT_MEMORY_MAX_MB)] + memory_max_mb: u64, + }, + /// Show the `st2.service` systemd status. + Status, + /// Stop, disable, and remove the `st2.service` unit. Idempotent. + Uninstall, +} + +#[derive(Subcommand)] +pub(crate) enum ClaudeChannelCmd { + /// Install or update the user plugin and its machine approval policy. + Install { + /// Install only the user plugin. An administrator will manage the machine policy. + #[arg(long)] + no_policy: bool, + }, + /// Verify the embedded files, Claude registration, plugin, and machine policy. + Status, + /// Remove the user plugin, marketplace, embedded files, and machine policy. + Uninstall { + /// Keep the machine approval policy in place. + #[arg(long)] + keep_policy: bool, + }, + /// Write only the machine policy. The main installer runs this through sudo. + #[command(hide = true)] + InstallPolicy, + /// Remove only the st2-owned machine policy fragment. + #[command(hide = true)] + UninstallPolicy, +} + +#[derive(Subcommand)] +pub(crate) enum HooksCmd { + /// Atomically publish this binary's immutable hook set and select it with a receipt. + Install { + /// Select this binary's exact hook set even when it is older or cannot be ordered. + #[arg(long)] + replace: bool, + /// Deprecated compatibility alias for `--replace`. + #[arg(long, hide = true)] + allow_downgrade: bool, + }, + /// Read-only verification of the selected receipt and every embedded hook byte. + Verify, + /// Verify this binary's immutable hook set without requiring it to be selected. + VerifyOwn, +} + +#[derive(Subcommand)] +pub(crate) enum ResourceCmd { + /// List an agent's declared Resource bindings. Defaults to your own. + Ls { + /// Whose declaration to read — bus id or bare identity. Defaults to you (`$ST_AGENT`). + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// Emit the bindings as a JSON array. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Read one declared binding. With a leading identity, from that agent; otherwise your own. + Read { + first: String, + second: Option, + /// Exact immutable agent ID (R24); `first` is then the binding name. + #[arg(long = "id", conflicts_with = "second")] + agent_id: Option, + /// Emit the binding as a JSON object. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Ask the resident profile runtime to observe one binding now and wait for exact evidence. + Refresh { + /// Binding name, or an agent selector when followed by a binding name. + first: String, + /// Binding name when the first positional selects the agent. + second: Option, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long, conflicts_with = "second")] + agent: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. + #[arg(long = "agent-id", conflicts_with_all = ["second", "agent"])] + agent_id: Option, + /// Client-only wait bound in seconds. Expiry never cancels or retracts queued demand. + #[arg(long, default_value_t = 30)] + wait: u64, + /// Emit the stable receipt (or timeout envelope) as JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Declare a Resource binding, or prove the identical binding already exists. + Add { + /// The agent-local binding name. + name: String, + /// The exact absolute URI this binding names (any `scheme:` — the identity is verbatim). + #[arg(long)] + uri: String, + /// Why this reference belongs in the declaration. + #[arg(long)] + reason: String, + /// Preserve the binding as no longer active for this agent, and say why. + #[arg(long = "inactive-reason", value_name = "TEXT")] + inactive_reason: Option, + /// Profile-specific observation selector as JSON. + #[arg(long = "selector-json", value_name = "JSON")] + selector_json: Option, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long)] + agent: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. + #[arg(long = "agent-id", conflicts_with = "agent")] + agent_id: Option, + /// Emit a stable JSON receipt. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Remove one declared binding, or prove it is already absent. + Remove { + /// The agent-local binding name. + name: String, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long)] + agent: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. + #[arg(long = "agent-id", conflicts_with = "agent")] + agent_id: Option, + /// Emit a stable JSON receipt. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Rename one declared binding's agent-local label, keeping its uri and reasons. + Rename { + /// The current binding name. + old: String, + /// The new binding name. + new: String, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long)] + agent: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. + #[arg(long = "agent-id", conflicts_with = "agent")] + agent_id: Option, + /// Emit a stable JSON receipt. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, +} + +#[derive(Subcommand)] +pub(crate) enum ContextCmd { + /// Print an agent's context. Default = `now.md` (working state); `--decisions` the log; `--full` both. + Read { + /// Whose context — bus id or identity. Defaults to you (`$ST_AGENT`). + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// Print the decision log instead of the working state. + #[arg(long)] + decisions: bool, + /// Print the working state and the decision log. + #[arg(long)] + full: bool, + /// Print `now.md` only when it is newer than this many seconds. + #[arg(long, value_name = "SECONDS")] + fresh_within: Option, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Overwrite an agent's working state (`now.md`) from stdin. + Write { + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Append a single decision (with its reasoning) to the log. + Append { + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// The decision — a single line. + #[arg(long)] + decision: String, + /// Why — a single line. + #[arg(long)] + why: String, + #[command(flatten)] + ctx: MsgCtx, + }, +} + +#[derive(Subcommand)] +pub(crate) enum MessageCmd { + /// Send a new message to a recipient's inbox. + Send { + /// Recipient: a bus address (`.
`) or a bare address in the catalog. + #[arg(required_unless_present = "to_id", conflicts_with = "to_id")] + to: Option, + /// Recipient by exact immutable agent ID (R24). Mutually exclusive with the positional. + #[arg(long = "to-id")] + to_id: Option, + /// The message body. Read from stdin when omitted. + #[arg(short = 'm', long = "message")] + body: Option, + #[arg(long)] + subject: Option, + #[arg(long = "in-reply-to")] + in_reply_to: Option, + /// Comma-separated tags. + #[arg(long, value_delimiter = ',')] + tags: Vec, + /// Reuse one sender-owned operation result across exact retries. + #[arg(long = "idempotency-key")] + idempotency_key: Option, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Reply to a message in your inbox — recipient and threading are derived from it. + Reply { + /// The message filename in your inbox to reply to. + filename: String, + /// The reply body. Read from stdin when omitted. + #[arg(short = 'm', long = "message")] + body: Option, + /// Override the subject (defaults to `re: `). + #[arg(long)] + subject: Option, + /// Reuse one sender-owned operation result across exact retries. + #[arg(long = "idempotency-key")] + idempotency_key: Option, + #[command(flatten)] + ctx: MsgCtx, + }, + /// List an inbox (or `--archive`), sorted by send time. Defaults to your own. + Ls { + /// Whose inbox — bus id or identity. Defaults to you (`--as` / `$ST_AGENT`). + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// List the archive instead of the inbox. + #[arg(long)] + archive: bool, + /// Recovery-only: list the raw flat `/` box without catalog resolution. + #[arg(long)] + orphan: bool, + /// Print only the message count. + #[arg(long)] + count: bool, + /// Include full message bodies in JSON output (opt-in; default shape is unchanged). + #[arg(long)] + include_body: bool, + /// Show only messages from this sender. + #[arg(long = "from")] + from: Option, + /// Show only messages sent after this unix-millisecond timestamp. + #[arg(long)] + since: Option, + /// Machine-readable JSON array. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// List sender-owned durable message history. Defaults to your own index. + Sent { + /// Whose sent index — bus id or identity. Defaults to you (`--as` / `$ST_AGENT`). + identity: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. + #[arg(long = "id", conflicts_with = "identity")] + agent_id: Option, + /// Print only the indexed message count. Refuses unavailable or partial coverage. + #[arg(long)] + count: bool, + /// Include full message bodies in JSON output. + #[arg(long)] + include_body: bool, + /// Show only messages addressed to this canonical recipient. + #[arg(long = "to")] + to: Option, + /// Show only messages sent after this unix-millisecond timestamp. + #[arg(long)] + since: Option, + /// Machine-readable coverage envelope and rows. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Read one message. With a leading identity, read from that agent's box; otherwise your own. + Read { + /// Either the message filename, or an identity followed by a filename. + first: String, + /// The message filename (when `first` is an identity). + second: Option, + /// Exact immutable agent ID (R24) owning the box; `first` is then the filename. + #[arg(long = "id", conflicts_with = "second")] + agent_id: Option, + /// Read from the archive instead of the inbox. + #[arg(long)] + archive: bool, + /// Print the file verbatim (frontmatter + body), not a formatted view. + #[arg(long)] + raw: bool, + /// Machine-readable JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Archive one message (inbox → archive). Defaults to your own inbox. + Archive { + /// Either the message filename, or an identity followed by a filename. + first: String, + /// The message filename (when `first` is an identity). + second: Option, + /// Exact immutable agent ID (R24) owning the box; `first` is then the filename. + #[arg(long = "id", conflicts_with = "second")] + agent_id: Option, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Show a message's thread — the message + everything replying to it (transitively), across the + /// catalog. `--tree` indents by reply depth; otherwise flat chronological. + Thread { + /// Either the message filename, or an identity followed by a filename. + first: String, + /// The message filename (when `first` is an identity). + second: Option, + /// Indented hierarchical output instead of flat chronological. + #[arg(long)] + tree: bool, + #[command(flatten)] + ctx: MsgCtx, + }, +} + +#[derive(Subcommand)] +pub(crate) enum EventCmd { + /// Emit one producer-identified event into a declared agent stream. + Emit { + /// Owning agent: a bus address (`.
`) or a bare local address. + #[arg(required_unless_present = "recipient_id", conflicts_with = "recipient_id")] + recipient: Option, + /// Owning agent by exact immutable agent ID (R24). + #[arg(long = "recipient-id")] + recipient_id: Option, + /// Declared stream name. + #[arg(long)] + stream: String, + /// Stable producer-supplied event identity. + #[arg(long = "event-id")] + event_id: String, + /// Producer grouping key used by --supersede. + #[arg(long)] + key: Option, + /// Archive the unread predecessor for the same key, or the stream-wide head without --key. + #[arg(long)] + supersede: bool, + /// One-line wake-time summary. + #[arg(long)] + subject: Option, + /// Event body. Read from stdin when omitted. + #[arg(short = 'm', long = "message")] + body: Option, + /// Emit the stable machine receipt. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, +} + +#[derive(Subcommand)] +pub(crate) enum StreamCmd { + /// Add a stream to your declaration, optionally with a supervised adapter launch. + Add { + name: String, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long)] + agent: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. + #[arg(long = "agent-id", conflicts_with = "agent")] + agent_id: Option, + /// Adapter command run under `sh -c`; omit both launch forms for external ingress. + #[arg(long, conflicts_with = "adapter_argv")] + command: Option, + /// Direct adapter argv after `--`. Element 0 is the program; values are preserved exactly. + #[arg(trailing_var_arg = true, allow_hyphen_values = true)] + adapter_argv: Vec, + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Remove a stream from your declaration. + Rm { + name: String, + /// Exact target agent; defaults to --as / $ST_AGENT. + #[arg(long)] + agent: Option, + /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. + #[arg(long = "agent-id", conflicts_with = "agent")] + agent_id: Option, + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, +} + +#[derive(Subcommand)] +pub(crate) enum RequestCmd { + /// Publish one idempotent JSON request from a declared service principal to an agent. + Send { + /// Recipient agent: a bus id (`.`) or a local bare identity. + to: String, + #[arg(long = "idempotency-key")] + idempotency_key: String, + /// Typed request tag as `key=value` (repeatable). + #[arg(long = "tag")] + tags: Vec, + /// JSON body. Read from stdin when omitted. + #[arg(short = 'm', long = "message")] + body: Option, + /// Emit the machine receipt as JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Decode one typed request from an agent's inbox. + Read { + request_filename: String, + /// Emit the request envelope as JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Reply once to a typed request in an agent's inbox. + Reply { + request_filename: String, + /// Typed reply tag as `key=value` (repeatable). + #[arg(long = "tag")] + tags: Vec, + /// JSON body. Read from stdin when omitted. + #[arg(short = 'm', long = "message")] + body: Option, + /// Emit the machine receipt as JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, + /// Observe the typed reply for one previously published request. + Status { + #[arg(long = "idempotency-key")] + idempotency_key: String, + /// Emit the tagged status union as JSON. + #[arg(long)] + json: bool, + #[command(flatten)] + ctx: MsgCtx, + }, +} diff --git a/src/codex_app_server.rs b/src/codex_app_server.rs index 5e1f217f..955fe5f5 100644 --- a/src/codex_app_server.rs +++ b/src/codex_app_server.rs @@ -9,7 +9,7 @@ //! inbox head and submits typed input only when that state proves an idle or one exact regular //! active turn. -use std::collections::{BTreeMap, BTreeSet, VecDeque}; +use std::collections::BTreeMap; use std::fs::{self, File, OpenOptions}; use std::io::{Read as _, Write}; use std::net::Shutdown; @@ -17,7 +17,6 @@ use std::os::unix::ffi::OsStrExt as _; use std::os::unix::fs::{FileTypeExt as _, OpenOptionsExt as _, PermissionsExt as _}; use std::os::unix::io::AsRawFd as _; use std::os::unix::net::UnixStream; -use std::os::unix::process::CommandExt as _; use std::os::unix::process::ExitStatusExt as _; use std::path::{Path, PathBuf}; use std::process::{Child, Command, ExitStatus, Stdio}; @@ -512,270 +511,8 @@ struct PendingCodexDelivery { /// from the unchanged literal. pub const CODEX_CONTEXT_VERIFIED_VERSION: &str = "0.151.0"; -/// Codex's `BASELINE_TOKENS`, subtracted from BOTH the numerator and the denominator of its -/// displayed occupancy: `codex-rs/protocol/src/protocol.rs:2332` and -/// `codex-rs/tui/src/token_usage.rs:9` at `rust-v0.151.0` carry the same literal with an identical -/// function body, and no configuration override exists. Its doc comment: "should capture tokens -/// that are always present in the context (e.g. system prompt and fixed tool instructions) so that -/// the percentage reflects the portion the user can influence." -const CODEX_BASELINE_TOKENS: i64 = 12_000; - -/// The seven-day rate-limit window, identified by its duration because -/// `account/rateLimits/updated` names its windows `primary`/`secondary` and nothing else. 10,080 -/// minutes = 7 days, and the one captured Codex rate-limit snapshot (rollout, 0.150.1) carries -/// exactly this window as `primary`. See [`CodexContextProducer::observe_rate_limits`] for why the -/// five-hour leg stays `null`. -const CODEX_SEVEN_DAY_WINDOW_MINUTES: i64 = 10_080; - -/// How many recent compaction identities the dedupe retains. One compaction reaches this observer -/// as both `item/started` and `item/completed` — and possibly also as the deprecated -/// `thread/compacted` — so counting the edge naively counts one compaction twice or three times. A -/// last-key-only memory would still miscount an interleaving (`started(A)`, `started(B)`, -/// `completed(A)`), which a small ring closes for the same cost. -const CODEX_COMPACTION_MEMORY: usize = 4; - -/// Codex's own occupancy arithmetic, mirrored rather than re-derived: the published number is -/// exactly `100 −` the "N% context left" the operator reads in the Codex footer. -/// -/// `codex-rs/tui/src/token_usage.rs:43` (and its protocol twin): -/// -/// ```text -/// if context_window <= BASELINE_TOKENS { return 0; } -/// effective = context_window - BASELINE_TOKENS -/// used = (last.total_tokens - BASELINE_TOKENS).max(0) -/// remaining = (effective - used).max(0) -/// ((remaining / effective) * 100).clamp(0,100).round() -/// ``` -/// -/// Three things this deliberately does NOT do: -/// -/// - It does not round the *used* percentage. Rounding `used/effective` and rounding -/// `remaining/effective` disagree on a half — effective 200, used 101 gives 51 one way and 50 -/// the other — and only the mirrored order satisfies the spec's "equals `100 −` Codex's -/// displayed '% context left'". -/// - It does not use `total`, which is cumulative session spend. Against the captured window a -/// `total`-based percent reads 100 where the true occupancy is 33. -/// - It does not use `last.inputTokens`, which gives ~36 against the same capture — close enough -/// to look right and wrong by construction. -/// -/// The one divergence from the source: where Codex returns `0` remaining for a window at or below -/// the baseline, mirroring blindly would publish "100% used" for a window it cannot normalize. st2 -/// withholds instead (HC-R02, HC-R03) — a saturation the harness never displayed is fabricated, -/// not observed. -/// -/// The result cannot exceed 100: Codex's `remaining` is floored at zero, so an occupancy above the -/// effective window saturates in the harness's own arithmetic before st2 ever sees it. That is a -/// property of mirroring Codex, not a clamp of st2's — the record still carries what a producer -/// computes, unclamped (HC-R02), and the harnesses that can report an overrun are the ones -/// publishing a float of their own. -fn codex_used_percent(window_tokens: Option, last_total_tokens: i64) -> Option { - let window = window_tokens?; - if window <= CODEX_BASELINE_TOKENS { - return None; - } - let effective = window - CODEX_BASELINE_TOKENS; - let used = (last_total_tokens - CODEX_BASELINE_TOKENS).max(0); - let remaining = (effective - used).max(0); - let remaining_percent = ((remaining as f64 / effective as f64) * 100.0) - .clamp(0.0, 100.0) - .round(); - Some(100.0 - remaining_percent) -} - -/// One compaction's identity as this observer can name it. The item events carry a stable item id -/// alongside the turn; the deprecated `thread/compacted` notification carries only the turn, so its -/// key collapses with any item key in the same turn rather than counting beside it. -#[derive(Debug, Clone, PartialEq, Eq)] -struct CodexCompactionKey { - turn_id: String, - item_id: Option, -} - -impl CodexCompactionKey { - /// Whether these two names describe the same compaction. Two distinct item ids in one turn are - /// two compactions; a turn-only name in a turn already counted is the same one under its other - /// spelling. - fn same_compaction(&self, other: &Self) -> bool { - self.turn_id == other.turn_id - && match (&self.item_id, &other.item_id) { - (Some(mine), Some(theirs)) => mine == theirs, - _ => true, - } - } -} - -/// The Codex half of the harness-context record (HC-R11). -/// -/// It owns a [`harness_context::Writer`] beside the harness-state writer, sharing the wrapper's -/// incarnation so both records name the same session as their provenance. It holds no guard of its -/// own: `thread/tokenUsage/updated` arrives once per model response — roughly 10–15 per turn, and -/// replayed to a newly attached connection on resume — and every one of them is handed to -/// [`harness_context::Writer::observe`], whose quantization is the only thing deciding what lands. -/// A second guard here would make the write policy per-harness, which HC-R09 exists to prevent. -/// -/// The only state it carries between notifications is what it cannot recover from the next one: -/// the account-scoped rate-limit windows (a separate notification with no reading behind it) and -/// the identities of recently counted compactions. -struct CodexContextProducer { - writer: harness_context::Writer, - /// Last-known account-scoped windows. `account/rateLimits/updated` is documented as a *sparse - /// rolling update* whose absent fields do not clear a previously observed value, so the last - /// known windows ride along with the next reading instead of blanking it. - rate_limits: harness_context::RateLimits, - counted_compactions: VecDeque, -} - -impl CodexContextProducer { - fn new(writer: harness_context::Writer) -> Self { - Self { - writer, - rate_limits: harness_context::RateLimits::default(), - counted_compactions: VecDeque::new(), - } - } - - /// Project one inbound control frame onto the context record, returning whether a write landed. - /// - /// Every unknown method, foreign thread, and malformed payload is ignored rather than failed: - /// this is observability riding a delivery socket, and a frame this producer cannot read must - /// not disturb the frame the delivery loop can. - fn observe(&mut self, message: &Value, thread_id: &str) -> Result { - let Some(method) = message.get("method").and_then(Value::as_str) else { - return Ok(false); - }; - match method { - "thread/tokenUsage/updated" => { - if message.pointer("/params/threadId").and_then(Value::as_str) != Some(thread_id) { - return Ok(false); - } - let Some(reading) = self.token_usage_reading(message) else { - return Ok(false); - }; - self.writer.observe(reading) - } - // Account-scoped and thread-free (HC-T06): it repeats across every runtime sharing the - // account, carries no occupancy, and therefore never writes on its own. It is held and - // published by the next reading. - "account/rateLimits/updated" => { - self.observe_rate_limits(message); - Ok(false) - } - "item/started" | "item/completed" => { - if message.pointer("/params/threadId").and_then(Value::as_str) != Some(thread_id) - || message.pointer("/params/item/type").and_then(Value::as_str) - != Some("contextCompaction") - { - return Ok(false); - } - let (Some(turn_id), Some(item_id)) = ( - message.pointer("/params/turnId").and_then(Value::as_str), - message.pointer("/params/item/id").and_then(Value::as_str), - ) else { - return Ok(false); - }; - self.compacted(CodexCompactionKey { - turn_id: turn_id.to_string(), - item_id: Some(item_id.to_string()), - }) - } - // Deprecated in the protocol in favour of the item ("Deprecated: Use - // `ContextCompaction` item type instead") and unobserved on 0.150.1. Handled anyway, - // and deduped against the item, because a harness emitting both must still count one - // compaction. - "thread/compacted" => { - if message.pointer("/params/threadId").and_then(Value::as_str) != Some(thread_id) { - return Ok(false); - } - let Some(turn_id) = message.pointer("/params/turnId").and_then(Value::as_str) - else { - return Ok(false); - }; - self.compacted(CodexCompactionKey { - turn_id: turn_id.to_string(), - item_id: None, - }) - } - _ => Ok(false), - } - } - - /// The reading a `thread/tokenUsage/updated` carries, in Codex's own arithmetic. - /// - /// `usedTokens` and `windowTokens` are the harness's raw operands and are published as they - /// arrive — a window at or below the baseline is still a window the harness reported, even - /// where it cannot produce a percent. `model` and `costUsd` are `null` because the channel - /// carries neither: the app-server `Thread` object has `modelProvider` and no model identifier, - /// and Codex reports no session cost anywhere in the protocol (HC-R16). - fn token_usage_reading(&self, message: &Value) -> Option { - let last_total = message - .pointer("/params/tokenUsage/last/totalTokens") - .and_then(Value::as_i64)?; - let window = message - .pointer("/params/tokenUsage/modelContextWindow") - .and_then(Value::as_i64) - .filter(|window| *window > 0); - Some(harness_context::Reading { - used_tokens: u64::try_from(last_total).ok(), - window_tokens: window.and_then(|window| u64::try_from(window).ok()), - used_percent: codex_used_percent(window, last_total), - model: None, - cost_usd: None, - // Cumulative lifetime spend and never occupancy (HC-R16): the captured session read - // 2,235,329 against a 258,400-token window. - session_total_tokens: message - .pointer("/params/tokenUsage/total/totalTokens") - .and_then(Value::as_i64) - .and_then(|total| u64::try_from(total).ok()), - rate_limits: self.rate_limits, - }) - } - - /// Merge a sparse rate-limit update into the last-known windows. - /// - /// Codex names its windows `primary` and `secondary` and identifies them only by - /// `windowDurationMins`, so the join is by duration. Only the seven-day window is carried: the - /// single captured Codex rate-limit snapshot (0.150.1) contains one window, `primary`, at - /// 10,080 minutes. No 300-minute window and no `secondary` was ever observed on this harness, - /// so mapping one onto `fiveHour` would be inference dressed as a measurement — and this - /// record's whole point is that its numbers were seen. `fiveHour` therefore stays `null` for - /// Codex until a capture shows the window; admitting it is a one-line change beside the - /// capture that justifies it. - fn observe_rate_limits(&mut self, message: &Value) { - for window in ["primary", "secondary"] { - let Some(snapshot) = message.pointer(&format!("/params/rateLimits/{window}")) else { - continue; - }; - if snapshot.get("windowDurationMins").and_then(Value::as_i64) - == Some(CODEX_SEVEN_DAY_WINDOW_MINUTES) - && let Some(used) = snapshot.get("usedPercent").and_then(Value::as_f64) - { - self.rate_limits.seven_day = Some(used); - } - } - } - - /// Count one compaction edge unless this compaction was already counted under another of its - /// spellings. The count is incarnation-scoped: Codex publishes an edge and nothing else, so st2 - /// does the counting and the relaunch claim's record removal resets it (HC-R12, HC-R15). The - /// trigger is `unknown` because `ContextCompactionThreadItem` carries `id` and `type` and no - /// reason at all. - fn compacted(&mut self, key: CodexCompactionKey) -> Result { - if self - .counted_compactions - .iter() - .any(|counted| counted.same_compaction(&key)) - { - return Ok(false); - } - self.counted_compactions.push_back(key); - while self.counted_compactions.len() > CODEX_COMPACTION_MEMORY { - self.counted_compactions.pop_front(); - } - self.writer.compacted(harness_context::Compaction::new( - harness_context::CompactionTrigger::Unknown, - )) - } -} +mod context; +use self::context::*; #[derive(Debug, Clone)] struct RejectedCodexDelivery { @@ -3162,698 +2899,8 @@ fn completed_tui(status: ExitStatus) -> Result<()> { Ok(()) } -struct CodexProtocolSchemas { - protocol: Value, - client_requests: Value, - client_notifications: Value, - server_requests: Value, - server_notifications: Value, -} - -/// Admit the installed Codex app-server protocol, returning the version that passed. -fn ensure_supported_protocol(codex: &str) -> Result { - let version = codex_version(codex)?; - let generated = tempfile::Builder::new() - .prefix("st2-codex-protocol-") - .tempdir() - .context("creating a temporary Codex protocol schema directory")?; - let output = Command::new(codex) - .args([ - "app-server", - "generate-json-schema", - "--experimental", - "--out", - ]) - .arg(generated.path()) - .output() - .with_context(|| format!("generating the Codex app-server schema from {codex}"))?; - anyhow::ensure!( - output.status.success(), - "{codex} app-server schema generation failed: {}", - String::from_utf8_lossy(&output.stderr).trim() - ); - let read_schema = |name: &str| -> Result { - let path = generated.path().join(name); - let bytes = - fs::read(&path).with_context(|| format!("reading generated Codex schema {name}"))?; - serde_json::from_slice(&bytes) - .with_context(|| format!("parsing generated Codex schema {name}")) - }; - let schemas = CodexProtocolSchemas { - protocol: read_schema("codex_app_server_protocol.v2.schemas.json")?, - client_requests: read_schema("ClientRequest.json")?, - client_notifications: read_schema("ClientNotification.json")?, - server_requests: read_schema("ServerRequest.json")?, - server_notifications: read_schema("ServerNotification.json")?, - }; - verify_codex_protocol_schemas(&schemas) - .with_context(|| format!("Codex app-server schema from {version} is incompatible"))?; - Ok(version) -} - -fn codex_version(codex: &str) -> Result { - let mut attempt_index = 0; - let output = loop { - let attempt = Command::new(codex).arg("--version").output(); - match attempt { - Ok(output) => break output, - Err(error) if error.raw_os_error() == Some(libc::ETXTBSY) && attempt_index + 1 < 5 => { - // Some Linux filesystems briefly retain writer exclusion after a binary install. - // Retry only this transient error and keep every other launch error immediate. - attempt_index += 1; - thread::sleep(Duration::from_millis(20)); - } - Err(error) => { - return Err(error).with_context(|| format!("reading Codex version from {codex}")); - } - } - }; - anyhow::ensure!( - output.status.success(), - "{codex} --version failed: {}", - String::from_utf8_lossy(&output.stderr).trim() - ); - let actual = String::from_utf8(output.stdout) - .context("Codex version output is not UTF-8")? - .trim() - .to_string(); - anyhow::ensure!(!actual.is_empty(), "{codex} --version printed nothing"); - Ok(actual) -} - -fn verify_codex_protocol_schemas(schemas: &CodexProtocolSchemas) -> Result<()> { - let definitions = schemas - .protocol - .get("definitions") - .and_then(Value::as_object) - .context("aggregate schema has no definitions object")?; - - require_methods( - &schemas.client_requests, - REQUIRED_CODEX_CLIENT_REQUESTS, - "client request", - )?; - require_methods( - &schemas.client_notifications, - REQUIRED_CODEX_CLIENT_NOTIFICATIONS, - "client notification", - )?; - require_methods( - &schemas.server_notifications, - REQUIRED_CODEX_SERVER_NOTIFICATIONS, - "server notification", - )?; - schema_methods(&schemas.server_requests, "server request")?; - - let status_variants = schema_variants(definitions, "ThreadStatus", "type")?; - for status in ["notLoaded", "idle", "systemError", "active"] { - anyhow::ensure!( - status_variants.contains_key(status), - "ThreadStatus has no '{status}' variant" - ); - } - let active = status_variants - .get("active") - .context("ThreadStatus has no active variant")?; - let active_flags = - required_property(definitions, active, "activeFlags", "ThreadStatus.active")?; - let active_flag = require_array(definitions, active_flags, "ThreadStatus.activeFlags")?; - anyhow::ensure!( - active_flag == schema_definition(definitions, "ThreadActiveFlag")?, - "ThreadStatus.activeFlags does not contain ThreadActiveFlag" - ); - let actual_active_flags = schema_enum(definitions, "ThreadActiveFlag")?; - anyhow::ensure!( - actual_active_flags == string_set(&["waitingOnApproval", "waitingOnUserInput"]), - "ThreadActiveFlag changed: {}", - actual_active_flags - .into_iter() - .collect::>() - .join(", ") - ); - - let item_variants = schema_variants(definitions, "ThreadItem", "type")?; - for item in [ - "contextCompaction", - "enteredReviewMode", - "exitedReviewMode", - "userMessage", - ] { - anyhow::ensure!( - item_variants.contains_key(item), - "ThreadItem has no '{item}' variant" - ); - } - let user_message = item_variants - .get("userMessage") - .context("ThreadItem has no userMessage variant")?; - require_property_type( - definitions, - user_message, - "clientId", - "string", - false, - "ThreadItem.userMessage", - )?; - - let user_input_variants = schema_variants(definitions, "UserInput", "type")?; - let text_input = user_input_variants - .get("text") - .context("UserInput has no text variant")?; - require_property_type( - definitions, - text_input, - "text", - "string", - true, - "UserInput.text", - )?; - let text_elements = property(definitions, text_input, "text_elements", "UserInput.text")?; - require_array(definitions, text_elements, "UserInput.text.text_elements")?; - - for (definition, path) in [ - ("ClientInfo", &["name"][..]), - ("ClientInfo", &["version"][..]), - ("Thread", &["id"][..]), - ("Turn", &["id"][..]), - ("ThreadStatusChangedNotification", &["threadId"][..]), - ("TurnStartedNotification", &["threadId"][..]), - ("TurnStartedNotification", &["turn", "id"][..]), - ("TurnCompletedNotification", &["threadId"][..]), - ("TurnCompletedNotification", &["turn", "id"][..]), - ("ItemStartedNotification", &["threadId"][..]), - ("ItemStartedNotification", &["turnId"][..]), - ("ItemCompletedNotification", &["threadId"][..]), - ("ItemCompletedNotification", &["turnId"][..]), - ("ThreadStartedNotification", &["thread", "id"][..]), - ("ThreadResumeParams", &["threadId"][..]), - ("ThreadResumeResponse", &["thread", "id"][..]), - ("TurnStartParams", &["threadId"][..]), - ("TurnStartResponse", &["turn", "id"][..]), - ("TurnSteerParams", &["threadId"][..]), - ("TurnSteerParams", &["expectedTurnId"][..]), - ("TurnSteerResponse", &["turnId"][..]), - ] { - let schema = required_schema_path(definitions, definition, path)?; - require_type( - definitions, - schema, - "string", - &format!("{definition}.{}", path.join(".")), - )?; - } - - require_property_type( - definitions, - schema_definition(definitions, "ClientInfo")?, - "title", - "string", - false, - "ClientInfo", - )?; - require_property_type( - definitions, - schema_definition(definitions, "InitializeCapabilities")?, - "experimentalApi", - "boolean", - false, - "InitializeCapabilities", - )?; - required_schema_path(definitions, "InitializeParams", &["clientInfo"])?; - - let thread_status = required_schema_path(definitions, "Thread", &["status"])?; - anyhow::ensure!( - thread_status == schema_definition(definitions, "ThreadStatus")?, - "Thread.status does not use ThreadStatus" - ); - let resume_status = - required_schema_path(definitions, "ThreadResumeResponse", &["thread", "status"])?; - anyhow::ensure!( - resume_status == schema_definition(definitions, "ThreadStatus")?, - "ThreadResumeResponse.thread.status does not use ThreadStatus" - ); - let started_status = required_schema_path( - definitions, - "ThreadStartedNotification", - &["thread", "status"], - )?; - anyhow::ensure!( - started_status == schema_definition(definitions, "ThreadStatus")?, - "ThreadStartedNotification.thread.status does not use ThreadStatus" - ); - let changed_status = - required_schema_path(definitions, "ThreadStatusChangedNotification", &["status"])?; - anyhow::ensure!( - changed_status == schema_definition(definitions, "ThreadStatus")?, - "ThreadStatusChangedNotification.status does not use ThreadStatus" - ); - - let turns = required_schema_path(definitions, "Thread", &["turns"])?; - let turn = require_array(definitions, turns, "Thread.turns")?; - anyhow::ensure!( - turn == schema_definition(definitions, "Turn")?, - "Thread.turns does not contain Turn" - ); - let items = required_schema_path(definitions, "Turn", &["items"])?; - let item = require_array(definitions, items, "Turn.items")?; - anyhow::ensure!( - item == schema_definition(definitions, "ThreadItem")?, - "Turn.items does not contain ThreadItem" - ); - // The typed turn result the provider-credential classifier reads. A release that renames the - // status word, drops the failure's typed error, or merges the credential arm into a quota arm - // must refuse the launch rather than let st2 silently stop classifying rejections — or, worse, - // report an exhausted allowance as a rejected credential. - let turn_status = required_schema_path(definitions, "Turn", &["status"])?; - anyhow::ensure!( - turn_status == schema_definition(definitions, "TurnStatus")?, - "Turn.status does not use TurnStatus" - ); - let turn_statuses = schema_enum(definitions, "TurnStatus")?; - for status in ["completed", "failed"] { - anyhow::ensure!( - turn_statuses.contains(status), - "TurnStatus has no '{status}' variant" - ); - } - let turn_error = nullable_schema( - definitions, - property( - definitions, - schema_definition(definitions, "Turn")?, - "error", - "Turn", - )?, - "Turn.error", - )?; - anyhow::ensure!( - turn_error == schema_definition(definitions, "TurnError")?, - "Turn.error does not use TurnError" - ); - let error_info = nullable_schema( - definitions, - property( - definitions, - schema_definition(definitions, "TurnError")?, - "codexErrorInfo", - "TurnError", - )?, - "TurnError.codexErrorInfo", - )?; - anyhow::ensure!( - error_info == schema_definition(definitions, "CodexErrorInfo")?, - "TurnError.codexErrorInfo does not use CodexErrorInfo" - ); - let error_words = schema_variant_words(definitions, "CodexErrorInfo")?; - for word in [ - CODEX_PROVIDER_AUTH_REJECTED, - "rateLimitExceeded", - "usageLimitExceeded", - ] { - anyhow::ensure!( - error_words.contains(word), - "CodexErrorInfo has no '{word}' word" - ); - } - for notification in ["ItemStartedNotification", "ItemCompletedNotification"] { - let item = required_schema_path(definitions, notification, &["item"])?; - anyhow::ensure!( - item == schema_definition(definitions, "ThreadItem")?, - "{notification}.item does not use ThreadItem" - ); - } - - for params in ["TurnStartParams", "TurnSteerParams"] { - let input = required_schema_path(definitions, params, &["input"])?; - let input_item = require_array(definitions, input, &format!("{params}.input"))?; - anyhow::ensure!( - input_item == schema_definition(definitions, "UserInput")?, - "{params}.input does not contain UserInput" - ); - require_property_type( - definitions, - schema_definition(definitions, params)?, - "clientUserMessageId", - "string", - false, - params, - )?; - } - let loaded = required_schema_path(definitions, "ThreadLoadedListResponse", &["data"])?; - let loaded_item = require_array(definitions, loaded, "ThreadLoadedListResponse.data")?; - require_type( - definitions, - loaded_item, - "string", - "ThreadLoadedListResponse.data item", - )?; - let hook_cwds = property( - definitions, - schema_definition(definitions, "HooksListParams")?, - "cwds", - "HooksListParams", - )?; - let hook_cwd = require_array(definitions, hook_cwds, "HooksListParams.cwds")?; - require_type(definitions, hook_cwd, "string", "HooksListParams.cwds item")?; - verify_hook_schema(definitions)?; - Ok(()) -} - -fn verify_hook_schema(definitions: &serde_json::Map) -> Result<()> { - let data = required_schema_path(definitions, "HooksListResponse", &["data"])?; - let entry = require_array(definitions, data, "HooksListResponse.data")?; - anyhow::ensure!( - entry == schema_definition(definitions, "HooksListEntry")?, - "HooksListResponse.data does not contain HooksListEntry" - ); - let hooks = required_schema_path(definitions, "HooksListEntry", &["hooks"])?; - let hook = require_array(definitions, hooks, "HooksListEntry.hooks")?; - anyhow::ensure!( - hook == schema_definition(definitions, "HookMetadata")?, - "HooksListEntry.hooks does not contain HookMetadata" - ); - for (property, expected_type) in [ - ("currentHash", "string"), - ("isManaged", "boolean"), - ("key", "string"), - ] { - require_property_type( - definitions, - schema_definition(definitions, "HookMetadata")?, - property, - expected_type, - true, - "HookMetadata", - )?; - } - let trust_status = required_schema_path(definitions, "HookMetadata", &["trustStatus"])?; - anyhow::ensure!( - trust_status == schema_definition(definitions, "HookTrustStatus")?, - "HookMetadata.trustStatus does not use HookTrustStatus" - ); - let statuses = schema_enum(definitions, "HookTrustStatus")?; - anyhow::ensure!( - statuses == string_set(&["managed", "modified", "trusted", "untrusted"]), - "HookTrustStatus changed: {}", - statuses.into_iter().collect::>().join(", ") - ); - Ok(()) -} - -fn string_set(values: &[&str]) -> BTreeSet { - values.iter().map(|value| (*value).to_string()).collect() -} - -fn schema_methods(schema: &Value, label: &str) -> Result> { - let arms = schema - .get("oneOf") - .and_then(Value::as_array) - .with_context(|| format!("{label} schema has no oneOf array"))?; - let mut methods = BTreeSet::new(); - for arm in arms { - let required = arm - .get("required") - .and_then(Value::as_array) - .with_context(|| format!("{label} arm has no required array"))?; - anyhow::ensure!( - required - .iter() - .any(|value| value.as_str() == Some("method")), - "{label} arm does not require method" - ); - let values = arm - .pointer("/properties/method/enum") - .and_then(Value::as_array) - .with_context(|| format!("{label} arm has no method enum"))?; - anyhow::ensure!(values.len() == 1, "{label} arm method enum is not exact"); - let method = values[0] - .as_str() - .with_context(|| format!("{label} arm method is not a string"))?; - anyhow::ensure!( - methods.insert(method.to_string()), - "{label} method '{method}' is duplicated" - ); - } - Ok(methods) -} - -fn require_methods(schema: &Value, required: &[&str], label: &str) -> Result<()> { - let methods = schema_methods(schema, label)?; - let missing = string_set(required) - .difference(&methods) - .cloned() - .collect::>(); - anyhow::ensure!( - missing.is_empty(), - "missing {label} methods: {}", - missing.join(", ") - ); - Ok(()) -} - -fn schema_definition<'a>( - definitions: &'a serde_json::Map, - name: &str, -) -> Result<&'a Value> { - definitions - .get(name) - .with_context(|| format!("aggregate schema has no {name} definition")) -} - -fn resolve_schema<'a>( - definitions: &'a serde_json::Map, - mut schema: &'a Value, -) -> Result<&'a Value> { - for _ in 0..16 { - if let Some(reference) = schema.get("$ref").and_then(Value::as_str) { - let name = reference - .strip_prefix("#/definitions/") - .with_context(|| format!("unsupported schema reference '{reference}'"))?; - schema = schema_definition(definitions, name)?; - continue; - } - if let Some(all_of) = schema.get("allOf").and_then(Value::as_array) { - anyhow::ensure!(all_of.len() == 1, "schema allOf is not a single reference"); - schema = &all_of[0]; - continue; - } - return Ok(schema); - } - anyhow::bail!("schema reference depth exceeds 16") -} - -/// Resolve `anyOf: [T, null]` — the shape the Codex generator emits for an optional typed field — -/// to `T`. A field that is not exactly one typed arm beside `null` is refused rather than guessed. -fn nullable_schema<'a>( - definitions: &'a serde_json::Map, - schema: &'a Value, - label: &str, -) -> Result<&'a Value> { - let schema = resolve_schema(definitions, schema)?; - let arms = schema - .get("anyOf") - .and_then(Value::as_array) - .with_context(|| format!("{label} is not a nullable schema"))?; - let mut typed = arms - .iter() - .filter(|arm| arm.get("type").and_then(Value::as_str) != Some("null")); - let only = typed - .next() - .with_context(|| format!("{label} has no typed arm"))?; - anyhow::ensure!( - typed.next().is_none(), - "{label} has more than one typed arm" - ); - resolve_schema(definitions, only) -} - -/// Every unit word of a `oneOf` union that mixes a string enum with data-carrying object arms — -/// the shape `CodexErrorInfo` has. Only the enum arms carry words st2 can match on. -fn schema_variant_words( - definitions: &serde_json::Map, - definition: &str, -) -> Result> { - let arms = schema_definition(definitions, definition)? - .get("oneOf") - .and_then(Value::as_array) - .with_context(|| format!("{definition} has no oneOf variants"))?; - let mut words = BTreeSet::new(); - for arm in arms { - let Some(values) = resolve_schema(definitions, arm)? - .get("enum") - .and_then(Value::as_array) - else { - continue; - }; - for value in values { - let word = value - .as_str() - .with_context(|| format!("{definition} has a non-string enum value"))?; - words.insert(word.to_string()); - } - } - anyhow::ensure!(!words.is_empty(), "{definition} has no enum words"); - Ok(words) -} - -fn schema_variants<'a>( - definitions: &'a serde_json::Map, - definition: &str, - discriminator: &str, -) -> Result> { - let schema = schema_definition(definitions, definition)?; - let variants = schema - .get("oneOf") - .and_then(Value::as_array) - .with_context(|| format!("{definition} has no oneOf variants"))?; - let mut found = BTreeMap::new(); - for variant in variants { - let variant = resolve_schema(definitions, variant)?; - let required = variant - .get("required") - .and_then(Value::as_array) - .with_context(|| format!("{definition} variant has no required array"))?; - anyhow::ensure!( - required - .iter() - .any(|value| value.as_str() == Some(discriminator)), - "{definition} variant does not require {discriminator}" - ); - let values = variant - .pointer(&format!("/properties/{discriminator}/enum")) - .and_then(Value::as_array) - .with_context(|| format!("{definition} variant has no {discriminator} enum"))?; - anyhow::ensure!( - values.len() == 1, - "{definition} variant discriminator is not exact" - ); - let value = values[0] - .as_str() - .with_context(|| format!("{definition} discriminator is not a string"))?; - anyhow::ensure!( - found.insert(value.to_string(), variant).is_none(), - "{definition} discriminator '{value}' is duplicated" - ); - } - Ok(found) -} - -fn schema_enum( - definitions: &serde_json::Map, - definition: &str, -) -> Result> { - let values = schema_definition(definitions, definition)? - .get("enum") - .and_then(Value::as_array) - .with_context(|| format!("{definition} has no enum"))?; - values - .iter() - .map(|value| { - value - .as_str() - .map(str::to_string) - .with_context(|| format!("{definition} has a non-string enum value")) - }) - .collect() -} - -fn property<'a>( - definitions: &'a serde_json::Map, - schema: &'a Value, - name: &str, - label: &str, -) -> Result<&'a Value> { - let schema = resolve_schema(definitions, schema)?; - let property = schema - .get("properties") - .and_then(Value::as_object) - .and_then(|properties| properties.get(name)) - .with_context(|| format!("{label} has no {name} property"))?; - resolve_schema(definitions, property) -} - -fn required_property<'a>( - definitions: &'a serde_json::Map, - schema: &'a Value, - name: &str, - label: &str, -) -> Result<&'a Value> { - let schema = resolve_schema(definitions, schema)?; - let required = schema - .get("required") - .and_then(Value::as_array) - .with_context(|| format!("{label} has no required array"))?; - anyhow::ensure!( - required.iter().any(|value| value.as_str() == Some(name)), - "{label} does not require {name}" - ); - property(definitions, schema, name, label) -} - -fn required_schema_path<'a>( - definitions: &'a serde_json::Map, - definition: &str, - path: &[&str], -) -> Result<&'a Value> { - let mut schema = schema_definition(definitions, definition)?; - let mut label = definition.to_string(); - for component in path { - schema = required_property(definitions, schema, component, &label)?; - label.push('.'); - label.push_str(component); - } - Ok(schema) -} - -fn require_property_type( - definitions: &serde_json::Map, - schema: &Value, - property_name: &str, - expected_type: &str, - required: bool, - label: &str, -) -> Result<()> { - let property = if required { - required_property(definitions, schema, property_name, label)? - } else { - property(definitions, schema, property_name, label)? - }; - require_type( - definitions, - property, - expected_type, - &format!("{label}.{property_name}"), - ) -} - -fn require_type( - definitions: &serde_json::Map, - schema: &Value, - expected: &str, - label: &str, -) -> Result<()> { - let schema = resolve_schema(definitions, schema)?; - let matches = match schema.get("type") { - Some(Value::String(actual)) => actual == expected, - Some(Value::Array(actual)) => actual.iter().any(|value| value.as_str() == Some(expected)), - _ => false, - }; - anyhow::ensure!(matches, "{label} does not accept {expected}"); - Ok(()) -} - -fn require_array<'a>( - definitions: &'a serde_json::Map, - schema: &'a Value, - label: &str, -) -> Result<&'a Value> { - let schema = resolve_schema(definitions, schema)?; - require_type(definitions, schema, "array", label)?; - let items = schema - .get("items") - .with_context(|| format!("{label} has no item schema"))?; - resolve_schema(definitions, items) -} +mod protocol; +use self::protocol::*; pub fn state_dir(catalog_root: &Path, identity: &str) -> PathBuf { let base = std::env::var_os("XDG_STATE_HOME") @@ -4163,126 +3210,8 @@ fn poll_json_message(websocket: &mut WebSocket) -> Result, - socket_path: Option, - active: bool, -} - -impl OwnedProcessGroup { - fn id(&self) -> u32 { - self.child.id() - } - - fn child_mut(&mut self) -> &mut Child { - &mut self.child - } - - fn terminate(&mut self) { - if !self.active { - return; - } - self.active = false; - let process_group = self.watchdog.id() as i32; - unsafe { - libc::kill(-process_group, libc::SIGKILL); - } - let _ = self.child.kill(); - let _ = self.child.wait(); - let _ = self.watchdog.kill(); - let _ = self.watchdog.wait(); - if let Some(socket_path) = self.socket_path.as_deref() { - let _ = fs::remove_file(socket_path); - } - self.owner_write.take(); - } -} - -impl Drop for OwnedProcessGroup { - fn drop(&mut self) { - self.terminate(); - } -} - -fn set_close_on_exec(fd: libc::c_int) -> std::io::Result<()> { - let mut flags = unsafe { libc::fcntl(fd, libc::F_GETFD) }; - if flags == -1 { - return Err(std::io::Error::last_os_error()); - } - flags |= libc::FD_CLOEXEC; - if unsafe { libc::fcntl(fd, libc::F_SETFD, flags) } == -1 { - return Err(std::io::Error::last_os_error()); - } - Ok(()) -} - -/// Spawn a provider launcher in an isolated, wrapper-owned process group. -/// -/// Explicit cleanup covers normal returns and Rust errors. The in-group watchdog covers wrapper -/// crashes, SIGKILL, and supervisor teardown. The watchdog holds the group ID until cleanup, so a -/// stale PID can never identify a process group that belongs to another live owner. A crash can -/// leave one dead socket file; the next launch proves that it has no listener and removes it. -fn spawn_process_group( - command: &mut Command, - socket_path: Option<&Path>, -) -> std::io::Result { - let (watchdog_read, owner_write) = UnixStream::pair()?; - set_close_on_exec(owner_write.as_raw_fd())?; - let owner_write_fd = owner_write.as_raw_fd(); - let mut watchdog_command = Command::new("/bin/sh"); - watchdog_command - .arg("-c") - .arg("IFS= read -r ignored; kill -KILL 0") - .arg("st2-codex-watchdog") - .stdin(Stdio::from(std::os::fd::OwnedFd::from(watchdog_read))) - .stdout(Stdio::null()) - .stderr(Stdio::null()); - unsafe { - watchdog_command.pre_exec(|| { - if libc::setpgid(0, 0) == -1 { - return Err(std::io::Error::last_os_error()); - } - Ok(()) - }); - } - let mut watchdog = watchdog_command.spawn()?; - let watchdog_process_group = watchdog.id() as i32; - unsafe { - command.pre_exec(move || { - if libc::setpgid(0, watchdog_process_group) == -1 { - return Err(std::io::Error::last_os_error()); - } - libc::close(owner_write_fd); - Ok(()) - }); - } - let child = match command.spawn() { - Ok(child) => child, - Err(error) => { - drop(owner_write); - unsafe { - libc::kill(-watchdog_process_group, libc::SIGKILL); - } - let _ = watchdog.kill(); - let _ = watchdog.wait(); - return Err(error); - } - }; - Ok(OwnedProcessGroup { - child, - watchdog, - owner_write: Some(owner_write), - socket_path: socket_path.map(Path::to_path_buf), - active: true, - }) -} +mod process_group; +use self::process_group::*; fn terminate_child(child: &mut Child) { match child.try_wait() { @@ -4295,4198 +3224,4 @@ fn terminate_child(child: &mut Child) { } #[cfg(test)] -mod tests { - use super::*; - - /// The stop flag is process-global, so every test that exercises a reader of it — - /// [`initialize_control`] above all — holds this lock against the one test that flips - /// the flag: parallel readers would otherwise observe the raised flag and fail their - /// `no stop raised in tests` expectations. - fn stop_flag_tests() -> std::sync::MutexGuard<'static, ()> { - static LOCK: std::sync::LazyLock> = - std::sync::LazyLock::new(std::sync::Mutex::default); - match LOCK.lock() { - Ok(guard) => guard, - Err(poisoned) => poisoned.into_inner(), - } - } - - #[test] - fn a_stop_during_the_websocket_handshake_ends_startup_gracefully() { - let _stop_exclusive = stop_flag_tests(); - let tmp = tempfile::tempdir().unwrap(); - let socket_path = tmp.path().join("control.sock"); - let listener = UnixListener::bind(&socket_path).unwrap(); - let silent_server = std::thread::spawn(move || listener.accept().map(|(stream, _)| stream)); - let stream = UnixStream::connect(&socket_path).unwrap(); - let stopper = std::thread::spawn(|| { - std::thread::sleep(Duration::from_millis(300)); - crate::provider_session::STOP.store(true, std::sync::atomic::Ordering::SeqCst); - }); - let started = Instant::now(); - let result = initialize_control(stream); - // Join before resetting: on an early failure return the stopper has not fired yet, - // and resetting first would let it re-poison the global flag for every later test. - stopper.join().unwrap(); - crate::provider_session::STOP.store(false, std::sync::atomic::Ordering::SeqCst); - let _held_open = silent_server.join().unwrap().unwrap(); - assert!( - result.unwrap().is_none(), - "a stop while the server sits silent mid-handshake must return the graceful None" - ); - assert!( - started.elapsed() < STARTUP_TIMEOUT, - "the stop must unblock the handshake well before the startup timeout" - ); - } - use std::os::unix::fs::PermissionsExt; - use std::os::unix::net::UnixListener; - - #[cfg(target_os = "linux")] - fn linux_process_state(pid: i32) -> Option { - std::fs::read_to_string(format!("/proc/{pid}/stat")) - .ok()? - .rsplit_once(") ")? - .1 - .chars() - .next() - } - - fn process_can_retain_cleanup_resources(pid: i32) -> bool { - #[cfg(target_os = "linux")] - if linux_process_state(pid) == Some('Z') { - return false; - } - crate::host_lock::process_alive(pid) - } - - fn object_schema(required: &[&str], properties: &[(&str, Value)]) -> Value { - json!({ - "type": "object", - "required": required, - "properties": properties - .iter() - .map(|(name, schema)| ((*name).to_string(), schema.clone())) - .collect::>() - }) - } - - fn reference(name: &str) -> Value { - json!({ "$ref": format!("#/definitions/{name}") }) - } - - fn array_of(items: Value) -> Value { - json!({ "type": "array", "items": items }) - } - - fn tagged_variant(name: &str, required: &[&str], properties: &[(&str, Value)]) -> Value { - let mut all_required = vec!["type"]; - all_required.extend(required); - let mut all_properties = vec![("type", json!({ "type": "string", "enum": [name] }))]; - all_properties.extend(properties.iter().cloned()); - object_schema(&all_required, &all_properties) - } - - fn method_schema(methods: &[&str]) -> Value { - json!({ - "oneOf": methods - .iter() - .map(|method| object_schema( - &["method"], - &[("method", json!({ "type": "string", "enum": [method] }))], - )) - .collect::>() - }) - } - - fn compatible_protocol_schemas() -> CodexProtocolSchemas { - let mut definitions = serde_json::Map::new(); - definitions.insert( - "ThreadActiveFlag".into(), - json!({ - "type": "string", - "enum": ["waitingOnApproval", "waitingOnUserInput"] - }), - ); - definitions.insert( - "ThreadStatus".into(), - json!({ - "oneOf": [ - tagged_variant("notLoaded", &[], &[]), - tagged_variant("idle", &[], &[]), - tagged_variant("systemError", &[], &[]), - tagged_variant( - "active", - &["activeFlags"], - &[("activeFlags", array_of(reference("ThreadActiveFlag")))], - ) - ] - }), - ); - definitions.insert( - "ThreadItem".into(), - json!({ - "oneOf": [ - tagged_variant("contextCompaction", &[], &[]), - tagged_variant("enteredReviewMode", &[], &[]), - tagged_variant("exitedReviewMode", &[], &[]), - tagged_variant( - "userMessage", - &[], - &[("clientId", json!({ "type": ["string", "null"] }))], - ) - ] - }), - ); - definitions.insert("TextElement".into(), object_schema(&[], &[])); - definitions.insert( - "UserInput".into(), - json!({ - "oneOf": [tagged_variant( - "text", - &["text"], - &[ - ("text", json!({ "type": "string" })), - ("text_elements", array_of(reference("TextElement"))), - ], - )] - }), - ); - definitions.insert( - "ClientInfo".into(), - object_schema( - &["name", "version"], - &[ - ("name", json!({ "type": "string" })), - ("title", json!({ "type": ["string", "null"] })), - ("version", json!({ "type": "string" })), - ], - ), - ); - definitions.insert( - "InitializeCapabilities".into(), - object_schema(&[], &[("experimentalApi", json!({ "type": "boolean" }))]), - ); - definitions.insert( - "InitializeParams".into(), - object_schema( - &["clientInfo"], - &[ - ("clientInfo", reference("ClientInfo")), - ("capabilities", reference("InitializeCapabilities")), - ], - ), - ); - definitions.insert( - "Thread".into(), - object_schema( - &["id", "status", "turns"], - &[ - ("id", json!({ "type": "string" })), - ("status", reference("ThreadStatus")), - ("turns", array_of(reference("Turn"))), - ], - ), - ); - definitions.insert( - "Turn".into(), - object_schema( - &["id", "items", "status"], - &[ - ("id", json!({ "type": "string" })), - ("items", array_of(reference("ThreadItem"))), - ("status", reference("TurnStatus")), - ( - "error", - json!({ "anyOf": [reference("TurnError"), { "type": "null" }] }), - ), - ], - ), - ); - definitions.insert( - "TurnStatus".into(), - json!({ - "type": "string", - "enum": ["completed", "interrupted", "failed", "inProgress"] - }), - ); - definitions.insert( - "TurnError".into(), - object_schema( - &["message"], - &[ - ("message", json!({ "type": "string" })), - ( - "codexErrorInfo", - json!({ "anyOf": [reference("CodexErrorInfo"), { "type": "null" }] }), - ), - ], - ), - ); - definitions.insert( - "CodexErrorInfo".into(), - json!({ - "oneOf": [ - { - "type": "string", - "enum": [ - "usageLimitExceeded", - "rateLimitExceeded", - "unauthorized", - "other" - ] - }, - object_schema( - &["httpConnectionFailed"], - &[("httpConnectionFailed", object_schema(&[], &[]))], - ) - ] - }), - ); - for notification in ["TurnStartedNotification", "TurnCompletedNotification"] { - definitions.insert( - notification.into(), - object_schema( - &["threadId", "turn"], - &[ - ("threadId", json!({ "type": "string" })), - ("turn", reference("Turn")), - ], - ), - ); - } - for notification in ["ItemStartedNotification", "ItemCompletedNotification"] { - definitions.insert( - notification.into(), - object_schema( - &["threadId", "turnId", "item"], - &[ - ("threadId", json!({ "type": "string" })), - ("turnId", json!({ "type": "string" })), - ("item", reference("ThreadItem")), - ], - ), - ); - } - definitions.insert( - "ThreadStartedNotification".into(), - object_schema(&["thread"], &[("thread", reference("Thread"))]), - ); - definitions.insert( - "ThreadStatusChangedNotification".into(), - object_schema( - &["threadId", "status"], - &[ - ("threadId", json!({ "type": "string" })), - ("status", reference("ThreadStatus")), - ], - ), - ); - definitions.insert( - "ThreadResumeParams".into(), - object_schema(&["threadId"], &[("threadId", json!({ "type": "string" }))]), - ); - definitions.insert( - "ThreadResumeResponse".into(), - object_schema(&["thread"], &[("thread", reference("Thread"))]), - ); - definitions.insert( - "TurnStartParams".into(), - object_schema( - &["threadId", "input"], - &[ - ("threadId", json!({ "type": "string" })), - ("input", array_of(reference("UserInput"))), - ("clientUserMessageId", json!({ "type": ["string", "null"] })), - ], - ), - ); - definitions.insert( - "TurnSteerParams".into(), - object_schema( - &["threadId", "expectedTurnId", "input"], - &[ - ("threadId", json!({ "type": "string" })), - ("expectedTurnId", json!({ "type": "string" })), - ("input", array_of(reference("UserInput"))), - ("clientUserMessageId", json!({ "type": ["string", "null"] })), - ], - ), - ); - definitions.insert( - "TurnStartResponse".into(), - object_schema(&["turn"], &[("turn", reference("Turn"))]), - ); - definitions.insert( - "TurnSteerResponse".into(), - object_schema(&["turnId"], &[("turnId", json!({ "type": "string" }))]), - ); - definitions.insert( - "ThreadLoadedListResponse".into(), - object_schema( - &["data"], - &[("data", array_of(json!({ "type": "string" })))], - ), - ); - definitions.insert( - "HooksListParams".into(), - object_schema(&[], &[("cwds", array_of(json!({ "type": "string" })))]), - ); - definitions.insert( - "HooksListResponse".into(), - object_schema( - &["data"], - &[("data", array_of(reference("HooksListEntry")))], - ), - ); - definitions.insert( - "HooksListEntry".into(), - object_schema( - &["hooks"], - &[("hooks", array_of(reference("HookMetadata")))], - ), - ); - definitions.insert( - "HookMetadata".into(), - object_schema( - &["currentHash", "isManaged", "key", "trustStatus"], - &[ - ("currentHash", json!({ "type": "string" })), - ("isManaged", json!({ "type": "boolean" })), - ("key", json!({ "type": "string" })), - ("trustStatus", reference("HookTrustStatus")), - ], - ), - ); - definitions.insert( - "HookTrustStatus".into(), - json!({ - "type": "string", - "enum": ["managed", "modified", "trusted", "untrusted"] - }), - ); - CodexProtocolSchemas { - protocol: json!({ "definitions": definitions }), - client_requests: method_schema(REQUIRED_CODEX_CLIENT_REQUESTS), - client_notifications: method_schema(REQUIRED_CODEX_CLIENT_NOTIFICATIONS), - server_requests: method_schema(&["currentTime/read"]), - server_notifications: method_schema(REQUIRED_CODEX_SERVER_NOTIFICATIONS), - } - } - - fn write_fake_codex( - root: &Path, - name: &str, - version: &str, - schemas: &CodexProtocolSchemas, - ) -> PathBuf { - let fixture = root.join(format!("{name}-schemas")); - fs::create_dir(&fixture).unwrap(); - for (filename, schema) in [ - ( - "codex_app_server_protocol.v2.schemas.json", - &schemas.protocol, - ), - ("ClientRequest.json", &schemas.client_requests), - ("ClientNotification.json", &schemas.client_notifications), - ("ServerRequest.json", &schemas.server_requests), - ("ServerNotification.json", &schemas.server_notifications), - ] { - fs::write(fixture.join(filename), serde_json::to_vec(schema).unwrap()).unwrap(); - } - let path = root.join(name); - fs::write( - &path, - format!( - "#!/bin/sh\nif [ \"$1\" = \"--version\" ]; then printf '%s\\n' '{version}'; exit 0; fi\nout=\nwhile [ \"$#\" -gt 0 ]; do if [ \"$1\" = \"--out\" ]; then out=$2; break; fi; shift; done\n[ -n \"$out\" ] || exit 2\ncp '{fixture}/'*.json \"$out/\"\n", - fixture = fixture.display() - ), - ) - .unwrap(); - fs::set_permissions(&path, fs::Permissions::from_mode(0o755)).unwrap(); - path - } - - #[test] - fn protocol_schema_gate_accepts_a_compatible_release_and_rejects_shape_drift() { - let tmp = tempfile::tempdir().unwrap(); - let compatible = compatible_protocol_schemas(); - let patch = write_fake_codex( - tmp.path(), - "codex-compatible-patch", - "codex-cli 0.150.0", - &compatible, - ); - ensure_supported_protocol(patch.to_str().unwrap()).unwrap(); - - let mut incompatible = compatible_protocol_schemas(); - incompatible - .protocol - .pointer_mut("/definitions/ThreadActiveFlag/enum") - .unwrap() - .as_array_mut() - .unwrap() - .push(Value::String("waitingOnFutureInput".into())); - let incompatible = write_fake_codex( - tmp.path(), - "codex-incompatible-schema", - "codex-cli 0.150.1", - &incompatible, - ); - let error = ensure_supported_protocol(incompatible.to_str().unwrap()).unwrap_err(); - assert!(format!("{error:#}").contains("ThreadActiveFlag changed")); - } - - #[test] - fn protocol_schema_gate_accepts_additive_items_and_server_requests() { - let mut schemas = compatible_protocol_schemas(); - schemas - .server_requests - .get_mut("oneOf") - .unwrap() - .as_array_mut() - .unwrap() - .push( - method_schema(&["future/request"]) - .get_mut("oneOf") - .unwrap() - .as_array_mut() - .unwrap() - .remove(0), - ); - schemas - .protocol - .pointer_mut("/definitions/ThreadItem/oneOf") - .unwrap() - .as_array_mut() - .unwrap() - .push(tagged_variant("futureItem", &[], &[])); - - verify_codex_protocol_schemas(&schemas).unwrap(); - } - - /// The classifier reads one word out of `Turn.error.codexErrorInfo` and depends on it being - /// distinct from the quota words. A release that dropped or merged it must refuse the launch - /// rather than let st2 report an exhausted allowance as a rejected credential. - #[test] - fn protocol_schema_gate_requires_the_distinct_credential_and_quota_error_words() { - let mut schemas = compatible_protocol_schemas(); - let words = schemas - .protocol - .pointer_mut("/definitions/CodexErrorInfo/oneOf/0/enum") - .unwrap() - .as_array_mut() - .unwrap(); - words.retain(|word| word.as_str() != Some("unauthorized")); - let error = verify_codex_protocol_schemas(&schemas).unwrap_err(); - assert!( - format!("{error:#}").contains("CodexErrorInfo has no 'unauthorized' word"), - "{error:#}" - ); - - let mut merged = compatible_protocol_schemas(); - merged - .protocol - .pointer_mut("/definitions/CodexErrorInfo/oneOf/0/enum") - .unwrap() - .as_array_mut() - .unwrap() - .retain(|word| word.as_str() != Some("rateLimitExceeded")); - let error = verify_codex_protocol_schemas(&merged).unwrap_err(); - assert!( - format!("{error:#}").contains("CodexErrorInfo has no 'rateLimitExceeded' word"), - "{error:#}" - ); - - let mut untyped = compatible_protocol_schemas(); - untyped - .protocol - .pointer_mut("/definitions/Turn/properties") - .unwrap() - .as_object_mut() - .unwrap() - .remove("error"); - let error = verify_codex_protocol_schemas(&untyped).unwrap_err(); - assert!( - format!("{error:#}").contains("Turn has no error property"), - "{error:#}" - ); - } - - #[test] - fn protocol_rejection_reaches_the_declared_supervisor_once() { - let tmp = tempfile::tempdir().unwrap(); - let worker = tmp.path().join("agents/h/worker/agent.kdl"); - let supervisor = tmp.path().join("agents/h/cos/agent.kdl"); - fs::create_dir_all(worker.parent().unwrap()).unwrap(); - fs::create_dir_all(supervisor.parent().unwrap()).unwrap(); - fs::write( - &worker, - r#"agent "worker" { - host "h" - supervisor "h.cos" - command "true" -} -"#, - ) - .unwrap(); - fs::write( - &supervisor, - r#"agent "cos" { - host "h" - command "true" -} -"#, - ) - .unwrap(); - let mut incompatible = compatible_protocol_schemas(); - incompatible - .protocol - .pointer_mut("/definitions/ThreadActiveFlag/enum") - .unwrap() - .as_array_mut() - .unwrap() - .push(Value::String("waitingOnFutureInput".into())); - let codex = write_fake_codex( - tmp.path(), - "codex-rejected", - "codex-cli 0.150.1", - &incompatible, - ); - let argv = vec![codex.display().to_string()]; - - for _ in 0..2 { - let error = run_controlled( - tmp.path(), - "h.worker".into(), - "h.worker".into(), - argv.clone(), - ) - .unwrap_err(); - assert!(format!("{error:#}").contains("ThreadActiveFlag changed")); - } - - let inbox = message::list_inbox(&message::inbox_dir(supervisor.parent().unwrap())).unwrap(); - assert_eq!(inbox.len(), 1, "the rejection report was not idempotent"); - assert_eq!(inbox[0].from.as_deref(), Some("h.worker")); - assert_eq!( - inbox[0].subject.as_deref(), - Some("Codex protocol rejected: h.worker") - ); - assert!(inbox[0].body.contains("Native delivery did not start")); - assert!(inbox[0].body.contains("ThreadActiveFlag changed")); - } - - #[test] - fn unknown_thread_status_remains_a_hold_not_a_terminal_system_error() { - let mut state = subscribed_state(CodexObservedState::Idle); - state.observe_thread_status("futureStatus", None); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::UnknownStatus, - turn_id: None, - } - ); - state.observe_turn_completed("turn-future", CodexTurnOutcome::Indeterminate); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::UnknownStatus, - turn_id: None, - } - ); - } - - #[test] - fn tui_loaded_deadline_precedes_the_outer_binding_deadline() { - assert!(TUI_LOADED_TIMEOUT < STARTUP_TIMEOUT); - } - - /// An agent directory with a parent to stage into, and a producer over it carrying a fixed - /// incarnation so the record's provenance is assertable. - fn context_producer(root: &Path) -> (PathBuf, CodexContextProducer) { - let agent_dir = root.join("agents/h/worker"); - fs::create_dir_all(&agent_dir).unwrap(); - let writer = - harness_context::Writer::new(&agent_dir, "h.worker", harness_context::Harness::Codex) - .unwrap() - .with_session("codex-incarnation"); - (agent_dir, CodexContextProducer::new(writer)) - } - - fn context_record(agent_dir: &Path) -> Option { - harness_context::read(&harness_context::harness_context_path(agent_dir)) - } - - fn token_usage_frame(last_total: i64, window: Value) -> Value { - json!({ - "method": "thread/tokenUsage/updated", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "tokenUsage": { - "last": { "totalTokens": last_total }, - "total": { "totalTokens": last_total }, - "modelContextWindow": window - } - } - }) - } - - fn compaction_item_frame(method: &str, turn_id: &str, item_id: &str) -> Value { - json!({ - "method": method, - "params": { - "threadId": "thread-main", - "turnId": turn_id, - "item": { "id": item_id, "type": "contextCompaction" } - } - }) - } - - /// HC-R13's Codex fixture. The frames are a transposition, and the comment says which half came - /// from where: the SHAPE is codex-cli 0.151.0's own app-server schema dump - /// (`ThreadTokenUsageUpdatedNotification`, `AccountRateLimitsUpdatedNotification`), while the - /// NUMBERS are verbatim from a real rollout captured on 2026-08-29 from a 0.150.1 session - /// (`session_meta.payload.cli_version = "0.150.1"`) — its first and last `token_count` events - /// and the `rate_limits` snapshot riding them. Fields the capture elided are omitted rather - /// than invented; this producer reads three numbers and must not need the rest. - /// - /// What must fail here when a codex bump moves something: the 12,000 baseline (the percent - /// changes), the numerator (`total` reads 100 and `last.inputTokens` without the baseline reads - /// 36 against this very capture, both asserted below), and the version literal itself, which is - /// the only thing tying this arithmetic to a build whose source was actually read. - #[test] - fn codex_context_recomputes_the_captured_reading_and_pins_its_verified_version() { - assert_eq!(CODEX_CONTEXT_VERIFIED_VERSION, "0.151.0"); - assert_eq!(CODEX_BASELINE_TOKENS, 12_000); - - let frames = include_str!("../tests/fixtures/codex_token_usage_inbound.jsonl") - .lines() - .map(|line| serde_json::from_str::(line).unwrap()) - .collect::>(); - assert_eq!(frames.len(), 3); - - let tmp = tempfile::tempdir().unwrap(); - let (agent_dir, mut producer) = context_producer(tmp.path()); - - // The session's FIRST reading: 32,237 of 258,400 with the baseline normalized out is 8%, - // and no rate-limit notification has arrived yet, so both windows are honestly absent. - assert!(producer.observe(&frames[0], "thread-main").unwrap()); - let first = context_record(&agent_dir).unwrap(); - assert_eq!(first.used_tokens, Some(32_237)); - assert_eq!(first.window_tokens, Some(258_400)); - assert_eq!(first.used_percent, Some(8.0)); - assert_eq!(first.rate_limits, harness_context::RateLimits::default()); - - // The account-scoped snapshot carries no occupancy, so it writes nothing on its own and is - // held for the next reading (HC-T06). - assert!(!producer.observe(&frames[1], "thread-main").unwrap()); - let mut unchanged = context_record(&agent_dir).unwrap(); - // `age_ms` is derived at read time, not stored, so it moves between two reads of one - // record. Everything the record itself carries — including `observed_at_ms`, which is what - // proves no write happened — must be identical. - assert!(unchanged.age_ms >= first.age_ms); - unchanged.age_ms = first.age_ms; - assert_eq!(unchanged, first); - - assert!(producer.observe(&frames[2], "thread-main").unwrap()); - let observed = context_record(&agent_dir).unwrap(); - assert_eq!(observed.harness, harness_context::Harness::Codex); - assert_eq!(observed.used_tokens, Some(92_283)); - assert_eq!(observed.window_tokens, Some(258_400)); - // 100 − Codex's displayed "67% context left" for this exact capture. - assert_eq!(observed.used_percent, Some(33.0)); - assert_eq!(observed.session_total_tokens, Some(2_235_329)); - // The channel carries neither: `Thread` has `modelProvider` and no model identifier, and - // Codex reports no session cost anywhere in the protocol. - assert_eq!(observed.model, None); - assert_eq!(observed.cost_usd, None); - // Only the seven-day window was ever captured on this harness; the five-hour leg is not - // inferred from a field name (see `observe_rate_limits`). - assert_eq!( - observed.rate_limits, - harness_context::RateLimits { - five_hour: None, - seven_day: Some(44.0), - } - ); - assert_eq!(observed.compactions, 0); - assert_eq!(observed.last_compaction_ms, None); - - // The trap, asserted rather than described: the cumulative session total is 2,235,329 - // against a 258,400-token window. A producer that used it as the numerator would publish a - // saturated 100 for a window that is a third full. - assert_eq!(codex_used_percent(Some(258_400), 2_235_329), Some(100.0)); - assert_ne!( - codex_used_percent(Some(258_400), 2_235_329), - observed.used_percent - ); - // And the baseline-free percent over the same operands is 36 — close enough to look right. - let baseline_free = (92_283.0_f64 / 258_400.0 * 100.0).round(); - assert_eq!(baseline_free, 36.0); - assert_ne!(Some(baseline_free), observed.used_percent); - - // Mirroring is not the same function as rounding the used percentage: at an exact half - // they disagree. Effective window 200, used 101 — Codex displays 50% left, so st2 publishes - // 50; rounding `used/effective` would publish 51. - assert_eq!(codex_used_percent(Some(12_200), 12_101), Some(50.0)); - assert_eq!((101.0_f64 / 200.0 * 100.0).round(), 51.0); - } - - /// HC-R02/HC-R03: the operands are the harness's and are published as they arrive; only the - /// percent is withheld, and only where Codex's own normalization cannot run. A window at or - /// below the baseline is the sharp case — Codex itself returns "0% remaining" there, which - /// mirrored blindly would publish a fabricated 100% used. - #[test] - fn a_missing_or_unnormalizable_window_withholds_the_percent_but_not_the_operands() { - for (window, expected_window) in [ - (Value::Null, None), - (json!(12_000), Some(12_000)), - (json!(0), None), - ] { - let tmp = tempfile::tempdir().unwrap(); - let (agent_dir, mut producer) = context_producer(tmp.path()); - assert!( - producer - .observe(&token_usage_frame(92_283, window.clone()), "thread-main") - .unwrap() - ); - let observed = context_record(&agent_dir).unwrap(); - assert_eq!(observed.used_tokens, Some(92_283), "window {window}"); - assert_eq!(observed.window_tokens, expected_window, "window {window}"); - assert_eq!(observed.used_percent, None, "window {window}"); - } - - // A window key that is absent rather than null reads the same way. - let tmp = tempfile::tempdir().unwrap(); - let (agent_dir, mut producer) = context_producer(tmp.path()); - assert!( - producer - .observe( - &json!({ - "method": "thread/tokenUsage/updated", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "tokenUsage": { - "last": { "totalTokens": 92_283 }, - "total": { "totalTokens": 92_283 } - } - } - }), - "thread-main", - ) - .unwrap() - ); - let observed = context_record(&agent_dir).unwrap(); - assert_eq!(observed.window_tokens, None); - assert_eq!(observed.used_percent, None); - } - - /// Codex speaks once per model response — roughly 10-15 times a turn, and again on resume or - /// re-attach. The core's quantization is the ONLY thing deciding what lands (HC-R09): this - /// producer holds no reading of its own and imposes no cadence. The shape that catches a second - /// guard is the last frame here — a bucket crossing arriving immediately after two suppressed - /// readings, which any time floor in the producer would swallow. - #[test] - fn every_reading_reaches_the_core_guard_and_the_producer_imposes_no_cadence_of_its_own() { - let tmp = tempfile::tempdir().unwrap(); - let (agent_dir, mut producer) = context_producer(tmp.path()); - let window = json!(258_400); - - assert!( - producer - .observe(&token_usage_frame(92_283, window.clone()), "thread-main") - .unwrap() - ); - assert_eq!(context_record(&agent_dir).unwrap().used_percent, Some(33.0)); - - // Both still round to 33% used, so both sit in the written bucket and neither lands. - for moved in [93_000, 94_000] { - assert_eq!(codex_used_percent(Some(258_400), moved), Some(33.0)); - assert!( - !producer - .observe(&token_usage_frame(moved, window.clone()), "thread-main") - .unwrap() - ); - assert_eq!( - context_record(&agent_dir).unwrap().used_tokens, - Some(92_283) - ); - } - - // The crossing lands at once, with no elapsed time behind it. - assert!( - producer - .observe(&token_usage_frame(95_000, window.clone()), "thread-main") - .unwrap() - ); - let observed = context_record(&agent_dir).unwrap(); - assert_eq!(observed.used_percent, Some(34.0)); - assert_eq!(observed.used_tokens, Some(95_000)); - - // A reading for another thread is not this seat's. - assert!( - !producer - .observe(&token_usage_frame(200_000, window), "thread-other") - .unwrap() - ); - assert_eq!( - context_record(&agent_dir).unwrap().used_tokens, - Some(95_000) - ); - } - - /// HC-R12: one compaction is one count, however many of its spellings arrive. Codex publishes - /// the live edge as an `item/started` AND an `item/completed` over the same - /// `ContextCompactionThreadItem` id, and the protocol still carries a deprecated - /// `thread/compacted` notification for the same event that names only the turn. - #[test] - fn one_compaction_is_counted_once_across_every_spelling_of_its_edge() { - let tmp = tempfile::tempdir().unwrap(); - let (agent_dir, mut producer) = context_producer(tmp.path()); - - assert!( - producer - .observe( - &compaction_item_frame("item/started", "turn-1", "item-a"), - "thread-main" - ) - .unwrap() - ); - let first = context_record(&agent_dir).unwrap(); - assert_eq!(first.compactions, 1); - assert_eq!( - first.last_compaction_trigger, - Some(harness_context::CompactionTrigger::Unknown), - "the item carries an id and a type and no reason at all" - ); - assert!(first.last_compaction_ms.is_some()); - - // The same compaction's closing edge, and the deprecated notification for the same event. - assert!( - !producer - .observe( - &compaction_item_frame("item/completed", "turn-1", "item-a"), - "thread-main" - ) - .unwrap() - ); - assert!( - !producer - .observe( - &json!({ - "method": "thread/compacted", - "params": { "threadId": "thread-main", "turnId": "turn-1" } - }), - "thread-main", - ) - .unwrap() - ); - assert_eq!(context_record(&agent_dir).unwrap().compactions, 1); - - // A genuinely second compaction inside the same turn is a second count. - assert!( - producer - .observe( - &compaction_item_frame("item/started", "turn-1", "item-b"), - "thread-main" - ) - .unwrap() - ); - assert_eq!(context_record(&agent_dir).unwrap().compactions, 2); - - // Interleaved lifecycles: two starts before either completion still count exactly two, so - // the dedupe cannot be a single last-key memory. - for (method, item) in [ - ("item/started", "item-c"), - ("item/started", "item-d"), - ("item/completed", "item-c"), - ("item/completed", "item-d"), - ] { - producer - .observe( - &compaction_item_frame(method, "turn-2", item), - "thread-main", - ) - .unwrap(); - } - assert_eq!(context_record(&agent_dir).unwrap().compactions, 4); - - // The deprecated notification arriving FIRST also claims the compaction, so the item that - // follows it does not count a second time. - assert!( - producer - .observe( - &json!({ - "method": "thread/compacted", - "params": { "threadId": "thread-main", "turnId": "turn-3" } - }), - "thread-main", - ) - .unwrap() - ); - assert!( - !producer - .observe( - &compaction_item_frame("item/started", "turn-3", "item-e"), - "thread-main" - ) - .unwrap() - ); - assert_eq!(context_record(&agent_dir).unwrap().compactions, 5); - - // Another thread's compaction is not this seat's, and a non-compaction item is not an edge. - assert!( - !producer - .observe( - &compaction_item_frame("item/started", "turn-9", "item-z"), - "thread-other" - ) - .unwrap() - ); - assert!( - !producer - .observe( - &json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-4", - "item": { "id": "item-y", "type": "agentMessage" } - } - }), - "thread-main", - ) - .unwrap() - ); - assert_eq!(context_record(&agent_dir).unwrap().compactions, 5); - } - - /// The producer runs beside a live delivery loop and sees every frame that loop sees. Replaying - /// the captured #263 session — 23 real inbound frames, none of them a token count — must leave - /// no record at all: absence here is "never observed", and a producer that manufactured a - /// reading from a turn boundary would break exactly the HC-R03 rule the record exists for. - #[test] - fn captured_delivery_frames_carrying_no_token_count_publish_no_record() { - let frames = include_str!("../tests/fixtures/codex_usage_limit_inbound.jsonl") - .lines() - .map(|line| serde_json::from_str::(line).unwrap()) - .collect::>(); - assert_eq!(frames.len(), 23); - - let tmp = tempfile::tempdir().unwrap(); - let (agent_dir, mut producer) = context_producer(tmp.path()); - for frame in &frames { - assert!( - !producer.observe(frame, "thread-main").unwrap(), - "no captured delivery frame carries a context reading: {frame}" - ); - } - assert!(context_record(&agent_dir).is_none()); - } - - fn delivery_config(root: &Path) -> CodexDeliveryConfig { - let agent_dir = root.join("agents/h/worker"); - CodexDeliveryConfig { - catalog_root: root.to_path_buf(), - inbox: message::inbox_dir(&agent_dir), - agent_dir, - identity: "h.worker".into(), - this_host: "h".into(), - supervisor: None, - producer_version: Some("codex-cli 0.153.0".into()), - } - } - - fn subscribed_state(observed: CodexObservedState) -> CodexControlState { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state.subscribed = true; - state.observed = observed; - state - } - - fn inbox_delivery(root: &Path, config: CodexDeliveryConfig) -> CodexInboxDelivery { - CodexInboxDelivery::new( - config, - root.join("state").join(delivery_ledger::LEDGER_FILE), - CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(), - ) - .unwrap() - } - - /// Read the ledger back through its own loader and the real correlation derivation: a test - /// that read the bytes directly would not notice a record the pump itself would refuse. - fn ledger_entry(root: &Path, filename: &str) -> Option { - delivery_ledger::Ledger::open( - &root.join("state").join(delivery_ledger::LEDGER_FILE), - delivery_ledger::Harness::Codex.profile(), - "h.worker", - "h.worker", - |thread, file| stable_client_user_message_id("h.worker", thread, file), - ) - .entry(filename) - .cloned() - } - - fn acknowledge_tui_thread_loaded(events: &Receiver) { - let ControlEvent::TuiThreadLoaded(acknowledge) = - events.recv_timeout(Duration::from_secs(10)).unwrap() - else { - panic!("control did not report the TUI-loaded gate"); - }; - acknowledge.send(()).unwrap(); - } - - #[test] - fn delivery_request_uses_typed_start_and_exact_turn_steer() { - let start = codex_delivery_request( - 2, - "thread-main", - "st2:client", - "notice", - &CodexDeliveryMethod::Start, - ); - assert_eq!(start["method"], "turn/start"); - assert_eq!(start["params"]["threadId"], "thread-main"); - assert_eq!(start["params"]["clientUserMessageId"], "st2:client"); - assert_eq!(start["params"]["input"][0]["type"], "text"); - assert_eq!(start["params"]["input"][0]["text"], "notice"); - assert!(start["params"].get("expectedTurnId").is_none()); - - let steer = codex_delivery_request( - 3, - "thread-main", - "st2:client", - "notice", - &CodexDeliveryMethod::Steer { - turn_id: "turn-current".into(), - }, - ); - assert_eq!(steer["method"], "turn/steer"); - assert_eq!(steer["params"]["expectedTurnId"], "turn-current"); - assert!(steer["params"].get("model").is_none()); - assert!(steer["params"].get("approvalPolicy").is_none()); - } - - /// Behavioral oracle for the #268 §B projection: a projection that withheld every row — or - /// that reported the two misclassified rows as indeterminate — fails here, because each - /// emitting row is asserted positively. - #[test] - fn harness_projection_is_faithful_and_withholds_only_unprovable_rows() { - use crate::harness_state::{Activity, Ask, BlockedOn, InputBuffer}; - let held = |reason| CodexObservedState::Held { - reason, - turn_id: None, - }; - - // Rows with no provable observation are withheld — and no absence may derive idle. - for state in [ - CodexObservedState::AwaitingStatus, - held(CodexHoldReason::NotLoaded), - held(CodexHoldReason::SystemError), - ] { - assert_eq!(state.harness_observation(), None, "{state:?}"); - } - - // Codex positively reported work: active, even where st2 cannot name a steerable turn - // (the two rows a naive steerability decomposition reported as unknown) or where the - // delivery gate holds. - for state in [ - CodexObservedState::Active { - turn_id: "turn-current".into(), - }, - held(CodexHoldReason::ActiveWithoutTurn), - held(CodexHoldReason::ConflictingTurn), - held(CodexHoldReason::Compaction), - // Review's edges are model-emitted items inside a running turn: plain activity, - // no human, no ask — the delivery hold is a separate axis. - held(CodexHoldReason::Review), - ] { - let observation = state - .harness_observation() - .unwrap_or_else(|| panic!("{state:?} must emit")); - assert_eq!(observation.state, Activity::Active, "{state:?}"); - assert_eq!(observation.blocked_on, BlockedOn::None, "{state:?}"); - assert_eq!(observation.input_buffer, InputBuffer::Unknown, "{state:?}"); - } - - // The holds a human resolves set the blocked axis instead of disappearing into active, - // and each names its machine-readable ask kind so consumers never branch on `reason`. - for (reason, ask) in [ - (CodexHoldReason::WaitingOnApproval, Ask::Permission), - (CodexHoldReason::WaitingOnUserInput, Ask::Question), - ] { - let observation = held(reason) - .harness_observation() - .unwrap_or_else(|| panic!("{reason:?} must emit")); - assert_eq!(observation.state, Activity::Active, "{reason:?}"); - assert_eq!(observation.blocked_on, BlockedOn::Human, "{reason:?}"); - assert_eq!(observation.ask, ask, "{reason:?}"); - } - - let idle = CodexObservedState::Idle.harness_observation().unwrap(); - assert_eq!(idle.state, Activity::Idle); - assert_eq!(idle.blocked_on, BlockedOn::None); - - let ended = CodexObservedState::TerminalError { - reason: CodexTerminalError::SystemError, - } - .harness_observation() - .unwrap(); - assert_eq!(ended.state, Activity::Ended); - assert_eq!(ended.reason.as_deref(), Some("systemError")); - } - - #[test] - #[cfg(unix)] - fn a_failed_transition_write_is_retried_before_any_heartbeat() { - use crate::harness_state::{self, Activity}; - use std::os::unix::fs::PermissionsExt as _; - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let agent_dir = config.agent_dir.clone(); - let record_path = harness_state::harness_state_path(&agent_dir); - let mut delivery = inbox_delivery(tmp.path(), config); - - delivery.observe_harness(&CodexObservedState::Active { - turn_id: "turn-current".into(), - }); - assert_eq!( - harness_state::read(&record_path, None).unwrap().state, - Activity::Active - ); - - // The transition to idle fails to land: the agent dir is briefly unwritable. - let live = fs::metadata(&agent_dir).unwrap().permissions(); - fs::set_permissions(&agent_dir, fs::Permissions::from_mode(0o555)).unwrap(); - delivery.observe_harness(&CodexObservedState::Idle); - fs::set_permissions(&agent_dir, live).unwrap(); - assert_eq!( - harness_state::read(&record_path, None).unwrap().state, - Activity::Active, - "the failed write cannot have landed" - ); - - // No heartbeat may re-stamp the contradicted on-disk state; the retry lands the pending - // transition on the NEXT pump pass — deliberately without advancing the presence - // cadence, which gates only heartbeats. - let stale_active = fs::read(&record_path).unwrap(); - delivery.next_presence_refresh = Instant::now() + status::STATUS_REFRESH; - delivery.refresh_if_due().unwrap(); - let after = harness_state::read(&record_path, None).unwrap(); - assert_eq!(after.state, Activity::Idle, "pending transition retried"); - assert_ne!(fs::read(&record_path).unwrap(), stale_active); - } - - #[test] - fn pump_publishes_observations_and_stops_heartbeating_on_evidence_loss() { - use crate::harness_state::{self, Activity}; - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let agent_dir = config.agent_dir.clone(); - let record_path = harness_state::harness_state_path(&agent_dir); - let mut delivery = inbox_delivery(tmp.path(), config); - - delivery.observe_harness(&CodexObservedState::Active { - turn_id: "turn-current".into(), - }); - let observed = harness_state::read(&record_path, None).expect("record written"); - assert_eq!(observed.state, Activity::Active); - assert_eq!(observed.harness.as_deref(), Some("codex")); - - // An indeterminate projection writes nothing and stops the heartbeat: the presence - // refresh still runs, but the record's bytes stay untouched and age toward unknown. - delivery.observe_harness(&CodexObservedState::Held { - reason: CodexHoldReason::NotLoaded, - turn_id: None, - }); - let before = fs::read(&record_path).unwrap(); - delivery.refresh_if_due().unwrap(); - assert!( - status::read_state(&status::status_path(&agent_dir)) != status::State::Offline, - "presence refresh must still run" - ); - assert_eq!( - fs::read(&record_path).unwrap(), - before, - "no heartbeat without evidence" - ); - - // Evidence returning resumes both observation and heartbeat. - delivery.observe_harness(&CodexObservedState::Idle); - assert_eq!( - harness_state::read(&record_path, None).unwrap().state, - Activity::Idle - ); - delivery.next_presence_refresh = Instant::now(); - delivery.refresh_if_due().unwrap(); - assert_ne!( - fs::read(&record_path).unwrap(), - before, - "heartbeat resumes with evidence" - ); - } - - #[test] - fn evidence_loss_marks_the_stream_discontinuous_for_a_restated_state() { - use crate::harness_state::{self, Activity}; - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let record_path = harness_state::harness_state_path(&config.agent_dir); - let mut delivery = inbox_delivery(tmp.path(), config); - - delivery.observe_harness(&CodexObservedState::Active { - turn_id: "turn-a".into(), - }); - let before = fs::read(&record_path).unwrap(); - - // The same tuple restated across an unproven interval must not coalesce into the - // pre-gap record — continuity was not observed, so a fresh transition opens. - delivery.observe_harness(&CodexObservedState::Held { - reason: CodexHoldReason::SystemError, - turn_id: None, - }); - delivery.observe_harness(&CodexObservedState::Active { - turn_id: "turn-a".into(), - }); - assert_ne!( - fs::read(&record_path).unwrap(), - before, - "a restated state after an evidence gap must open a fresh transition" - ); - assert_eq!( - harness_state::read(&record_path, None).unwrap().state, - Activity::Active - ); - } - - #[test] - fn delivery_client_id_is_stable_and_binds_every_identity_component() { - let id = - stable_client_user_message_id("h.worker", "thread-main", "1786380000000-abc123.md"); - assert_eq!( - id, - stable_client_user_message_id("h.worker", "thread-main", "1786380000000-abc123.md") - ); - assert!(id.starts_with("st2:")); - assert_ne!( - id, - stable_client_user_message_id("h.other", "thread-main", "1786380000000-abc123.md") - ); - assert_ne!( - id, - stable_client_user_message_id("h.worker", "thread-other", "1786380000000-abc123.md") - ); - assert_ne!( - id, - stable_client_user_message_id("h.worker", "thread-main", "1786380000000-def456.md") - ); - } - - #[test] - fn review_compaction_and_dnd_hold_the_unread_fifo_head() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = - message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - for reason in [CodexHoldReason::Review, CodexHoldReason::Compaction] { - let state = subscribed_state(CodexObservedState::Held { - reason, - turn_id: Some("turn-current".into()), - }); - assert_eq!(delivery.maybe_request(&state).unwrap(), None); - assert!(config.inbox.join(&filename).is_file()); - } - - status::set_state(&status::status_path(&config.agent_dir), status::State::Dnd).unwrap(); - delivery.next_inbox_refresh = Instant::now(); - assert_eq!( - delivery - .maybe_request(&subscribed_state(CodexObservedState::Idle)) - .unwrap(), - None - ); - assert_eq!(message::list_inbox(&config.inbox).unwrap().len(), 1); - } - - #[test] - fn failed_turn_without_idle_allows_next_native_delivery_and_preserves_system_error() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - message::send_to_inbox( - &config.inbox, - "h.sender", - Some("after error"), - None, - &[], - "body", - ) - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config); - let mut state = subscribed_state(CodexObservedState::Idle); - - state - .observe(&json!({ - "method": "turn/started", - "params": { - "threadId": "thread-main", - "turn": { "id": "turn-failed" } - } - })) - .unwrap(); - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "systemError" } - } - })) - .unwrap(); - state - .observe(&json!({ - "method": "turn/completed", - "params": { - "threadId": "thread-main", - "turn": { "id": "turn-failed", "status": "failed" } - } - })) - .unwrap(); - - assert_eq!( - state.observed(), - &CodexObservedState::TerminalError { - reason: CodexTerminalError::SystemError, - } - ); - - let request = delivery - .maybe_request(&state) - .unwrap() - .expect("a terminal system error must not block the next native delivery"); - assert_eq!(request["method"], "turn/start"); - } - - #[test] - fn captured_usage_limit_boundary_allows_next_native_delivery() { - // This fixture is a payload-minimized projection of all 23 inbound frames from the - // #263 trivial capture. It preserves their order and methods while removing fields this - // observer never reads. The second capture has the same method sequence. The recorder - // stops at turn completion, so this test pins the boundary state only. The provider - // source establishes that no later idle notification follows the system error. - let frames = include_str!("../tests/fixtures/codex_usage_limit_inbound.jsonl") - .lines() - .map(|line| serde_json::from_str::(line).unwrap()) - .collect::>(); - assert_eq!(frames.len(), 23); - - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - message::send_to_inbox( - &config.inbox, - "h.sender", - Some("after capture"), - None, - &[], - "body", - ) - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config); - let mut state = subscribed_state(CodexObservedState::AwaitingStatus); - - for frame in &frames { - state.observe(frame).unwrap(); - } - - assert_eq!( - frames - .last() - .and_then(|frame| frame.get("method")) - .and_then(Value::as_str), - Some("turn/completed") - ); - assert_eq!( - state.observed(), - &CodexObservedState::TerminalError { - reason: CodexTerminalError::SystemError, - } - ); - let request = delivery - .maybe_request(&state) - .unwrap() - .expect("a captured terminal system error must permit the next native delivery"); - assert_eq!(request["method"], "turn/start"); - } - - /// The credential class and the quota class arrive through the SAME frame sequence, differing - /// only in one word of `Turn.error.codexErrorInfo`. This replays the auth-rejected shape and - /// asserts the fork: `providerAuth` on the observed record, a native-driver diagnostic, and - /// delivery still permitted — while the captured usage-limit fixture beside it keeps reading - /// `systemError` with no diagnostic at all. - #[test] - fn a_rejected_codex_credential_reads_provider_auth_while_a_quota_failure_does_not() { - let rejected = include_str!("../tests/fixtures/codex_provider_auth_inbound.jsonl") - .lines() - .map(|line| serde_json::from_str::(line).unwrap()) - .collect::>(); - let quota = include_str!("../tests/fixtures/codex_usage_limit_inbound.jsonl") - .lines() - .map(|line| serde_json::from_str::(line).unwrap()) - .collect::>(); - - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let agent_dir = config.agent_dir.clone(); - message::send_to_inbox( - &config.inbox, - "h.sender", - Some("after rejection"), - None, - &[], - "body", - ) - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config); - let mut state = subscribed_state(CodexObservedState::AwaitingStatus); - for frame in &rejected { - state.observe(frame).unwrap(); - delivery.observe_provider_auth(frame, "thread-main"); - } - - assert_eq!( - state.observed(), - &CodexObservedState::TerminalError { - reason: CodexTerminalError::ProviderAuthRejected, - } - ); - let observation = state.observed().harness_observation().unwrap(); - assert_eq!(observation.state, harness_state::Activity::Ended); - assert_eq!( - observation.reason.as_deref(), - Some("providerAuth"), - "the same word OpenCode's ProviderAuthError already publishes" - ); - let record = driver_diagnostic::path(&agent_dir); - let driver_diagnostic::Observed::Failure(failure) = driver_diagnostic::read(&record) else { - panic!("a rejected credential must publish a native-driver diagnostic") - }; - assert_eq!(failure.driver, driver_diagnostic::Driver::Codex); - assert_eq!(failure.stage, driver_diagnostic::Stage::ProviderAuth); - assert_eq!( - failure.reason, - driver_diagnostic::Reason::ProviderAuthRejected - ); - assert_eq!(failure.source, driver_diagnostic::Source::TurnResult); - assert_eq!( - failure.producer_version.as_deref(), - Some("codex-cli 0.153.0") - ); - assert_eq!(failure.support, driver_diagnostic::Support::Supported); - let request = delivery - .maybe_request(&state) - .unwrap() - .expect("a rejected credential must not block the next native delivery"); - assert_eq!(request["method"], "turn/start"); - - // A turn that reaches its ordinary end is the recovery edge. - delivery.observe_provider_auth( - &json!({ - "method": "turn/completed", - "params": { - "threadId": "thread-main", - "turn": { "id": "turn-ok", "status": "completed" } - } - }), - "thread-main", - ); - assert_eq!( - driver_diagnostic::read(&record), - driver_diagnostic::Observed::Absent - ); - - // The quota capture walks the same methods and must stay unclassified. - let quota_tmp = tempfile::tempdir().unwrap(); - let quota_config = delivery_config(quota_tmp.path()); - let quota_agent_dir = quota_config.agent_dir.clone(); - let mut quota_delivery = inbox_delivery(quota_tmp.path(), quota_config); - let mut quota_state = subscribed_state(CodexObservedState::AwaitingStatus); - for frame in "a { - quota_state.observe(frame).unwrap(); - quota_delivery.observe_provider_auth(frame, "thread-main"); - } - assert_eq!( - quota_state.observed(), - &CodexObservedState::TerminalError { - reason: CodexTerminalError::SystemError, - } - ); - assert_eq!( - driver_diagnostic::read(&driver_diagnostic::path("a_agent_dir)), - driver_diagnostic::Observed::Absent, - "an exhausted allowance is not a rejected credential" - ); - } - - #[test] - fn idle_session_refreshes_stale_presence_without_inbox_activity() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let presence = status::status_path(&config.agent_dir); - std::fs::create_dir_all(&config.agent_dir).unwrap(); - std::fs::write(&presence, "available\n").unwrap(); - std::fs::File::open(&presence) - .unwrap() - .set_modified(SystemTime::now() - status::STATUS_STALE - Duration::from_secs(1)) - .unwrap(); - assert_eq!(status::read_state(&presence), status::State::Unknown); - - let mut delivery = inbox_delivery(tmp.path(), config); - delivery.refresh_if_due().unwrap(); - - assert_eq!(status::read_state(&presence), status::State::Available); - assert!( - std::fs::read_to_string(&presence) - .unwrap() - .contains("\nv1 ") - ); - assert!(delivery.head.is_none()); - } - - #[test] - fn inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let presence = status::status_path(&config.agent_dir); - status::set_state(&presence, status::State::Available).unwrap(); - let before = std::fs::read_to_string(&presence).unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config); - delivery.next_inbox_refresh = Instant::now(); - delivery.next_presence_refresh = Instant::now() + status::STATUS_REFRESH; - - delivery.refresh_if_due().unwrap(); - - assert_eq!(std::fs::read_to_string(&presence).unwrap(), before); - } - - #[test] - fn a_rejected_exact_steer_has_no_fallback_and_remains_retryable_after_state_changes() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = - message::send_to_inbox(&config.inbox, "h.sender", Some("retry"), None, &[], "body") - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - let active = subscribed_state(CodexObservedState::Active { - turn_id: "turn-current".into(), - }); - let steer = delivery.maybe_request(&active).unwrap().unwrap(); - assert_eq!(steer["method"], "turn/steer"); - assert_eq!(steer["params"]["expectedTurnId"], "turn-current"); - let request_id = steer["id"].clone(); - let client_id = steer["params"]["clientUserMessageId"].clone(); - - assert!( - !delivery - .accept_response( - &json!({ - "id": request_id, - "method": "item/commandExecution/requestApproval", - "params": {} - }), - active.observed(), - ) - .unwrap() - ); - assert!(delivery - .accept_response( - &json!({ "id": request_id, "error": { "code": -32600, "message": "stale turn" } }), - active.observed(), - ) - .unwrap()); - assert_eq!(delivery.maybe_request(&active).unwrap(), None); - assert!(config.inbox.join(&filename).is_file()); - - let retry = delivery - .maybe_request(&subscribed_state(CodexObservedState::Idle)) - .unwrap() - .unwrap(); - assert_eq!(retry["method"], "turn/start"); - assert_eq!(retry["params"]["clientUserMessageId"], client_id); - assert!(config.inbox.join(&filename).is_file()); - } - - #[test] - fn a_success_response_is_only_an_attempt_and_does_not_archive_the_message() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = message::send_to_inbox( - &config.inbox, - "h.sender", - Some("submitted"), - None, - &[], - "body", - ) - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - let idle = subscribed_state(CodexObservedState::Idle); - let request = delivery.maybe_request(&idle).unwrap().unwrap(); - assert_eq!( - delivery.ledger.entry(&filename).unwrap().phase, - delivery_ledger::Phase::Attempted, - "submission ownership is durable before transport" - ); - assert!( - delivery - .accept_response( - &json!({ "id": request["id"], "result": { "turn": { "id": "turn-new" } } }), - idle.observed(), - ) - .unwrap() - ); - assert_eq!( - delivery.ledger.entry(&filename).unwrap().phase, - delivery_ledger::Phase::TransportAccepted, - "a well-formed JSON result is transport, never typed acceptance" - ); - assert_eq!(delivery.maybe_request(&idle).unwrap(), None); - assert!(config.inbox.join(&filename).is_file()); - } - - #[test] - fn only_a_completed_matching_user_message_persists_acceptance() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = message::send_to_inbox( - &config.inbox, - "h.sender", - Some("receipt"), - None, - &[], - "body", - ) - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - let mut idle = CodexControlState::new(&delivery.runtime, "thread-main".into()); - idle.subscribed = true; - idle.observed = CodexObservedState::Idle; - let request = delivery.maybe_request(&idle).unwrap().unwrap(); - let client_id = request["params"]["clientUserMessageId"] - .as_str() - .unwrap() - .to_string(); - - assert!( - !delivery - .accept_typed_receipt( - &json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-delivery", - "item": { "type": "userMessage", "clientId": client_id } - } - }), - &idle, - ) - .unwrap(), - "item/started is progress, not acceptance" - ); - assert!( - !delivery - .accept_typed_receipt( - &json!({ - "method": "item/completed", - "params": { - "threadId": "thread-other", - "turnId": "turn-delivery", - "item": { "type": "userMessage", "clientId": client_id } - } - }), - &idle, - ) - .unwrap(), - "another thread cannot acknowledge this delivery" - ); - assert!( - delivery - .accept_typed_receipt( - &json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-delivery", - "item": { "type": "userMessage", "clientId": client_id } - } - }), - &idle, - ) - .unwrap() - ); - assert_eq!( - ledger_entry(tmp.path(), &filename).unwrap().phase, - delivery_ledger::Phase::Consumed - ); - assert!(config.inbox.join(&filename).is_file()); - - drop(delivery); - let mut replacement = inbox_delivery(tmp.path(), config.clone()); - assert_eq!( - replacement.maybe_request(&idle).unwrap(), - None, - "a fresh runtime incarnation restores accepted duplicate control" - ); - - message::archive_msg( - &config.inbox, - &message::archive_dir(&config.agent_dir), - &filename, - ) - .unwrap(); - replacement.next_inbox_refresh = Instant::now(); - assert_eq!(replacement.maybe_request(&idle).unwrap(), None); - assert!( - ledger_entry(tmp.path(), &filename).is_none(), - "archive precedence — the recipient agent's own act — releases the ledger entry" - ); - } - - #[test] - fn an_ambiguous_attempt_reconciles_resume_history_before_retry() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = message::send_to_inbox( - &config.inbox, - "h.sender", - Some("reconcile"), - None, - &[], - "body", - ) - .unwrap(); - let idle = subscribed_state(CodexObservedState::Idle); - let mut first = inbox_delivery(tmp.path(), config.clone()); - let request = first.maybe_request(&idle).unwrap().unwrap(); - let client_id = request["params"]["clientUserMessageId"] - .as_str() - .unwrap() - .to_string(); - drop(first); - - let mut recovered = inbox_delivery(tmp.path(), config.clone()); - assert_eq!(recovered.maybe_request(&idle).unwrap(), None); - recovered - .reconcile_resume( - &json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "result": { - "thread": { - "id": "thread-main", - "turns": [{ - "id": "turn-delivery", - "items": [{ - "type": "userMessage", - "id": "item-delivery", - "clientId": client_id, - "content": [] - }] - }] - } - } - }), - &idle, - ) - .unwrap(); - assert_eq!( - recovered.ledger.entry(&filename).unwrap().phase, - delivery_ledger::Phase::Consumed, - "a resumed history carrying the client ID is the same typed receipt, found late" - ); - assert_eq!(recovered.maybe_request(&idle).unwrap(), None); - assert!(config.inbox.join(&filename).is_file()); - - // An authoritative resumed history WITHOUT the client ID proves the pre-crash attempt - // never landed. Only that absence may re-authorize the same stable ID — so it needs its - // own scenario, because the delivery above is settled and can never be un-settled. - let absent_tmp = tempfile::tempdir().unwrap(); - let absent_config = delivery_config(absent_tmp.path()); - let absent_filename = message::send_to_inbox( - &absent_config.inbox, - "h.sender", - Some("absent"), - None, - &[], - "body", - ) - .unwrap(); - let mut attempted = inbox_delivery(absent_tmp.path(), absent_config.clone()); - let absent_client_id = attempted.maybe_request(&idle).unwrap().unwrap() - ["params"]["clientUserMessageId"] - .as_str() - .unwrap() - .to_string(); - drop(attempted); - - let mut replacement = inbox_delivery(absent_tmp.path(), absent_config); - assert_eq!( - replacement.maybe_request(&idle).unwrap(), - None, - "an ambiguous attempt is held and surfaced, never replayed on its own" - ); - replacement - .reconcile_resume( - &json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "result": { "thread": { "id": "thread-main", "turns": [] } } - }), - &idle, - ) - .unwrap(); - assert_eq!( - replacement - .ledger - .entry(&absent_filename) - .unwrap() - .negative, - Some(delivery_ledger::NegativeReceipt::Absent), - "the absence is retained as evidence, not erased" - ); - let retry = replacement.maybe_request(&idle).unwrap().unwrap(); - assert_eq!(retry["params"]["clientUserMessageId"], absent_client_id); - } - - - #[test] - fn subscribed_control_pump_delivers_a_typed_reference_to_the_real_fifo_head() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let config = delivery_config(tmp.path()); - let filename = - message::send_to_inbox(&config.inbox, "h.sender", Some("wired"), None, &[], "body") - .unwrap(); - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let server_filename = filename.clone(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - stream - // Parallel Darwin test runs can deschedule the in-process peer - // for longer than the Linux-oriented two-second budget. - .set_read_timeout(Some(Duration::from_secs(10))) - .unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialize" - ); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialized" - ); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/started", - "params": { "thread": { "id": "thread-main", "status": { "type": "idle" } } } - }), - ) - .unwrap(); - let delivery = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(delivery["id"], FIRST_DELIVERY_REQUEST_ID); - assert_eq!(delivery["method"], "turn/start"); - assert_eq!(delivery["params"]["threadId"], "thread-main"); - let head_id = server_filename - .trim_end_matches(".md") - .rsplit_once('-') - .unwrap() - .1; - assert!( - delivery["params"]["input"][0]["text"] - .as_str() - .unwrap() - .contains(head_id), - "the transport payload must identify the actionable FIFO head" - ); - assert_eq!( - delivery["params"]["clientUserMessageId"], - stable_client_user_message_id("h.worker", "thread-main", &server_filename) - ); - let client_id = delivery["params"]["clientUserMessageId"] - .as_str() - .unwrap() - .to_string(); - write_json_message( - &mut websocket, - &json!({ - "id": FIRST_DELIVERY_REQUEST_ID, - "result": { "turn": { "id": "turn-delivery" } } - }), - ) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-delivery", - "item": { - "type": "userMessage", - "id": "item-delivery", - "clientId": client_id, - "content": [] - } - } - }), - ) - .unwrap(); - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let binding_path = tmp.path().join("state/binding.json"); - let control_state_path = tmp.path().join("state/control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let runtime_for_pump = runtime.clone(); - let binding_for_pump = binding_path.clone(); - let control_state_for_pump = control_state_path.clone(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_for_pump, - &control_state_for_pump, - &runtime_for_pump, - None, - Some(config), - tx, - ) - }); - assert!(matches!( - rx.recv_timeout(Duration::from_secs(10)).unwrap(), - ControlEvent::Bound - )); - server.join().unwrap(); - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - assert!(delivery_config(tmp.path()).inbox.join(&filename).is_file()); - assert_eq!( - ledger_entry(tmp.path(), &filename).unwrap().phase, - delivery_ledger::Phase::Consumed - ); - } - - /// The wiring, not the arithmetic: a `thread/tokenUsage/updated` arriving on the real control - /// socket reaches the record. Every other context test drives the producer directly, so all of - /// them would stay green if the pump stopped handing it frames — which is exactly how a - /// producer silently stops producing. - #[test] - fn the_control_pump_publishes_a_context_reading_from_a_live_token_usage_notification() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let config = delivery_config(tmp.path()); - let agent_dir = config.agent_dir.clone(); - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - stream - .set_read_timeout(Some(Duration::from_secs(10))) - .unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialize" - ); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialized" - ); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/started", - "params": { "thread": { "id": "thread-main", "status": { "type": "idle" } } } - }), - ) - .unwrap(); - write_json_message(&mut websocket, &token_usage_frame(92_283, json!(258_400))).unwrap(); - // Hold the connection open until the reading has landed: closing here would race the - // pump's read of the frame just written. Bounded, so a pump that stopped handing - // frames to the producer fails this test instead of hanging it. - let deadline = Instant::now() + Duration::from_secs(10); - while harness_context::read(&harness_context::harness_context_path(&agent_dir)) - .is_none() - && Instant::now() < deadline - { - std::thread::sleep(Duration::from_millis(10)); - } - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let binding_path = tmp.path().join("state/binding.json"); - let control_state_path = tmp.path().join("state/control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let binding_for_pump = binding_path.clone(); - let control_state_for_pump = control_state_path.clone(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_for_pump, - &control_state_for_pump, - &runtime, - None, - Some(config), - tx, - ) - }); - assert!(matches!( - rx.recv_timeout(Duration::from_secs(10)).unwrap(), - ControlEvent::Bound - )); - server.join().unwrap(); - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - - let observed = context_record(&tmp.path().join("agents/h/worker")) - .expect("the pump published nothing"); - assert_eq!(observed.harness, harness_context::Harness::Codex); - assert_eq!(observed.used_tokens, Some(92_283)); - assert_eq!(observed.window_tokens, Some(258_400)); - assert_eq!(observed.used_percent, Some(33.0)); - } - - - #[test] - fn control_initializes_before_recording_the_first_thread_only() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - let initialize = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(initialize["method"], "initialize"); - assert_eq!(initialize["params"]["clientInfo"]["name"], "st2"); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - let initialized = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(initialized["method"], "initialized"); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/started", - "params": { "thread": { "id": "thread-main", "status": { "type": "idle" } } } - }), - ) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/status/changed", - "params": { "threadId": "thread-main", "status": { "type": "idle" } } - }), - ) - .unwrap(); - // JSON-RPC request IDs are per direction. A server request may reuse the client's - // subscription ID and must not be consumed as a client response. - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "method": "item/commandExecution/requestApproval", - "params": {} - }), - ) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/started", - "params": { "thread": { "id": "thread-review", "status": { "type": "idle" } } } - }), - ) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-main" } } - }), - ) - .unwrap(); - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let state = tmp.path().join("state"); - let binding_path = state.join("binding.json"); - let control_state_path = state.join("control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let runtime_for_pump = runtime.clone(); - let binding_for_pump = binding_path.clone(); - let control_state_for_pump = control_state_path.clone(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_for_pump, - &control_state_for_pump, - &runtime_for_pump, - None, - None, - tx, - ) - }); - let first_event = rx.recv_timeout(Duration::from_secs(2)).unwrap(); - assert!( - matches!(first_event, ControlEvent::Bound), - "first control event: {first_event:?}" - ); - server.join().unwrap(); - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - - let binding = load_current_binding(&binding_path, &runtime) - .unwrap() - .unwrap(); - assert_eq!(binding.thread_id(), "thread-main"); - let state = - load_current_control_state(&state.join("control-state.json"), &runtime, &binding) - .unwrap() - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-main".into() - } - ); - assert!(state.subscribed()); - } - - #[test] - fn expected_resume_waits_for_tui_loaded_thread_and_binds_from_control_response() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let (pre_gate_checked_tx, pre_gate_checked_rx) = mpsc::channel(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - stream - .set_read_timeout(Some(Duration::from_millis(100))) - .unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - let initialize = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(initialize["method"], "initialize"); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - let initialized = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(initialized["method"], "initialized"); - assert!(matches!( - poll_json_message(&mut websocket).unwrap(), - ControlRead::Timeout - )); - pre_gate_checked_tx.send(()).unwrap(); - websocket - .get_mut() - .set_read_timeout(Some(Duration::from_millis(500))) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/started", - "params": { - "thread": { "id": "thread-unrelated", "status": { "type": "idle" } } - } - }), - ) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-unrelated", - "status": { "type": "active", "activeFlags": [] } - } - }), - ) - .unwrap(); - let first_loaded = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(first_loaded["method"], "thread/loaded/list"); - assert_eq!(first_loaded["id"], CONTROL_TUI_LOADED_REQUEST_ID); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_TUI_LOADED_REQUEST_ID, - "result": { "data": ["thread-unrelated"] } - }), - ) - .unwrap(); - let second_loaded = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(second_loaded["method"], "thread/loaded/list"); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_TUI_LOADED_REQUEST_ID, - "result": { "data": ["thread-unrelated", "thread-prior"] } - }), - ) - .unwrap(); - let subscribe = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(subscribe["method"], "thread/resume"); - assert_eq!(subscribe["params"]["threadId"], "thread-prior"); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "result": { - "thread": { "id": "thread-prior", "status": { "type": "idle" } } - } - }), - ) - .unwrap(); - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let binding_path = tmp.path().join("state/binding.json"); - let control_state_path = tmp.path().join("state/control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); - let runtime_for_pump = runtime.clone(); - let binding_for_pump = binding_path.clone(); - let control_state_for_pump = control_state_path.clone(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_for_pump, - &control_state_for_pump, - &runtime_for_pump, - Some(ControlResume { - thread_id: "thread-prior", - ready: resume_ready_rx, - tui_loaded_timeout: TUI_LOADED_TIMEOUT, - }), - None, - tx, - ) - }); - pre_gate_checked_rx - .recv_timeout(Duration::from_secs(2)) - .unwrap(); - resume_ready_tx.send(()).unwrap(); - acknowledge_tui_thread_loaded(&rx); - assert!(matches!( - rx.recv_timeout(Duration::from_secs(2)).unwrap(), - ControlEvent::Bound - )); - server.join().unwrap(); - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - - let binding = load_current_binding(&binding_path, &runtime) - .unwrap() - .unwrap(); - assert_eq!(binding.thread_id(), "thread-prior"); - let state = load_current_control_state(&control_state_path, &runtime, &binding) - .unwrap() - .unwrap(); - assert!(state.subscribed()); - assert_eq!(state.observed(), &CodexObservedState::Idle); - } - - /// A resumed thread still holds its context, and the app-server replays - /// `thread/tokenUsage/updated` to the newly attached connection — before the resume response, - /// which the binding handshake otherwise discards along with every other notification. The - /// construction that resumed this seat has already removed the predecessor's record, so a - /// dropped replay leaves a resumed-and-idle seat reading `null` against a full window with - /// nothing to correct it until its next model response. - #[test] - fn a_token_usage_replayed_before_the_resume_response_still_reaches_the_record() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let config = delivery_config(tmp.path()); - let agent_dir = config.agent_dir.clone(); - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let agent_dir_for_server = agent_dir.clone(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - stream - .set_read_timeout(Some(Duration::from_secs(10))) - .unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialize" - ); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialized" - ); - let loaded = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(loaded["method"], "thread/loaded/list"); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_TUI_LOADED_REQUEST_ID, - "result": { "data": ["thread-prior"] } - }), - ) - .unwrap(); - let subscribe = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(subscribe["method"], "thread/resume"); - assert_eq!(subscribe["params"]["threadId"], "thread-prior"); - // The replay, ahead of the response the handshake is waiting for. - write_json_message( - &mut websocket, - &json!({ - "method": "thread/tokenUsage/updated", - "params": { - "threadId": "thread-prior", - "turnId": "turn-prior", - "tokenUsage": { - "last": { "totalTokens": 92_283 }, - "total": { "totalTokens": 2_235_329 }, - "modelContextWindow": 258_400 - } - } - }), - ) - .unwrap(); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "result": { - "thread": { "id": "thread-prior", "status": { "type": "idle" } } - } - }), - ) - .unwrap(); - let deadline = Instant::now() + Duration::from_secs(10); - while harness_context::read(&harness_context::harness_context_path( - &agent_dir_for_server, - )) - .is_none() - && Instant::now() < deadline - { - std::thread::sleep(Duration::from_millis(10)); - } - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let binding_path = tmp.path().join("state/binding.json"); - let control_state_path = tmp.path().join("state/control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); - let binding_for_pump = binding_path.clone(); - let control_state_for_pump = control_state_path.clone(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_for_pump, - &control_state_for_pump, - &runtime, - Some(ControlResume { - thread_id: "thread-prior", - ready: resume_ready_rx, - tui_loaded_timeout: TUI_LOADED_TIMEOUT, - }), - Some(config), - tx, - ) - }); - resume_ready_tx.send(()).unwrap(); - acknowledge_tui_thread_loaded(&rx); - assert!(matches!( - rx.recv_timeout(Duration::from_secs(10)).unwrap(), - ControlEvent::Bound - )); - server.join().unwrap(); - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - - let observed = - context_record(&agent_dir).expect("the replayed reading never reached the record"); - assert_eq!(observed.used_percent, Some(33.0)); - assert_eq!(observed.used_tokens, Some(92_283)); - assert_eq!(observed.session_total_tokens, Some(2_235_329)); - } - - #[test] - fn tui_loaded_timeout_reports_the_specific_failure_before_outer_binding_timeout() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialize" - ); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialized" - ); - let loaded = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(loaded["method"], "thread/loaded/list"); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_TUI_LOADED_REQUEST_ID, - "result": { "data": [] } - }), - ) - .unwrap(); - thread::sleep(Duration::from_millis(250)); - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let binding_path = tmp.path().join("state/binding.json"); - let control_state_path = tmp.path().join("state/control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_path, - &control_state_path, - &runtime, - Some(ControlResume { - thread_id: "thread-prior", - ready: resume_ready_rx, - tui_loaded_timeout: Duration::from_millis(50), - }), - None, - tx, - ) - }); - resume_ready_tx.send(()).unwrap(); - let ControlEvent::Failed(error) = rx.recv_timeout(Duration::from_secs(2)).unwrap() else { - panic!("inner TUI-loaded deadline did not report its specific failure"); - }; - assert!( - error.contains( - "controlled Codex TUI did not load preserved thread thread-prior before control resume" - ), - "unexpected control failure: {error}" - ); - - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - server.join().unwrap(); - } - - #[test] - fn missing_saved_rollout_fails_without_rebinding_the_incarnation() { - let tmp = tempfile::tempdir().unwrap(); - let _stop_exclusive = stop_flag_tests(); - let binding_path = tmp.path().join("state/binding.json"); - let control_state_path = tmp.path().join("state/control-state.json"); - let prior_runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let prior_binding = CodexThreadBinding::new(&prior_runtime, "thread-prior".into()); - atomic_json(&binding_path, &prior_binding).unwrap(); - - let socket = tmp.path().join("server.sock"); - let listener = UnixListener::bind(&socket).unwrap(); - let server = thread::spawn(move || { - let (stream, _) = listener.accept().unwrap(); - let mut websocket = tungstenite::accept(stream).unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialize" - ); - write_json_message( - &mut websocket, - &json!({ "id": 0, "result": { "userAgent": "fake" } }), - ) - .unwrap(); - assert_eq!( - read_json_message(&mut websocket).unwrap().unwrap()["method"], - "initialized" - ); - let loaded = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(loaded["method"], "thread/loaded/list"); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_TUI_LOADED_REQUEST_ID, - "result": { "data": ["thread-prior"] } - }), - ) - .unwrap(); - let resume = read_json_message(&mut websocket).unwrap().unwrap(); - assert_eq!(resume["method"], "thread/resume"); - assert_eq!(resume["params"]["threadId"], "thread-prior"); - write_json_message( - &mut websocket, - &json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "error": { - "code": -32600, - "message": "no rollout found for thread id thread-prior" - } - }), - ) - .unwrap(); - }); - - let stream = UnixStream::connect(&socket).unwrap(); - let shutdown = stream.try_clone().unwrap(); - let websocket = initialize_control(stream) - .unwrap() - .expect("no stop raised in tests"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let (tx, rx) = mpsc::channel(); - let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); - let runtime_for_pump = runtime.clone(); - let binding_for_pump = binding_path.clone(); - let control_state_for_pump = control_state_path.clone(); - let pump = thread::spawn(move || { - pump_control( - websocket, - &binding_for_pump, - &control_state_for_pump, - &runtime_for_pump, - Some(ControlResume { - thread_id: "thread-prior", - ready: resume_ready_rx, - tui_loaded_timeout: TUI_LOADED_TIMEOUT, - }), - None, - tx, - ) - }); - resume_ready_tx.send(()).unwrap(); - acknowledge_tui_thread_loaded(&rx); - let ControlEvent::Failed(error) = rx.recv_timeout(Duration::from_secs(2)).unwrap() else { - panic!("missing saved rollout did not fail closed"); - }; - assert!(error.contains("saved Codex resume binding has no persisted rollout")); - - server.join().unwrap(); - let _ = shutdown.shutdown(Shutdown::Both); - pump.join().unwrap(); - assert_eq!( - serde_json::from_slice::(&fs::read(&binding_path).unwrap()) - .unwrap(), - prior_binding - ); - assert!(!control_state_path.exists()); - } - - #[test] - fn a_binding_from_another_runtime_incarnation_is_rejected() { - let tmp = tempfile::tempdir().unwrap(); - let path = tmp.path().join("binding.json"); - let prior = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let current = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - atomic_json( - &path, - &CodexThreadBinding::new(&prior, "thread-prior".into()), - ) - .unwrap(); - assert_eq!( - load_resume_thread(&path, "h.worker", "h.worker").unwrap(), - Some("thread-prior".into()), - "a validated prior binding may select resume but must not become current ownership" - ); - let error = load_current_binding(&path, ¤t).unwrap_err(); - assert!(error.to_string().contains("different runtime incarnation")); - } - - #[test] - fn watcher_holds_without_an_exact_turn_and_tracks_one_unmatched_lifecycle() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - - assert!( - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "active", "activeFlags": [] } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::ActiveWithoutTurn, - turn_id: None, - } - ); - - assert!( - state - .observe(&json!({ - "method": "turn/started", - "params": { - "threadId": "thread-main", - "turn": { "id": "turn-1" } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-1".into() - } - ); - - assert!( - !state - .observe(&json!({ - "method": "turn/started", - "params": { - "threadId": "thread-other", - "turn": { "id": "turn-other" } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-1".into() - } - ); - - assert!( - state - .observe(&json!({ - "method": "turn/completed", - "params": { - "threadId": "thread-main", - "turn": { "id": "turn-1" } - } - })) - .unwrap() - ); - assert_eq!(state.observed(), &CodexObservedState::Idle); - } - - #[test] - fn watcher_holds_review_compaction_and_conflicting_turns_until_safe() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - state - .observe(&json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "item": { "type": "enteredReviewMode" } - } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-1".into()), - } - ); - - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "active", "activeFlags": [] } - } - })) - .unwrap(); - assert!(matches!( - state.observed(), - CodexObservedState::Held { - reason: CodexHoldReason::Review, - .. - } - )); - - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-2" } } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-2".into()), - } - ); - - // Codex can complete the preparatory review item after the reviewer turn starts. That - // duplicate review event keeps the typed hold bound to the newer turn. - assert!( - !state - .observe(&json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "item": { "type": "enteredReviewMode" } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-2".into()), - } - ); - - // The review hold also survives the stale turn completion. Only an idle thread releases - // it. - assert!( - !state - .observe(&json!({ - "method": "turn/completed", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-2".into()), - } - ); - - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { "threadId": "thread-main", "status": { "type": "idle" } } - })) - .unwrap(); - assert_eq!(state.observed(), &CodexObservedState::Idle); - - // A real review can start its reviewer turn before Codex reports the preparatory turn's - // typed review item. The typed non-steerable event refines that generic conflict. - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-late-1" } } - })) - .unwrap(); - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-late-2" } } - })) - .unwrap(); - assert!(matches!( - state.observed(), - CodexObservedState::Held { - reason: CodexHoldReason::ConflictingTurn, - .. - } - )); - state - .observe(&json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-late-1", - "item": { "type": "enteredReviewMode" } - } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-late-1".into()), - } - ); - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { "threadId": "thread-main", "status": { "type": "idle" } } - })) - .unwrap(); - - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-3" } } - })) - .unwrap(); - state - .observe(&json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-3", - "item": { "type": "contextCompaction" } - } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Compaction, - turn_id: Some("turn-3".into()), - } - ); - assert!( - !state - .observe(&json!({ - "method": "turn/completed", - "params": { "threadId": "thread-main", "turn": { "id": "turn-3" } } - })) - .unwrap() - ); - assert!(matches!( - state.observed(), - CodexObservedState::Held { - reason: CodexHoldReason::Compaction, - .. - } - )); - } - - #[test] - fn exiting_review_mode_mid_turn_restores_the_steerable_turn() { - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = - message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state.subscribed = true; - - // An inline review runs as its own turn on the reviewed thread, so the hold binds to the - // reviewer turn that `exitedReviewMode` later reports. - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-review" } } - })) - .unwrap(); - state - .observe(&json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-review", - "item": { "type": "enteredReviewMode", "id": "item-1", "review": "review" } - } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-review".into()), - } - ); - assert_eq!(delivery.maybe_request(&state).unwrap(), None); - - // Review ends while the turn keeps running: the typed exit item is the only signal, and it - // must restore the exact turn the hold carried instead of waiting for the next idle. - assert!( - state - .observe(&json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-review", - "item": { "type": "exitedReviewMode", "id": "item-2", "review": "review" } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-review".into(), - } - ); - - // Codex reports both lifecycle edges of the same item; the second one changes nothing. - assert!( - !state - .observe(&json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-review", - "item": { "type": "exitedReviewMode", "id": "item-2", "review": "review" } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-review".into(), - } - ); - - // The payoff: native delivery steers the still-running turn instead of waiting for idle. - let request = delivery.maybe_request(&state).unwrap().unwrap(); - assert_eq!(request["method"], "turn/steer"); - assert_eq!(request["params"]["threadId"], "thread-main"); - assert_eq!(request["params"]["expectedTurnId"], "turn-review"); - assert!(config.inbox.join(&filename).is_file()); - } - - #[test] - fn delivery_irrelevant_items_and_foreign_turn_review_exits_keep_the_observed_state() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - - // Most item types say nothing about steerability. They are ignored on purpose, not by - // omission: the observed state and the changed flag both stay put. - for item_type in ["agentMessage", "commandExecution", "webSearch"] { - assert!( - !state - .observe(&json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "item": { "type": item_type, "id": "item-1" } - } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-1".into(), - } - ); - } - - // A review exit reporting a turn the hold does not carry proves nothing about the held - // turn, so the hold survives exactly as it did before typed exits were observed. - state.observed = CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-2".into()), - }; - let stale_exit = json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "item": { "type": "exitedReviewMode", "id": "item-2", "review": "review" } - } - }); - assert!(!state.observe(&stale_exit).unwrap()); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::Review, - turn_id: Some("turn-2".into()), - } - ); - - // A review exit never invents a turn on an idle thread and never releases another hold. - for observed in [ - CodexObservedState::Idle, - CodexObservedState::AwaitingStatus, - CodexObservedState::Held { - reason: CodexHoldReason::Compaction, - turn_id: Some("turn-1".into()), - }, - CodexObservedState::Held { - reason: CodexHoldReason::ConflictingTurn, - turn_id: None, - }, - ] { - state.observed = observed.clone(); - assert!( - !state - .observe(&json!({ - "method": "item/started", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "item": { - "type": "exitedReviewMode", - "id": "item-3", - "review": "review" - } - } - })) - .unwrap() - ); - assert_eq!(state.observed(), &observed); - } - } - - #[test] - fn an_unclassified_item_holds_until_the_next_idle_status() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - - assert!( - state - .observe(&json!({ - "method": "item/completed", - "params": { - "threadId": "thread-main", - "turnId": "turn-1", - "item": { "type": "futureBlockingItem", "id": "item-1" } - } - })) - .unwrap() - ); - assert!(matches!( - state.observed(), - CodexObservedState::Held { - reason: CodexHoldReason::UnknownProtocol, - turn_id: Some(turn_id), - } if turn_id == "turn-1" - )); - - assert!( - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "idle" } - } - })) - .unwrap() - ); - assert_eq!(state.observed(), &CodexObservedState::Idle); - } - - #[test] - fn an_unclassified_server_request_holds_until_the_next_idle_status() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - - assert!( - !state - .observe(&json!({ - "id": 1, - "method": "item/commandExecution/requestApproval", - "params": {} - })) - .unwrap() - ); - assert!(matches!( - state.observed(), - CodexObservedState::Active { .. } - )); - - assert!( - state - .observe(&json!({ - "id": 2, - "method": "future/request", - "params": {} - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::UnknownProtocol, - turn_id: Some("turn-1".into()), - } - ); - - assert!( - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "idle" } - } - })) - .unwrap() - ); - assert_eq!(state.observed(), &CodexObservedState::Idle); - } - - #[test] - fn an_errored_turn_completes_into_the_named_error_not_a_conflicting_turn() { - // Replays the captured terminal-error ordering (#264): a usage limit emits - // `thread/status/changed -> systemError` immediately before the failed turn's - // `turn/completed`. That completion reports one turn's lifecycle and carries no thread - // status, so it is not evidence the thread recovered, and it is not evidence of a second - // live turn either. The honest resolution is the condition the thread itself reported. - for (status, reason) in [ - ("systemError", CodexHoldReason::SystemError), - ("notLoaded", CodexHoldReason::NotLoaded), - ] { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state.subscribed = true; - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - assert!( - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { "threadId": "thread-main", "status": { "type": status } } - })) - .unwrap() - ); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason, - turn_id: None - } - ); - - // Completion makes a reported system error terminal. It preserves `notLoaded`, whose - // owner is the later thread status that proves the thread loaded again. - let changed = state - .observe(&json!({ - "method": "turn/completed", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - - if reason == CodexHoldReason::SystemError { - assert!(changed); - assert_eq!( - state.observed(), - &CodexObservedState::TerminalError { - reason: CodexTerminalError::SystemError, - } - ); - } else { - assert!(!changed); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason, - turn_id: None, - } - ); - } - - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = - message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - if reason == CodexHoldReason::SystemError { - let request = delivery - .maybe_request(&state) - .unwrap() - .expect("a terminal system error must permit the next turn"); - assert_eq!(request["method"], "turn/start"); - } else { - assert_eq!(delivery.maybe_request(&state).unwrap(), None); - assert!( - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "idle" } - } - })) - .unwrap() - ); - assert_eq!(state.observed(), &CodexObservedState::Idle); - assert!(delivery.maybe_request(&state).unwrap().is_some()); - } - assert!(config.inbox.join(&filename).is_file()); - } - - // The next provider turn replaces the terminal diagnostic with the exact live turn. - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state.subscribed = true; - for message in [ - json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - }), - json!({ - "method": "thread/status/changed", - "params": { "threadId": "thread-main", "status": { "type": "systemError" } } - }), - json!({ - "method": "turn/completed", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - }), - ] { - state.observe(&message).unwrap(); - } - assert_eq!( - state.observed(), - &CodexObservedState::TerminalError { - reason: CodexTerminalError::SystemError, - } - ); - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "active", "activeFlags": [] } - } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::ActiveWithoutTurn, - turn_id: None, - } - ); - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-2" } } - })) - .unwrap(); - assert_eq!( - state.observed(), - &CodexObservedState::Active { - turn_id: "turn-2".into(), - } - ); - } - - #[test] - fn persisted_control_state_is_bound_to_the_exact_runtime_incarnation() { - let tmp = tempfile::tempdir().unwrap(); - let path = tmp.path().join("control-state.json"); - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let binding = CodexThreadBinding::new(&runtime, "thread-main".into()); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - state.observed = CodexObservedState::Active { - turn_id: "turn-1".into(), - }; - atomic_json(&path, &state).unwrap(); - let persisted: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); - assert_eq!(persisted["observed"]["turnId"], "turn-1"); - assert!(persisted["observed"].get("turn_id").is_none()); - - assert_eq!( - load_current_control_state(&path, &runtime, &binding) - .unwrap() - .unwrap(), - state - ); - - let replacement = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let replacement_binding = CodexThreadBinding::new(&replacement, "thread-main".into()); - let error = - load_current_control_state(&path, &replacement, &replacement_binding).unwrap_err(); - assert!(error.to_string().contains("different runtime binding")); - } - - #[test] - fn subscription_waits_for_a_rollout_without_claiming_success() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - let acceptance = state - .accept_subscription(&json!({ - "id": CONTROL_SUBSCRIBE_REQUEST_ID, - "error": { - "code": -32600, - "message": "no rollout found for thread id thread-main" - } - })) - .unwrap(); - - assert!(matches!(acceptance, SubscriptionAcceptance::Deferred)); - assert!(!state.subscribed()); - assert_eq!(state.observed(), &CodexObservedState::AwaitingStatus); - } - - #[test] - fn app_server_receives_only_its_supported_global_configuration() { - let authored = vec![ - "-c".into(), - "projects={\"/workspace\"={trust_level=\"trusted\"}}".into(), - "--model".into(), - "gpt-test".into(), - "--enable".into(), - "one".into(), - "--disable=two".into(), - "--strict-config".into(), - "--dangerously-bypass-approvals-and-sandbox".into(), - "--dangerously-bypass-hook-trust".into(), - "boot".into(), - ]; - - assert_eq!( - controlled_app_server_args("unix:///server.sock", &authored).unwrap(), - [ - "app-server", - "-c", - "projects={\"/workspace\"={trust_level=\"trusted\"}}", - "--enable", - "one", - "--disable=two", - "--strict-config", - "--listen", - "unix:///server.sock", - ] - ); - } - - #[test] - fn remote_resume_projects_exact_hook_hashes_without_persisted_state() { - let tmp = tempfile::tempdir().unwrap(); - let cwd = fs::canonicalize(tmp.path()).unwrap(); - let source = cwd.join(".codex/hooks.json"); - let untrusted_key = format!("{}:session_start:0:0", source.display()); - let modified_key = format!("{}:stop:1:0", source.display()); - let response = json!({ - "id": HOOK_TRUST_PREFLIGHT_REQUEST_ID, - "result": { - "data": [{ - "cwd": cwd, - "hooks": [ - { - "key": untrusted_key, - "currentHash": "sha256:one", - "trustStatus": "untrusted", - "isManaged": false, - "enabled": true - }, - { - "key": modified_key, - "currentHash": "sha256:two", - "trustStatus": "modified", - "isManaged": false, - "enabled": false - }, - { - "key": "already-trusted", - "currentHash": "sha256:three", - "trustStatus": "trusted", - "isManaged": false, - "enabled": true - }, - { - "key": "managed", - "currentHash": "sha256:four", - "trustStatus": "managed", - "isManaged": true, - "enabled": true - } - ] - }] - } - }); - - let projection = hook_trust_projection_from_response(&response, &cwd) - .unwrap() - .unwrap(); - assert_eq!(projection.count, 2); - let parsed: toml::Value = toml::from_str(&projection.override_value).unwrap(); - let state = parsed - .get("hooks") - .and_then(|hooks| hooks.get("state")) - .and_then(toml::Value::as_table) - .unwrap(); - assert_eq!( - state[&untrusted_key]["trusted_hash"].as_str(), - Some("sha256:one") - ); - assert_eq!( - state[&modified_key]["trusted_hash"].as_str(), - Some("sha256:two") - ); - assert!(!state.contains_key("already-trusted")); - assert!(!state.contains_key("managed")); - - let mut args = controlled_app_server_args( - "unix:///server.sock", - &["--dangerously-bypass-hook-trust".into(), "boot".into()], - ) - .unwrap(); - insert_app_server_config_override(&mut args, projection.override_value).unwrap(); - assert_eq!(args[args.len() - 4], "-c"); - assert!(args[args.len() - 3].starts_with("hooks.state=")); - assert_eq!(&args[args.len() - 2..], ["--listen", "unix:///server.sock"]); - } - - #[test] - fn hook_trust_projection_fails_closed_on_provider_shape_drift() { - let tmp = tempfile::tempdir().unwrap(); - let cwd = fs::canonicalize(tmp.path()).unwrap(); - let response = json!({ - "result": { - "data": [{ - "cwd": cwd, - "hooks": [{ - "key": "hook", - "currentHash": "not-a-provider-hash", - "trustStatus": "untrusted", - "isManaged": false - }] - }] - } - }); - let error = hook_trust_projection_from_response(&response, &cwd).unwrap_err(); - assert!(error.to_string().contains("typed currentHash")); - - let response = json!({ - "result": { - "data": [{ - "cwd": cwd, - "hooks": [{ - "key": "hook", - "currentHash": "sha256:value", - "trustStatus": "future-status", - "isManaged": false - }] - }] - } - }); - let error = hook_trust_projection_from_response(&response, &cwd).unwrap_err(); - assert!(error.to_string().contains("unknown trustStatus")); - } - - #[test] - fn hook_preflight_uses_the_explicit_controlled_workspace() { - let tmp = tempfile::tempdir().unwrap(); - let explicit = tmp.path().join("workspace"); - fs::create_dir(&explicit).unwrap(); - assert_eq!( - controlled_hook_cwd(&[ - "--dangerously-bypass-hook-trust".into(), - "--cd".into(), - explicit.display().to_string(), - "boot".into(), - ]) - .unwrap(), - fs::canonicalize(explicit).unwrap() - ); - assert!( - authored_bypasses_hook_trust(&[ - "--dangerously-bypass-hook-trust".into(), - "boot".into() - ]) - .unwrap() - ); - assert!( - !authored_bypasses_hook_trust(&["--".into(), "--dangerously-bypass-hook-trust".into()]) - .unwrap() - ); - } - - #[test] - fn process_group_cleanup_reaps_a_native_launcher_descendant() { - let temporary = tempfile::tempdir().unwrap(); - let descendant_pidfile = temporary.path().join("descendant.pid"); - let mut command = Command::new("sh"); - command - .arg("-c") - .arg(r#"sh -c 'printf "%s" "$$" > "$DESCENDANT_PIDFILE"; exec sleep 60' & sleep 60"#) - .env("DESCENDANT_PIDFILE", &descendant_pidfile) - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::null()); - let mut launcher = spawn_process_group(&mut command, None).unwrap(); - let mut foreign_command = Command::new("/bin/sh"); - foreign_command - .arg("-c") - .arg("sleep 60") - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::null()); - let mut foreign_owner = spawn_process_group(&mut foreign_command, None).unwrap(); - let foreign_pid = foreign_owner.id() as i32; - let deadline = Instant::now() + Duration::from_secs(1); - // The shell's `>` redirection creates an empty pidfile before `printf` - // writes, so wait for parsable content, not mere file existence. - let mut descendant = None; - while descendant.is_none() && Instant::now() < deadline { - if let Ok(content) = std::fs::read_to_string(&descendant_pidfile) { - descendant = content.trim().parse::().ok(); - } - if descendant.is_none() { - std::thread::sleep(Duration::from_millis(10)); - } - } - let descendant = descendant.expect("the launcher did not create its native descendant"); - assert!( - process_can_retain_cleanup_resources(descendant), - "the native descendant was not alive before cleanup" - ); - - launcher.terminate(); - assert!( - process_can_retain_cleanup_resources(foreign_pid), - "cleanup killed a different live owner" - ); - foreign_owner.terminate(); - let deadline = Instant::now() + Duration::from_secs(1); - while process_can_retain_cleanup_resources(descendant) && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(10)); - } - let survived = process_can_retain_cleanup_resources(descendant); - if survived { - unsafe { - libc::kill(descendant, libc::SIGKILL); - } - } - assert!( - !survived, - "native descendant {descendant} survived process-group cleanup" - ); - } - - #[test] - fn dropping_a_process_group_owner_reaps_the_group_and_socket() { - let temporary = tempfile::tempdir().unwrap(); - let socket_path = temporary.path().join("app-server.sock"); - let _listener = UnixListener::bind(&socket_path).unwrap(); - let mut command = Command::new("/bin/sh"); - command - .arg("-c") - .arg("sleep 60") - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::null()); - let launcher = spawn_process_group(&mut command, Some(&socket_path)).unwrap(); - let launcher_pid = launcher.id() as i32; - assert!(process_can_retain_cleanup_resources(launcher_pid)); - - drop(launcher); - let deadline = Instant::now() + Duration::from_secs(1); - while (process_can_retain_cleanup_resources(launcher_pid) || socket_path.exists()) - && Instant::now() < deadline - { - std::thread::sleep(Duration::from_millis(10)); - } - assert!( - !process_can_retain_cleanup_resources(launcher_pid), - "the app-server survived owner cleanup" - ); - assert!( - !socket_path.exists(), - "the app-server socket survived owner cleanup" - ); - } - - #[test] - fn a_live_socket_refuses_a_second_control_owner() { - let temporary = tempfile::tempdir().unwrap(); - let socket_path = temporary.path().join("app-server.sock"); - let listener = UnixListener::bind(&socket_path).unwrap(); - - let error = prepare_socket_for_launch(&socket_path).unwrap_err(); - - assert!( - error - .to_string() - .contains("refusing a second control owner") - ); - assert!(socket_path.exists(), "the live owner socket was removed"); - assert!( - UnixStream::connect(&socket_path).is_ok(), - "the first owner stopped accepting connections" - ); - drop(listener); - } - - #[test] - fn a_dead_socket_is_removed_before_launch() { - let temporary = tempfile::tempdir().unwrap(); - let socket_path = temporary.path().join("app-server.sock"); - let listener = UnixListener::bind(&socket_path).unwrap(); - drop(listener); - assert!(socket_path.exists()); - - prepare_socket_for_launch(&socket_path).unwrap(); - - assert!(!socket_path.exists(), "the dead socket was not removed"); - } - - #[test] - fn a_killed_wrapper_reaps_its_app_server_and_the_next_launch_recovers_its_socket() { - const TEST_NAME: &str = "codex_app_server::tests::a_killed_wrapper_reaps_its_app_server_and_the_next_launch_recovers_its_socket"; - const ROLE: &str = "ST2_CODEX_ORPHAN_TEST_ROLE"; - const SOCKET_PATH: &str = "ST2_CODEX_ORPHAN_TEST_SOCKET"; - const PID_PATH: &str = "ST2_CODEX_ORPHAN_TEST_PID"; - const READY_PATH: &str = "ST2_CODEX_ORPHAN_TEST_READY"; - - match std::env::var(ROLE).as_deref() { - Ok("server") => { - let socket_path = PathBuf::from(std::env::var_os(SOCKET_PATH).unwrap()); - let ready_path = PathBuf::from(std::env::var_os(READY_PATH).unwrap()); - let _listener = UnixListener::bind(socket_path).unwrap(); - fs::write(ready_path, b"ready").unwrap(); - loop { - std::thread::sleep(Duration::from_secs(60)); - } - } - Ok("wrapper") => { - let pid_path = PathBuf::from(std::env::var_os(PID_PATH).unwrap()); - let socket_path = PathBuf::from(std::env::var_os(SOCKET_PATH).unwrap()); - let mut command = Command::new(std::env::current_exe().unwrap()); - command - .arg("--exact") - .arg(TEST_NAME) - .arg("--nocapture") - .env(ROLE, "server") - .env(SOCKET_PATH, std::env::var_os(SOCKET_PATH).unwrap()) - .env(READY_PATH, std::env::var_os(READY_PATH).unwrap()) - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::null()); - let server = spawn_process_group(&mut command, Some(&socket_path)).unwrap(); - fs::write(pid_path, server.id().to_string()).unwrap(); - loop { - std::thread::sleep(Duration::from_secs(60)); - } - } - Ok(role) => panic!("unknown orphan test role {role}"), - Err(_) => {} - } - - let temporary = tempfile::tempdir().unwrap(); - let socket_path = temporary.path().join("app-server.sock"); - let pid_path = temporary.path().join("app-server.pid"); - let ready_path = temporary.path().join("app-server.ready"); - let mut wrapper = Command::new(std::env::current_exe().unwrap()) - .arg("--exact") - .arg(TEST_NAME) - .arg("--nocapture") - .env(ROLE, "wrapper") - .env(SOCKET_PATH, &socket_path) - .env(PID_PATH, &pid_path) - .env(READY_PATH, &ready_path) - .stdin(Stdio::null()) - .stdout(Stdio::null()) - .stderr(Stdio::null()) - .spawn() - .unwrap(); - let deadline = Instant::now() + Duration::from_secs(2); - while (!pid_path.is_file() || !ready_path.is_file()) && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(10)); - } - if !pid_path.is_file() || !ready_path.is_file() { - let _ = wrapper.kill(); - let _ = wrapper.wait(); - panic!("the wrapper did not start its app-server"); - } - let server_pid = fs::read_to_string(&pid_path) - .expect("the wrapper did not report its app-server PID") - .parse::() - .unwrap(); - assert!( - process_can_retain_cleanup_resources(server_pid), - "the app-server was not alive before the wrapper died" - ); - assert!( - fs::symlink_metadata(&socket_path) - .unwrap() - .file_type() - .is_socket(), - "the app-server did not bind its socket" - ); - - unsafe { - libc::kill(wrapper.id() as i32, libc::SIGKILL); - } - let _ = wrapper.wait(); - let deadline = Instant::now() + Duration::from_secs(2); - while process_can_retain_cleanup_resources(server_pid) && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(10)); - } - let server_survived = process_can_retain_cleanup_resources(server_pid); - if server_survived { - unsafe { - libc::kill(server_pid, libc::SIGKILL); - } - } - assert!(!server_survived, "the app-server survived its wrapper"); - assert!( - socket_path.exists(), - "the app-server did not leave the expected recoverable socket" - ); - let refusal_deadline = Instant::now() + Duration::from_secs(2); - let refusal = loop { - match UnixStream::connect(&socket_path) { - Ok(stream) if Instant::now() < refusal_deadline => { - drop(stream); - std::thread::sleep(Duration::from_millis(10)); - } - Ok(_) => panic!("the residual socket still had a live listener"), - Err(error) => break error, - } - }; - assert_eq!(refusal.kind(), std::io::ErrorKind::ConnectionRefused); - - prepare_socket_for_launch(&socket_path) - .expect("the next launch did not recover the residual socket"); - assert!( - !socket_path.exists(), - "the next launch did not remove the residual socket" - ); - let replacement = UnixListener::bind(&socket_path) - .expect("the next app-server could not bind the recovered socket"); - assert!( - UnixStream::connect(&socket_path).is_ok(), - "the replacement app-server socket did not accept a connection" - ); - drop(replacement); - } - - #[test] - fn app_server_configuration_extraction_fails_closed_at_ambiguous_boundaries() { - let missing = - controlled_app_server_args("unix:///server.sock", &["-c".into()]).unwrap_err(); - assert!(missing.to_string().contains("has no value")); - - let unknown = controlled_app_server_args( - "unix:///server.sock", - &["--future-option".into(), "value".into(), "boot".into()], - ) - .unwrap_err(); - assert!(unknown.to_string().contains("unknown Codex option")); - - let sensitive = controlled_app_server_args( - "unix:///server.sock", - &["--future-token=do-not-log-this".into(), "boot".into()], - ) - .unwrap_err(); - assert!(sensitive.to_string().contains("--future-token")); - assert!(!sensitive.to_string().contains("do-not-log-this")); - - assert_eq!( - controlled_app_server_args( - "unix:///server.sock", - &[ - "--config=projects.x.trust_level=\"trusted\"".into(), - "resume".into(), - "thread-explicit".into(), - ], - ) - .unwrap(), - [ - "app-server", - "--config=projects.x.trust_level=\"trusted\"", - "--listen", - "unix:///server.sock", - ] - ); - } - - #[test] - fn controlled_tui_resumes_a_prior_binding_without_overriding_authored_selection() { - let authored = vec!["--model".into(), "gpt-test".into(), "boot".into()]; - assert_eq!( - controlled_tui_args("unix:///server.sock", &authored, None).unwrap(), - [ - "--remote", - "unix:///server.sock", - "--model", - "gpt-test", - "boot" - ] - ); - assert_eq!( - controlled_tui_args("unix:///server.sock", &authored, Some("thread-prior")).unwrap(), - [ - "--remote", - "unix:///server.sock", - "resume", - "--model", - "gpt-test", - "thread-prior", - "boot" - ] - ); - assert_eq!( - controlled_tui_args( - "unix:///server.sock", - &["resume".into(), "thread-explicit".into()], - Some("thread-prior") - ) - .unwrap(), - [ - "--remote", - "unix:///server.sock", - "resume", - "thread-explicit" - ] - ); - assert_eq!( - expected_resume_thread( - &["resume".into(), "thread-explicit".into()], - Some("thread-prior") - ) - .unwrap(), - None - ); - - let fork = vec![ - "--dangerously-bypass-hook-trust".into(), - "fork".into(), - "thread-explicit".into(), - ]; - assert_eq!( - controlled_tui_args("unix:///server.sock", &fork, Some("thread-prior")).unwrap(), - [ - "--remote", - "unix:///server.sock", - "--dangerously-bypass-hook-trust", - "fork", - "thread-explicit" - ] - ); - assert_eq!( - expected_resume_thread(&fork, Some("thread-prior")).unwrap(), - None - ); - assert_eq!( - expected_resume_thread(&authored, Some("thread-prior")).unwrap(), - Some("thread-prior") - ); - } - - #[test] - fn controlled_tui_resume_fails_closed_at_ambiguous_option_boundaries() { - let unknown = controlled_tui_args( - "unix:///server.sock", - &["--future-option".into(), "value".into(), "prompt".into()], - Some("thread-prior"), - ) - .unwrap_err(); - assert!(unknown.to_string().contains("unknown Codex option")); - - let image = controlled_tui_args( - "unix:///server.sock", - &["--image".into(), "one.png".into(), "prompt".into()], - Some("thread-prior"), - ) - .unwrap_err(); - assert!(image.to_string().contains("explicit `--`")); - - assert_eq!( - controlled_tui_args( - "unix:///server.sock", - &[ - "--image".into(), - "one.png".into(), - "--".into(), - "prompt".into(), - ], - Some("thread-prior"), - ) - .unwrap(), - [ - "--remote", - "unix:///server.sock", - "resume", - "--image", - "one.png", - "thread-prior", - "--", - "prompt" - ] - ); - } - - #[test] - fn state_key_is_path_and_identity_specific_without_embedding_either() { - let base = Path::new("/state"); - let first = state_dir_in(base, Path::new("/catalog/a"), "h.worker"); - let second = state_dir_in(base, Path::new("/catalog/b"), "h.worker"); - assert_ne!(first, second); - assert!(first.starts_with("/state/st2/codex")); - assert!(!first.display().to_string().contains("worker")); - assert!(!first.display().to_string().contains("catalog/a")); - } - - #[test] - fn wrapper_diagnostics_keep_one_bounded_run_without_authored_input() { - let tmp = tempfile::tempdir().unwrap(); - let state = tmp.path().join("state"); - secure_dir(&state).unwrap(); - - { - let mut diagnostics = WrapperDiagnostics::open(&state, "h.worker", "h.worker").unwrap(); - diagnostics.record("ownerAcquired", json!({})).unwrap(); - diagnostics - .record("failed", json!({ "error": "control socket was not ready" })) - .unwrap(); - } - let path = state.join("wrapper.log"); - let first = fs::read_to_string(&path).unwrap(); - let entries = first - .lines() - .map(|line| serde_json::from_str::(line).unwrap()) - .collect::>(); - assert_eq!(entries.len(), 2); - assert_eq!(entries[0]["schema"], WRAPPER_DIAGNOSTIC_SCHEMA); - assert_eq!(entries[0]["agent"], "h.worker"); - assert_eq!(entries[1]["stage"], "failed"); - assert!(first.contains("control socket was not ready")); - assert!(!first.contains("prompt")); - - { - let mut replacement = WrapperDiagnostics::open(&state, "h.worker", "h.worker").unwrap(); - replacement.record("ownerAcquired", json!({})).unwrap(); - } - let replacement = fs::read_to_string(&path).unwrap(); - assert_eq!(replacement.lines().count(), 1); - assert!(!replacement.contains("control socket was not ready")); - assert_eq!( - fs::metadata(path).unwrap().permissions().mode() & 0o777, - 0o600 - ); - } - - #[test] - fn runtime_owner_lock_is_nonblocking_and_released_on_close() { - let tmp = tempfile::tempdir().unwrap(); - let first = acquire_owner_lock(tmp.path()).unwrap(); - let error = acquire_owner_lock(tmp.path()).unwrap_err(); - assert!(error.to_string().contains("already has an owner")); - drop(first); - acquire_owner_lock(tmp.path()).unwrap(); - } - - #[test] - fn waiting_on_a_human_holds_the_exact_turn_and_releases_it_when_the_flag_clears() { - let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); - let mut state = CodexControlState::new(&runtime, "thread-main".into()); - let status_changed = |flags: Value| { - json!({ - "method": "thread/status/changed", - "params": { - "threadId": "thread-main", - "status": { "type": "active", "activeFlags": flags } - } - }) - }; - let active_turn_1 = CodexObservedState::Active { - turn_id: "turn-1".into(), - }; - - state.observe(&status_changed(json!([]))).unwrap(); - state - .observe(&json!({ - "method": "turn/started", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - assert_eq!(state.observed(), &active_turn_1); - - for (flags, reason) in [ - ( - json!(["waitingOnApproval"]), - CodexHoldReason::WaitingOnApproval, - ), - ( - json!(["waitingOnUserInput"]), - CodexHoldReason::WaitingOnUserInput, - ), - ( - json!(["conversationHandoff", "waitingOnApproval"]), - CodexHoldReason::WaitingOnApproval, - ), - ] { - assert!(state.observe(&status_changed(flags)).unwrap()); - assert_eq!( - state.observed(), - &CodexObservedState::Held { - reason, - turn_id: Some("turn-1".into()), - } - ); - // Clearing the flag releases the same turn: no `turn/started` repeats mid-turn. - assert!(state.observe(&status_changed(json!([]))).unwrap()); - assert_eq!(state.observed(), &active_turn_1); - } - - // An unknown future flag value degrades to plain `active` instead of failing the frame. - assert!(!state.observe(&status_changed(json!(["handoff"]))).unwrap()); - assert_eq!(state.observed(), &active_turn_1); - - // The same field is carried by `thread/started`, before any turn is known. - let mut resumed = CodexControlState::new(&runtime, "thread-main".into()); - assert!( - resumed - .observe(&json!({ - "method": "thread/started", - "params": { - "thread": { - "id": "thread-main", - "status": { - "type": "active", - "activeFlags": ["waitingOnUserInput"] - } - } - } - })) - .unwrap() - ); - assert_eq!( - resumed.observed(), - &CodexObservedState::Held { - reason: CodexHoldReason::WaitingOnUserInput, - turn_id: None, - } - ); - - // A turn that ends while still flagged stays unsteerable and is released by the next - // idle status. `observe_turn_completed` is not modified here; this pins only that the - // flagged hold cannot decay into a steerable turn. - state - .observe(&status_changed(json!(["waitingOnApproval"]))) - .unwrap(); - state - .observe(&json!({ - "method": "turn/completed", - "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } - })) - .unwrap(); - assert!(matches!(state.observed(), CodexObservedState::Held { .. })); - - // A status arm without `activeFlags` keeps reading exactly as before. - assert!( - state - .observe(&json!({ - "method": "thread/status/changed", - "params": { "threadId": "thread-main", "status": { "type": "idle" } } - })) - .unwrap() - ); - assert_eq!(state.observed(), &CodexObservedState::Idle); - - // Delivery declines to steer a session that is waiting on a human, and retains the head. - let tmp = tempfile::tempdir().unwrap(); - let config = delivery_config(tmp.path()); - let filename = - message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") - .unwrap(); - let mut delivery = inbox_delivery(tmp.path(), config.clone()); - for reason in [ - CodexHoldReason::WaitingOnApproval, - CodexHoldReason::WaitingOnUserInput, - ] { - let blocked = subscribed_state(CodexObservedState::Held { - reason, - turn_id: Some("turn-1".into()), - }); - assert_eq!(delivery.maybe_request(&blocked).unwrap(), None); - assert!(config.inbox.join(&filename).is_file()); - } - let released = delivery - .maybe_request(&subscribed_state(active_turn_1.clone())) - .unwrap() - .expect("the retained head steers once the human has answered"); - assert_eq!(released["method"], "turn/steer"); - assert_eq!(released["params"]["expectedTurnId"], "turn-1"); - } -} +mod tests; diff --git a/src/codex_app_server/context.rs b/src/codex_app_server/context.rs new file mode 100644 index 00000000..a1f558f8 --- /dev/null +++ b/src/codex_app_server/context.rs @@ -0,0 +1,276 @@ +//! The Codex half of the harness-context record. +//! +//! Moved verbatim out of the parent module: Codex's occupancy arithmetic and the producer that +//! turns its token-usage and rate-limit frames into a harness-context reading. + +use std::collections::VecDeque; + +use anyhow::Result; +use serde_json::Value; + +use crate::harness_context; + +/// Codex's `BASELINE_TOKENS`, subtracted from BOTH the numerator and the denominator of its +/// displayed occupancy: `codex-rs/protocol/src/protocol.rs:2332` and +/// `codex-rs/tui/src/token_usage.rs:9` at `rust-v0.151.0` carry the same literal with an identical +/// function body, and no configuration override exists. Its doc comment: "should capture tokens +/// that are always present in the context (e.g. system prompt and fixed tool instructions) so that +/// the percentage reflects the portion the user can influence." +pub(super) const CODEX_BASELINE_TOKENS: i64 = 12_000; + +/// The seven-day rate-limit window, identified by its duration because +/// `account/rateLimits/updated` names its windows `primary`/`secondary` and nothing else. 10,080 +/// minutes = 7 days, and the one captured Codex rate-limit snapshot (rollout, 0.150.1) carries +/// exactly this window as `primary`. See [`CodexContextProducer::observe_rate_limits`] for why the +/// five-hour leg stays `null`. +const CODEX_SEVEN_DAY_WINDOW_MINUTES: i64 = 10_080; + +/// How many recent compaction identities the dedupe retains. One compaction reaches this observer +/// as both `item/started` and `item/completed` — and possibly also as the deprecated +/// `thread/compacted` — so counting the edge naively counts one compaction twice or three times. A +/// last-key-only memory would still miscount an interleaving (`started(A)`, `started(B)`, +/// `completed(A)`), which a small ring closes for the same cost. +const CODEX_COMPACTION_MEMORY: usize = 4; + +/// Codex's own occupancy arithmetic, mirrored rather than re-derived: the published number is +/// exactly `100 −` the "N% context left" the operator reads in the Codex footer. +/// +/// `codex-rs/tui/src/token_usage.rs:43` (and its protocol twin): +/// +/// ```text +/// if context_window <= BASELINE_TOKENS { return 0; } +/// effective = context_window - BASELINE_TOKENS +/// used = (last.total_tokens - BASELINE_TOKENS).max(0) +/// remaining = (effective - used).max(0) +/// ((remaining / effective) * 100).clamp(0,100).round() +/// ``` +/// +/// Three things this deliberately does NOT do: +/// +/// - It does not round the *used* percentage. Rounding `used/effective` and rounding +/// `remaining/effective` disagree on a half — effective 200, used 101 gives 51 one way and 50 +/// the other — and only the mirrored order satisfies the spec's "equals `100 −` Codex's +/// displayed '% context left'". +/// - It does not use `total`, which is cumulative session spend. Against the captured window a +/// `total`-based percent reads 100 where the true occupancy is 33. +/// - It does not use `last.inputTokens`, which gives ~36 against the same capture — close enough +/// to look right and wrong by construction. +/// +/// The one divergence from the source: where Codex returns `0` remaining for a window at or below +/// the baseline, mirroring blindly would publish "100% used" for a window it cannot normalize. st2 +/// withholds instead (HC-R02, HC-R03) — a saturation the harness never displayed is fabricated, +/// not observed. +/// +/// The result cannot exceed 100: Codex's `remaining` is floored at zero, so an occupancy above the +/// effective window saturates in the harness's own arithmetic before st2 ever sees it. That is a +/// property of mirroring Codex, not a clamp of st2's — the record still carries what a producer +/// computes, unclamped (HC-R02), and the harnesses that can report an overrun are the ones +/// publishing a float of their own. +pub(super) fn codex_used_percent(window_tokens: Option, last_total_tokens: i64) -> Option { + let window = window_tokens?; + if window <= CODEX_BASELINE_TOKENS { + return None; + } + let effective = window - CODEX_BASELINE_TOKENS; + let used = (last_total_tokens - CODEX_BASELINE_TOKENS).max(0); + let remaining = (effective - used).max(0); + let remaining_percent = ((remaining as f64 / effective as f64) * 100.0) + .clamp(0.0, 100.0) + .round(); + Some(100.0 - remaining_percent) +} + +/// One compaction's identity as this observer can name it. The item events carry a stable item id +/// alongside the turn; the deprecated `thread/compacted` notification carries only the turn, so its +/// key collapses with any item key in the same turn rather than counting beside it. +#[derive(Debug, Clone, PartialEq, Eq)] +struct CodexCompactionKey { + turn_id: String, + item_id: Option, +} + +impl CodexCompactionKey { + /// Whether these two names describe the same compaction. Two distinct item ids in one turn are + /// two compactions; a turn-only name in a turn already counted is the same one under its other + /// spelling. + fn same_compaction(&self, other: &Self) -> bool { + self.turn_id == other.turn_id + && match (&self.item_id, &other.item_id) { + (Some(mine), Some(theirs)) => mine == theirs, + _ => true, + } + } +} + +/// The Codex half of the harness-context record (HC-R11). +/// +/// It owns a [`harness_context::Writer`] beside the harness-state writer, sharing the wrapper's +/// incarnation so both records name the same session as their provenance. It holds no guard of its +/// own: `thread/tokenUsage/updated` arrives once per model response — roughly 10–15 per turn, and +/// replayed to a newly attached connection on resume — and every one of them is handed to +/// [`harness_context::Writer::observe`], whose quantization is the only thing deciding what lands. +/// A second guard here would make the write policy per-harness, which HC-R09 exists to prevent. +/// +/// The only state it carries between notifications is what it cannot recover from the next one: +/// the account-scoped rate-limit windows (a separate notification with no reading behind it) and +/// the identities of recently counted compactions. +pub(super) struct CodexContextProducer { + writer: harness_context::Writer, + /// Last-known account-scoped windows. `account/rateLimits/updated` is documented as a *sparse + /// rolling update* whose absent fields do not clear a previously observed value, so the last + /// known windows ride along with the next reading instead of blanking it. + rate_limits: harness_context::RateLimits, + counted_compactions: VecDeque, +} + +impl CodexContextProducer { + pub(super) fn new(writer: harness_context::Writer) -> Self { + Self { + writer, + rate_limits: harness_context::RateLimits::default(), + counted_compactions: VecDeque::new(), + } + } + + /// Project one inbound control frame onto the context record, returning whether a write landed. + /// + /// Every unknown method, foreign thread, and malformed payload is ignored rather than failed: + /// this is observability riding a delivery socket, and a frame this producer cannot read must + /// not disturb the frame the delivery loop can. + pub(super) fn observe(&mut self, message: &Value, thread_id: &str) -> Result { + let Some(method) = message.get("method").and_then(Value::as_str) else { + return Ok(false); + }; + match method { + "thread/tokenUsage/updated" => { + if message.pointer("/params/threadId").and_then(Value::as_str) != Some(thread_id) { + return Ok(false); + } + let Some(reading) = self.token_usage_reading(message) else { + return Ok(false); + }; + self.writer.observe(reading) + } + // Account-scoped and thread-free (HC-T06): it repeats across every runtime sharing the + // account, carries no occupancy, and therefore never writes on its own. It is held and + // published by the next reading. + "account/rateLimits/updated" => { + self.observe_rate_limits(message); + Ok(false) + } + "item/started" | "item/completed" => { + if message.pointer("/params/threadId").and_then(Value::as_str) != Some(thread_id) + || message.pointer("/params/item/type").and_then(Value::as_str) + != Some("contextCompaction") + { + return Ok(false); + } + let (Some(turn_id), Some(item_id)) = ( + message.pointer("/params/turnId").and_then(Value::as_str), + message.pointer("/params/item/id").and_then(Value::as_str), + ) else { + return Ok(false); + }; + self.compacted(CodexCompactionKey { + turn_id: turn_id.to_string(), + item_id: Some(item_id.to_string()), + }) + } + // Deprecated in the protocol in favour of the item ("Deprecated: Use + // `ContextCompaction` item type instead") and unobserved on 0.150.1. Handled anyway, + // and deduped against the item, because a harness emitting both must still count one + // compaction. + "thread/compacted" => { + if message.pointer("/params/threadId").and_then(Value::as_str) != Some(thread_id) { + return Ok(false); + } + let Some(turn_id) = message.pointer("/params/turnId").and_then(Value::as_str) + else { + return Ok(false); + }; + self.compacted(CodexCompactionKey { + turn_id: turn_id.to_string(), + item_id: None, + }) + } + _ => Ok(false), + } + } + + /// The reading a `thread/tokenUsage/updated` carries, in Codex's own arithmetic. + /// + /// `usedTokens` and `windowTokens` are the harness's raw operands and are published as they + /// arrive — a window at or below the baseline is still a window the harness reported, even + /// where it cannot produce a percent. `model` and `costUsd` are `null` because the channel + /// carries neither: the app-server `Thread` object has `modelProvider` and no model identifier, + /// and Codex reports no session cost anywhere in the protocol (HC-R16). + fn token_usage_reading(&self, message: &Value) -> Option { + let last_total = message + .pointer("/params/tokenUsage/last/totalTokens") + .and_then(Value::as_i64)?; + let window = message + .pointer("/params/tokenUsage/modelContextWindow") + .and_then(Value::as_i64) + .filter(|window| *window > 0); + Some(harness_context::Reading { + used_tokens: u64::try_from(last_total).ok(), + window_tokens: window.and_then(|window| u64::try_from(window).ok()), + used_percent: codex_used_percent(window, last_total), + model: None, + cost_usd: None, + // Cumulative lifetime spend and never occupancy (HC-R16): the captured session read + // 2,235,329 against a 258,400-token window. + session_total_tokens: message + .pointer("/params/tokenUsage/total/totalTokens") + .and_then(Value::as_i64) + .and_then(|total| u64::try_from(total).ok()), + rate_limits: self.rate_limits, + }) + } + + /// Merge a sparse rate-limit update into the last-known windows. + /// + /// Codex names its windows `primary` and `secondary` and identifies them only by + /// `windowDurationMins`, so the join is by duration. Only the seven-day window is carried: the + /// single captured Codex rate-limit snapshot (0.150.1) contains one window, `primary`, at + /// 10,080 minutes. No 300-minute window and no `secondary` was ever observed on this harness, + /// so mapping one onto `fiveHour` would be inference dressed as a measurement — and this + /// record's whole point is that its numbers were seen. `fiveHour` therefore stays `null` for + /// Codex until a capture shows the window; admitting it is a one-line change beside the + /// capture that justifies it. + fn observe_rate_limits(&mut self, message: &Value) { + for window in ["primary", "secondary"] { + let Some(snapshot) = message.pointer(&format!("/params/rateLimits/{window}")) else { + continue; + }; + if snapshot.get("windowDurationMins").and_then(Value::as_i64) + == Some(CODEX_SEVEN_DAY_WINDOW_MINUTES) + && let Some(used) = snapshot.get("usedPercent").and_then(Value::as_f64) + { + self.rate_limits.seven_day = Some(used); + } + } + } + + /// Count one compaction edge unless this compaction was already counted under another of its + /// spellings. The count is incarnation-scoped: Codex publishes an edge and nothing else, so st2 + /// does the counting and the relaunch claim's record removal resets it (HC-R12, HC-R15). The + /// trigger is `unknown` because `ContextCompactionThreadItem` carries `id` and `type` and no + /// reason at all. + fn compacted(&mut self, key: CodexCompactionKey) -> Result { + if self + .counted_compactions + .iter() + .any(|counted| counted.same_compaction(&key)) + { + return Ok(false); + } + self.counted_compactions.push_back(key); + while self.counted_compactions.len() > CODEX_COMPACTION_MEMORY { + self.counted_compactions.pop_front(); + } + self.writer.compacted(harness_context::Compaction::new( + harness_context::CompactionTrigger::Unknown, + )) + } +} diff --git a/src/codex_app_server/process_group.rs b/src/codex_app_server/process_group.rs new file mode 100644 index 00000000..f91bd9a3 --- /dev/null +++ b/src/codex_app_server/process_group.rs @@ -0,0 +1,132 @@ +//! The wrapper-owned process group a Codex provider launcher runs in. +//! +//! Moved verbatim out of the parent module: the group's lifetime, its close-on-exec discipline, +//! and the spawn that establishes it. + +use std::fs; +use std::os::unix::io::AsRawFd as _; +use std::os::unix::net::UnixStream; +use std::os::unix::process::CommandExt as _; +use std::path::{Path, PathBuf}; +use std::process::{Child, Command, Stdio}; + +/// One app-server process group and the write end of its wrapper-liveness channel. +/// +/// A watchdog in the dedicated process group owns the read end. The watchdog kills only that +/// group if this wrapper disappears without running Rust cleanup. Its membership also prevents +/// the operating system from reusing the group ID before cleanup. +pub(super) struct OwnedProcessGroup { + child: Child, + watchdog: Child, + owner_write: Option, + socket_path: Option, + active: bool, +} + +impl OwnedProcessGroup { + pub(super) fn id(&self) -> u32 { + self.child.id() + } + + pub(super) fn child_mut(&mut self) -> &mut Child { + &mut self.child + } + + pub(super) fn terminate(&mut self) { + if !self.active { + return; + } + self.active = false; + let process_group = self.watchdog.id() as i32; + unsafe { + libc::kill(-process_group, libc::SIGKILL); + } + let _ = self.child.kill(); + let _ = self.child.wait(); + let _ = self.watchdog.kill(); + let _ = self.watchdog.wait(); + if let Some(socket_path) = self.socket_path.as_deref() { + let _ = fs::remove_file(socket_path); + } + self.owner_write.take(); + } +} + +impl Drop for OwnedProcessGroup { + fn drop(&mut self) { + self.terminate(); + } +} + +fn set_close_on_exec(fd: libc::c_int) -> std::io::Result<()> { + let mut flags = unsafe { libc::fcntl(fd, libc::F_GETFD) }; + if flags == -1 { + return Err(std::io::Error::last_os_error()); + } + flags |= libc::FD_CLOEXEC; + if unsafe { libc::fcntl(fd, libc::F_SETFD, flags) } == -1 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) +} + +/// Spawn a provider launcher in an isolated, wrapper-owned process group. +/// +/// Explicit cleanup covers normal returns and Rust errors. The in-group watchdog covers wrapper +/// crashes, SIGKILL, and supervisor teardown. The watchdog holds the group ID until cleanup, so a +/// stale PID can never identify a process group that belongs to another live owner. A crash can +/// leave one dead socket file; the next launch proves that it has no listener and removes it. +pub(super) fn spawn_process_group( + command: &mut Command, + socket_path: Option<&Path>, +) -> std::io::Result { + let (watchdog_read, owner_write) = UnixStream::pair()?; + set_close_on_exec(owner_write.as_raw_fd())?; + let owner_write_fd = owner_write.as_raw_fd(); + let mut watchdog_command = Command::new("/bin/sh"); + watchdog_command + .arg("-c") + .arg("IFS= read -r ignored; kill -KILL 0") + .arg("st2-codex-watchdog") + .stdin(Stdio::from(std::os::fd::OwnedFd::from(watchdog_read))) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + unsafe { + watchdog_command.pre_exec(|| { + if libc::setpgid(0, 0) == -1 { + return Err(std::io::Error::last_os_error()); + } + Ok(()) + }); + } + let mut watchdog = watchdog_command.spawn()?; + let watchdog_process_group = watchdog.id() as i32; + unsafe { + command.pre_exec(move || { + if libc::setpgid(0, watchdog_process_group) == -1 { + return Err(std::io::Error::last_os_error()); + } + libc::close(owner_write_fd); + Ok(()) + }); + } + let child = match command.spawn() { + Ok(child) => child, + Err(error) => { + drop(owner_write); + unsafe { + libc::kill(-watchdog_process_group, libc::SIGKILL); + } + let _ = watchdog.kill(); + let _ = watchdog.wait(); + return Err(error); + } + }; + Ok(OwnedProcessGroup { + child, + watchdog, + owner_write: Some(owner_write), + socket_path: socket_path.map(Path::to_path_buf), + active: true, + }) +} diff --git a/src/codex_app_server/protocol.rs b/src/codex_app_server/protocol.rs new file mode 100644 index 00000000..5bce983a --- /dev/null +++ b/src/codex_app_server/protocol.rs @@ -0,0 +1,711 @@ +//! The Codex app-server protocol admission gate. +//! +//! Moved verbatim out of the parent module: the version probe and the JSON-schema walk that +//! decides whether the installed app-server speaks the protocol this build was written for. + +use std::collections::{BTreeMap, BTreeSet}; +use std::fs; +use std::process::Command; +use std::thread; +use std::time::Duration; + +use anyhow::{Context as _, Result}; +use serde_json::Value; + +use super::{ + CODEX_PROVIDER_AUTH_REJECTED, REQUIRED_CODEX_CLIENT_NOTIFICATIONS, + REQUIRED_CODEX_CLIENT_REQUESTS, REQUIRED_CODEX_SERVER_NOTIFICATIONS, +}; + +pub(super) struct CodexProtocolSchemas { + pub(super) protocol: Value, + pub(super) client_requests: Value, + pub(super) client_notifications: Value, + pub(super) server_requests: Value, + pub(super) server_notifications: Value, +} + +/// Admit the installed Codex app-server protocol, returning the version that passed. +pub(super) fn ensure_supported_protocol(codex: &str) -> Result { + let version = codex_version(codex)?; + let generated = tempfile::Builder::new() + .prefix("st2-codex-protocol-") + .tempdir() + .context("creating a temporary Codex protocol schema directory")?; + let output = Command::new(codex) + .args([ + "app-server", + "generate-json-schema", + "--experimental", + "--out", + ]) + .arg(generated.path()) + .output() + .with_context(|| format!("generating the Codex app-server schema from {codex}"))?; + anyhow::ensure!( + output.status.success(), + "{codex} app-server schema generation failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + ); + let read_schema = |name: &str| -> Result { + let path = generated.path().join(name); + let bytes = + fs::read(&path).with_context(|| format!("reading generated Codex schema {name}"))?; + serde_json::from_slice(&bytes) + .with_context(|| format!("parsing generated Codex schema {name}")) + }; + let schemas = CodexProtocolSchemas { + protocol: read_schema("codex_app_server_protocol.v2.schemas.json")?, + client_requests: read_schema("ClientRequest.json")?, + client_notifications: read_schema("ClientNotification.json")?, + server_requests: read_schema("ServerRequest.json")?, + server_notifications: read_schema("ServerNotification.json")?, + }; + verify_codex_protocol_schemas(&schemas) + .with_context(|| format!("Codex app-server schema from {version} is incompatible"))?; + Ok(version) +} + +fn codex_version(codex: &str) -> Result { + let mut attempt_index = 0; + let output = loop { + let attempt = Command::new(codex).arg("--version").output(); + match attempt { + Ok(output) => break output, + Err(error) if error.raw_os_error() == Some(libc::ETXTBSY) && attempt_index + 1 < 5 => { + // Some Linux filesystems briefly retain writer exclusion after a binary install. + // Retry only this transient error and keep every other launch error immediate. + attempt_index += 1; + thread::sleep(Duration::from_millis(20)); + } + Err(error) => { + return Err(error).with_context(|| format!("reading Codex version from {codex}")); + } + } + }; + anyhow::ensure!( + output.status.success(), + "{codex} --version failed: {}", + String::from_utf8_lossy(&output.stderr).trim() + ); + let actual = String::from_utf8(output.stdout) + .context("Codex version output is not UTF-8")? + .trim() + .to_string(); + anyhow::ensure!(!actual.is_empty(), "{codex} --version printed nothing"); + Ok(actual) +} + +pub(super) fn verify_codex_protocol_schemas(schemas: &CodexProtocolSchemas) -> Result<()> { + let definitions = schemas + .protocol + .get("definitions") + .and_then(Value::as_object) + .context("aggregate schema has no definitions object")?; + + require_methods( + &schemas.client_requests, + REQUIRED_CODEX_CLIENT_REQUESTS, + "client request", + )?; + require_methods( + &schemas.client_notifications, + REQUIRED_CODEX_CLIENT_NOTIFICATIONS, + "client notification", + )?; + require_methods( + &schemas.server_notifications, + REQUIRED_CODEX_SERVER_NOTIFICATIONS, + "server notification", + )?; + schema_methods(&schemas.server_requests, "server request")?; + + let status_variants = schema_variants(definitions, "ThreadStatus", "type")?; + for status in ["notLoaded", "idle", "systemError", "active"] { + anyhow::ensure!( + status_variants.contains_key(status), + "ThreadStatus has no '{status}' variant" + ); + } + let active = status_variants + .get("active") + .context("ThreadStatus has no active variant")?; + let active_flags = + required_property(definitions, active, "activeFlags", "ThreadStatus.active")?; + let active_flag = require_array(definitions, active_flags, "ThreadStatus.activeFlags")?; + anyhow::ensure!( + active_flag == schema_definition(definitions, "ThreadActiveFlag")?, + "ThreadStatus.activeFlags does not contain ThreadActiveFlag" + ); + let actual_active_flags = schema_enum(definitions, "ThreadActiveFlag")?; + anyhow::ensure!( + actual_active_flags == string_set(&["waitingOnApproval", "waitingOnUserInput"]), + "ThreadActiveFlag changed: {}", + actual_active_flags + .into_iter() + .collect::>() + .join(", ") + ); + + let item_variants = schema_variants(definitions, "ThreadItem", "type")?; + for item in [ + "contextCompaction", + "enteredReviewMode", + "exitedReviewMode", + "userMessage", + ] { + anyhow::ensure!( + item_variants.contains_key(item), + "ThreadItem has no '{item}' variant" + ); + } + let user_message = item_variants + .get("userMessage") + .context("ThreadItem has no userMessage variant")?; + require_property_type( + definitions, + user_message, + "clientId", + "string", + false, + "ThreadItem.userMessage", + )?; + + let user_input_variants = schema_variants(definitions, "UserInput", "type")?; + let text_input = user_input_variants + .get("text") + .context("UserInput has no text variant")?; + require_property_type( + definitions, + text_input, + "text", + "string", + true, + "UserInput.text", + )?; + let text_elements = property(definitions, text_input, "text_elements", "UserInput.text")?; + require_array(definitions, text_elements, "UserInput.text.text_elements")?; + + for (definition, path) in [ + ("ClientInfo", &["name"][..]), + ("ClientInfo", &["version"][..]), + ("Thread", &["id"][..]), + ("Turn", &["id"][..]), + ("ThreadStatusChangedNotification", &["threadId"][..]), + ("TurnStartedNotification", &["threadId"][..]), + ("TurnStartedNotification", &["turn", "id"][..]), + ("TurnCompletedNotification", &["threadId"][..]), + ("TurnCompletedNotification", &["turn", "id"][..]), + ("ItemStartedNotification", &["threadId"][..]), + ("ItemStartedNotification", &["turnId"][..]), + ("ItemCompletedNotification", &["threadId"][..]), + ("ItemCompletedNotification", &["turnId"][..]), + ("ThreadStartedNotification", &["thread", "id"][..]), + ("ThreadResumeParams", &["threadId"][..]), + ("ThreadResumeResponse", &["thread", "id"][..]), + ("TurnStartParams", &["threadId"][..]), + ("TurnStartResponse", &["turn", "id"][..]), + ("TurnSteerParams", &["threadId"][..]), + ("TurnSteerParams", &["expectedTurnId"][..]), + ("TurnSteerResponse", &["turnId"][..]), + ] { + let schema = required_schema_path(definitions, definition, path)?; + require_type( + definitions, + schema, + "string", + &format!("{definition}.{}", path.join(".")), + )?; + } + + require_property_type( + definitions, + schema_definition(definitions, "ClientInfo")?, + "title", + "string", + false, + "ClientInfo", + )?; + require_property_type( + definitions, + schema_definition(definitions, "InitializeCapabilities")?, + "experimentalApi", + "boolean", + false, + "InitializeCapabilities", + )?; + required_schema_path(definitions, "InitializeParams", &["clientInfo"])?; + + let thread_status = required_schema_path(definitions, "Thread", &["status"])?; + anyhow::ensure!( + thread_status == schema_definition(definitions, "ThreadStatus")?, + "Thread.status does not use ThreadStatus" + ); + let resume_status = + required_schema_path(definitions, "ThreadResumeResponse", &["thread", "status"])?; + anyhow::ensure!( + resume_status == schema_definition(definitions, "ThreadStatus")?, + "ThreadResumeResponse.thread.status does not use ThreadStatus" + ); + let started_status = required_schema_path( + definitions, + "ThreadStartedNotification", + &["thread", "status"], + )?; + anyhow::ensure!( + started_status == schema_definition(definitions, "ThreadStatus")?, + "ThreadStartedNotification.thread.status does not use ThreadStatus" + ); + let changed_status = + required_schema_path(definitions, "ThreadStatusChangedNotification", &["status"])?; + anyhow::ensure!( + changed_status == schema_definition(definitions, "ThreadStatus")?, + "ThreadStatusChangedNotification.status does not use ThreadStatus" + ); + + let turns = required_schema_path(definitions, "Thread", &["turns"])?; + let turn = require_array(definitions, turns, "Thread.turns")?; + anyhow::ensure!( + turn == schema_definition(definitions, "Turn")?, + "Thread.turns does not contain Turn" + ); + let items = required_schema_path(definitions, "Turn", &["items"])?; + let item = require_array(definitions, items, "Turn.items")?; + anyhow::ensure!( + item == schema_definition(definitions, "ThreadItem")?, + "Turn.items does not contain ThreadItem" + ); + // The typed turn result the provider-credential classifier reads. A release that renames the + // status word, drops the failure's typed error, or merges the credential arm into a quota arm + // must refuse the launch rather than let st2 silently stop classifying rejections — or, worse, + // report an exhausted allowance as a rejected credential. + let turn_status = required_schema_path(definitions, "Turn", &["status"])?; + anyhow::ensure!( + turn_status == schema_definition(definitions, "TurnStatus")?, + "Turn.status does not use TurnStatus" + ); + let turn_statuses = schema_enum(definitions, "TurnStatus")?; + for status in ["completed", "failed"] { + anyhow::ensure!( + turn_statuses.contains(status), + "TurnStatus has no '{status}' variant" + ); + } + let turn_error = nullable_schema( + definitions, + property( + definitions, + schema_definition(definitions, "Turn")?, + "error", + "Turn", + )?, + "Turn.error", + )?; + anyhow::ensure!( + turn_error == schema_definition(definitions, "TurnError")?, + "Turn.error does not use TurnError" + ); + let error_info = nullable_schema( + definitions, + property( + definitions, + schema_definition(definitions, "TurnError")?, + "codexErrorInfo", + "TurnError", + )?, + "TurnError.codexErrorInfo", + )?; + anyhow::ensure!( + error_info == schema_definition(definitions, "CodexErrorInfo")?, + "TurnError.codexErrorInfo does not use CodexErrorInfo" + ); + let error_words = schema_variant_words(definitions, "CodexErrorInfo")?; + for word in [ + CODEX_PROVIDER_AUTH_REJECTED, + "rateLimitExceeded", + "usageLimitExceeded", + ] { + anyhow::ensure!( + error_words.contains(word), + "CodexErrorInfo has no '{word}' word" + ); + } + for notification in ["ItemStartedNotification", "ItemCompletedNotification"] { + let item = required_schema_path(definitions, notification, &["item"])?; + anyhow::ensure!( + item == schema_definition(definitions, "ThreadItem")?, + "{notification}.item does not use ThreadItem" + ); + } + + for params in ["TurnStartParams", "TurnSteerParams"] { + let input = required_schema_path(definitions, params, &["input"])?; + let input_item = require_array(definitions, input, &format!("{params}.input"))?; + anyhow::ensure!( + input_item == schema_definition(definitions, "UserInput")?, + "{params}.input does not contain UserInput" + ); + require_property_type( + definitions, + schema_definition(definitions, params)?, + "clientUserMessageId", + "string", + false, + params, + )?; + } + let loaded = required_schema_path(definitions, "ThreadLoadedListResponse", &["data"])?; + let loaded_item = require_array(definitions, loaded, "ThreadLoadedListResponse.data")?; + require_type( + definitions, + loaded_item, + "string", + "ThreadLoadedListResponse.data item", + )?; + let hook_cwds = property( + definitions, + schema_definition(definitions, "HooksListParams")?, + "cwds", + "HooksListParams", + )?; + let hook_cwd = require_array(definitions, hook_cwds, "HooksListParams.cwds")?; + require_type(definitions, hook_cwd, "string", "HooksListParams.cwds item")?; + verify_hook_schema(definitions)?; + Ok(()) +} + +fn verify_hook_schema(definitions: &serde_json::Map) -> Result<()> { + let data = required_schema_path(definitions, "HooksListResponse", &["data"])?; + let entry = require_array(definitions, data, "HooksListResponse.data")?; + anyhow::ensure!( + entry == schema_definition(definitions, "HooksListEntry")?, + "HooksListResponse.data does not contain HooksListEntry" + ); + let hooks = required_schema_path(definitions, "HooksListEntry", &["hooks"])?; + let hook = require_array(definitions, hooks, "HooksListEntry.hooks")?; + anyhow::ensure!( + hook == schema_definition(definitions, "HookMetadata")?, + "HooksListEntry.hooks does not contain HookMetadata" + ); + for (property, expected_type) in [ + ("currentHash", "string"), + ("isManaged", "boolean"), + ("key", "string"), + ] { + require_property_type( + definitions, + schema_definition(definitions, "HookMetadata")?, + property, + expected_type, + true, + "HookMetadata", + )?; + } + let trust_status = required_schema_path(definitions, "HookMetadata", &["trustStatus"])?; + anyhow::ensure!( + trust_status == schema_definition(definitions, "HookTrustStatus")?, + "HookMetadata.trustStatus does not use HookTrustStatus" + ); + let statuses = schema_enum(definitions, "HookTrustStatus")?; + anyhow::ensure!( + statuses == string_set(&["managed", "modified", "trusted", "untrusted"]), + "HookTrustStatus changed: {}", + statuses.into_iter().collect::>().join(", ") + ); + Ok(()) +} + +fn string_set(values: &[&str]) -> BTreeSet { + values.iter().map(|value| (*value).to_string()).collect() +} + +fn schema_methods(schema: &Value, label: &str) -> Result> { + let arms = schema + .get("oneOf") + .and_then(Value::as_array) + .with_context(|| format!("{label} schema has no oneOf array"))?; + let mut methods = BTreeSet::new(); + for arm in arms { + let required = arm + .get("required") + .and_then(Value::as_array) + .with_context(|| format!("{label} arm has no required array"))?; + anyhow::ensure!( + required + .iter() + .any(|value| value.as_str() == Some("method")), + "{label} arm does not require method" + ); + let values = arm + .pointer("/properties/method/enum") + .and_then(Value::as_array) + .with_context(|| format!("{label} arm has no method enum"))?; + anyhow::ensure!(values.len() == 1, "{label} arm method enum is not exact"); + let method = values[0] + .as_str() + .with_context(|| format!("{label} arm method is not a string"))?; + anyhow::ensure!( + methods.insert(method.to_string()), + "{label} method '{method}' is duplicated" + ); + } + Ok(methods) +} + +fn require_methods(schema: &Value, required: &[&str], label: &str) -> Result<()> { + let methods = schema_methods(schema, label)?; + let missing = string_set(required) + .difference(&methods) + .cloned() + .collect::>(); + anyhow::ensure!( + missing.is_empty(), + "missing {label} methods: {}", + missing.join(", ") + ); + Ok(()) +} + +fn schema_definition<'a>( + definitions: &'a serde_json::Map, + name: &str, +) -> Result<&'a Value> { + definitions + .get(name) + .with_context(|| format!("aggregate schema has no {name} definition")) +} + +fn resolve_schema<'a>( + definitions: &'a serde_json::Map, + mut schema: &'a Value, +) -> Result<&'a Value> { + for _ in 0..16 { + if let Some(reference) = schema.get("$ref").and_then(Value::as_str) { + let name = reference + .strip_prefix("#/definitions/") + .with_context(|| format!("unsupported schema reference '{reference}'"))?; + schema = schema_definition(definitions, name)?; + continue; + } + if let Some(all_of) = schema.get("allOf").and_then(Value::as_array) { + anyhow::ensure!(all_of.len() == 1, "schema allOf is not a single reference"); + schema = &all_of[0]; + continue; + } + return Ok(schema); + } + anyhow::bail!("schema reference depth exceeds 16") +} + +/// Resolve `anyOf: [T, null]` — the shape the Codex generator emits for an optional typed field — +/// to `T`. A field that is not exactly one typed arm beside `null` is refused rather than guessed. +fn nullable_schema<'a>( + definitions: &'a serde_json::Map, + schema: &'a Value, + label: &str, +) -> Result<&'a Value> { + let schema = resolve_schema(definitions, schema)?; + let arms = schema + .get("anyOf") + .and_then(Value::as_array) + .with_context(|| format!("{label} is not a nullable schema"))?; + let mut typed = arms + .iter() + .filter(|arm| arm.get("type").and_then(Value::as_str) != Some("null")); + let only = typed + .next() + .with_context(|| format!("{label} has no typed arm"))?; + anyhow::ensure!( + typed.next().is_none(), + "{label} has more than one typed arm" + ); + resolve_schema(definitions, only) +} + +/// Every unit word of a `oneOf` union that mixes a string enum with data-carrying object arms — +/// the shape `CodexErrorInfo` has. Only the enum arms carry words st2 can match on. +fn schema_variant_words( + definitions: &serde_json::Map, + definition: &str, +) -> Result> { + let arms = schema_definition(definitions, definition)? + .get("oneOf") + .and_then(Value::as_array) + .with_context(|| format!("{definition} has no oneOf variants"))?; + let mut words = BTreeSet::new(); + for arm in arms { + let Some(values) = resolve_schema(definitions, arm)? + .get("enum") + .and_then(Value::as_array) + else { + continue; + }; + for value in values { + let word = value + .as_str() + .with_context(|| format!("{definition} has a non-string enum value"))?; + words.insert(word.to_string()); + } + } + anyhow::ensure!(!words.is_empty(), "{definition} has no enum words"); + Ok(words) +} + +fn schema_variants<'a>( + definitions: &'a serde_json::Map, + definition: &str, + discriminator: &str, +) -> Result> { + let schema = schema_definition(definitions, definition)?; + let variants = schema + .get("oneOf") + .and_then(Value::as_array) + .with_context(|| format!("{definition} has no oneOf variants"))?; + let mut found = BTreeMap::new(); + for variant in variants { + let variant = resolve_schema(definitions, variant)?; + let required = variant + .get("required") + .and_then(Value::as_array) + .with_context(|| format!("{definition} variant has no required array"))?; + anyhow::ensure!( + required + .iter() + .any(|value| value.as_str() == Some(discriminator)), + "{definition} variant does not require {discriminator}" + ); + let values = variant + .pointer(&format!("/properties/{discriminator}/enum")) + .and_then(Value::as_array) + .with_context(|| format!("{definition} variant has no {discriminator} enum"))?; + anyhow::ensure!( + values.len() == 1, + "{definition} variant discriminator is not exact" + ); + let value = values[0] + .as_str() + .with_context(|| format!("{definition} discriminator is not a string"))?; + anyhow::ensure!( + found.insert(value.to_string(), variant).is_none(), + "{definition} discriminator '{value}' is duplicated" + ); + } + Ok(found) +} + +fn schema_enum( + definitions: &serde_json::Map, + definition: &str, +) -> Result> { + let values = schema_definition(definitions, definition)? + .get("enum") + .and_then(Value::as_array) + .with_context(|| format!("{definition} has no enum"))?; + values + .iter() + .map(|value| { + value + .as_str() + .map(str::to_string) + .with_context(|| format!("{definition} has a non-string enum value")) + }) + .collect() +} + +fn property<'a>( + definitions: &'a serde_json::Map, + schema: &'a Value, + name: &str, + label: &str, +) -> Result<&'a Value> { + let schema = resolve_schema(definitions, schema)?; + let property = schema + .get("properties") + .and_then(Value::as_object) + .and_then(|properties| properties.get(name)) + .with_context(|| format!("{label} has no {name} property"))?; + resolve_schema(definitions, property) +} + +fn required_property<'a>( + definitions: &'a serde_json::Map, + schema: &'a Value, + name: &str, + label: &str, +) -> Result<&'a Value> { + let schema = resolve_schema(definitions, schema)?; + let required = schema + .get("required") + .and_then(Value::as_array) + .with_context(|| format!("{label} has no required array"))?; + anyhow::ensure!( + required.iter().any(|value| value.as_str() == Some(name)), + "{label} does not require {name}" + ); + property(definitions, schema, name, label) +} + +fn required_schema_path<'a>( + definitions: &'a serde_json::Map, + definition: &str, + path: &[&str], +) -> Result<&'a Value> { + let mut schema = schema_definition(definitions, definition)?; + let mut label = definition.to_string(); + for component in path { + schema = required_property(definitions, schema, component, &label)?; + label.push('.'); + label.push_str(component); + } + Ok(schema) +} + +fn require_property_type( + definitions: &serde_json::Map, + schema: &Value, + property_name: &str, + expected_type: &str, + required: bool, + label: &str, +) -> Result<()> { + let property = if required { + required_property(definitions, schema, property_name, label)? + } else { + property(definitions, schema, property_name, label)? + }; + require_type( + definitions, + property, + expected_type, + &format!("{label}.{property_name}"), + ) +} + +fn require_type( + definitions: &serde_json::Map, + schema: &Value, + expected: &str, + label: &str, +) -> Result<()> { + let schema = resolve_schema(definitions, schema)?; + let matches = match schema.get("type") { + Some(Value::String(actual)) => actual == expected, + Some(Value::Array(actual)) => actual.iter().any(|value| value.as_str() == Some(expected)), + _ => false, + }; + anyhow::ensure!(matches, "{label} does not accept {expected}"); + Ok(()) +} + +fn require_array<'a>( + definitions: &'a serde_json::Map, + schema: &'a Value, + label: &str, +) -> Result<&'a Value> { + let schema = resolve_schema(definitions, schema)?; + require_type(definitions, schema, "array", label)?; + let items = schema + .get("items") + .with_context(|| format!("{label} has no item schema"))?; + resolve_schema(definitions, items) +} diff --git a/src/codex_app_server/tests.rs b/src/codex_app_server/tests.rs new file mode 100644 index 00000000..3d117063 --- /dev/null +++ b/src/codex_app_server/tests.rs @@ -0,0 +1,4193 @@ +use super::*; + +/// The stop flag is process-global, so every test that exercises a reader of it — +/// [`initialize_control`] above all — holds this lock against the one test that flips +/// the flag: parallel readers would otherwise observe the raised flag and fail their +/// `no stop raised in tests` expectations. +fn stop_flag_tests() -> std::sync::MutexGuard<'static, ()> { + static LOCK: std::sync::LazyLock> = + std::sync::LazyLock::new(std::sync::Mutex::default); + match LOCK.lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + } +} + +#[test] +fn a_stop_during_the_websocket_handshake_ends_startup_gracefully() { + let _stop_exclusive = stop_flag_tests(); + let tmp = tempfile::tempdir().unwrap(); + let socket_path = tmp.path().join("control.sock"); + let listener = UnixListener::bind(&socket_path).unwrap(); + let silent_server = std::thread::spawn(move || listener.accept().map(|(stream, _)| stream)); + let stream = UnixStream::connect(&socket_path).unwrap(); + let stopper = std::thread::spawn(|| { + std::thread::sleep(Duration::from_millis(300)); + crate::provider_session::STOP.store(true, std::sync::atomic::Ordering::SeqCst); + }); + let started = Instant::now(); + let result = initialize_control(stream); + // Join before resetting: on an early failure return the stopper has not fired yet, + // and resetting first would let it re-poison the global flag for every later test. + stopper.join().unwrap(); + crate::provider_session::STOP.store(false, std::sync::atomic::Ordering::SeqCst); + let _held_open = silent_server.join().unwrap().unwrap(); + assert!( + result.unwrap().is_none(), + "a stop while the server sits silent mid-handshake must return the graceful None" + ); + assert!( + started.elapsed() < STARTUP_TIMEOUT, + "the stop must unblock the handshake well before the startup timeout" + ); +} +use std::os::unix::fs::PermissionsExt; +use std::os::unix::net::UnixListener; + +#[cfg(target_os = "linux")] +fn linux_process_state(pid: i32) -> Option { + std::fs::read_to_string(format!("/proc/{pid}/stat")) + .ok()? + .rsplit_once(") ")? + .1 + .chars() + .next() +} + +fn process_can_retain_cleanup_resources(pid: i32) -> bool { + #[cfg(target_os = "linux")] + if linux_process_state(pid) == Some('Z') { + return false; + } + crate::host_lock::process_alive(pid) +} + +fn object_schema(required: &[&str], properties: &[(&str, Value)]) -> Value { + json!({ + "type": "object", + "required": required, + "properties": properties + .iter() + .map(|(name, schema)| ((*name).to_string(), schema.clone())) + .collect::>() + }) +} + +fn reference(name: &str) -> Value { + json!({ "$ref": format!("#/definitions/{name}") }) +} + +fn array_of(items: Value) -> Value { + json!({ "type": "array", "items": items }) +} + +fn tagged_variant(name: &str, required: &[&str], properties: &[(&str, Value)]) -> Value { + let mut all_required = vec!["type"]; + all_required.extend(required); + let mut all_properties = vec![("type", json!({ "type": "string", "enum": [name] }))]; + all_properties.extend(properties.iter().cloned()); + object_schema(&all_required, &all_properties) +} + +fn method_schema(methods: &[&str]) -> Value { + json!({ + "oneOf": methods + .iter() + .map(|method| object_schema( + &["method"], + &[("method", json!({ "type": "string", "enum": [method] }))], + )) + .collect::>() + }) +} + +fn compatible_protocol_schemas() -> CodexProtocolSchemas { + let mut definitions = serde_json::Map::new(); + definitions.insert( + "ThreadActiveFlag".into(), + json!({ + "type": "string", + "enum": ["waitingOnApproval", "waitingOnUserInput"] + }), + ); + definitions.insert( + "ThreadStatus".into(), + json!({ + "oneOf": [ + tagged_variant("notLoaded", &[], &[]), + tagged_variant("idle", &[], &[]), + tagged_variant("systemError", &[], &[]), + tagged_variant( + "active", + &["activeFlags"], + &[("activeFlags", array_of(reference("ThreadActiveFlag")))], + ) + ] + }), + ); + definitions.insert( + "ThreadItem".into(), + json!({ + "oneOf": [ + tagged_variant("contextCompaction", &[], &[]), + tagged_variant("enteredReviewMode", &[], &[]), + tagged_variant("exitedReviewMode", &[], &[]), + tagged_variant( + "userMessage", + &[], + &[("clientId", json!({ "type": ["string", "null"] }))], + ) + ] + }), + ); + definitions.insert("TextElement".into(), object_schema(&[], &[])); + definitions.insert( + "UserInput".into(), + json!({ + "oneOf": [tagged_variant( + "text", + &["text"], + &[ + ("text", json!({ "type": "string" })), + ("text_elements", array_of(reference("TextElement"))), + ], + )] + }), + ); + definitions.insert( + "ClientInfo".into(), + object_schema( + &["name", "version"], + &[ + ("name", json!({ "type": "string" })), + ("title", json!({ "type": ["string", "null"] })), + ("version", json!({ "type": "string" })), + ], + ), + ); + definitions.insert( + "InitializeCapabilities".into(), + object_schema(&[], &[("experimentalApi", json!({ "type": "boolean" }))]), + ); + definitions.insert( + "InitializeParams".into(), + object_schema( + &["clientInfo"], + &[ + ("clientInfo", reference("ClientInfo")), + ("capabilities", reference("InitializeCapabilities")), + ], + ), + ); + definitions.insert( + "Thread".into(), + object_schema( + &["id", "status", "turns"], + &[ + ("id", json!({ "type": "string" })), + ("status", reference("ThreadStatus")), + ("turns", array_of(reference("Turn"))), + ], + ), + ); + definitions.insert( + "Turn".into(), + object_schema( + &["id", "items", "status"], + &[ + ("id", json!({ "type": "string" })), + ("items", array_of(reference("ThreadItem"))), + ("status", reference("TurnStatus")), + ( + "error", + json!({ "anyOf": [reference("TurnError"), { "type": "null" }] }), + ), + ], + ), + ); + definitions.insert( + "TurnStatus".into(), + json!({ + "type": "string", + "enum": ["completed", "interrupted", "failed", "inProgress"] + }), + ); + definitions.insert( + "TurnError".into(), + object_schema( + &["message"], + &[ + ("message", json!({ "type": "string" })), + ( + "codexErrorInfo", + json!({ "anyOf": [reference("CodexErrorInfo"), { "type": "null" }] }), + ), + ], + ), + ); + definitions.insert( + "CodexErrorInfo".into(), + json!({ + "oneOf": [ + { + "type": "string", + "enum": [ + "usageLimitExceeded", + "rateLimitExceeded", + "unauthorized", + "other" + ] + }, + object_schema( + &["httpConnectionFailed"], + &[("httpConnectionFailed", object_schema(&[], &[]))], + ) + ] + }), + ); + for notification in ["TurnStartedNotification", "TurnCompletedNotification"] { + definitions.insert( + notification.into(), + object_schema( + &["threadId", "turn"], + &[ + ("threadId", json!({ "type": "string" })), + ("turn", reference("Turn")), + ], + ), + ); + } + for notification in ["ItemStartedNotification", "ItemCompletedNotification"] { + definitions.insert( + notification.into(), + object_schema( + &["threadId", "turnId", "item"], + &[ + ("threadId", json!({ "type": "string" })), + ("turnId", json!({ "type": "string" })), + ("item", reference("ThreadItem")), + ], + ), + ); + } + definitions.insert( + "ThreadStartedNotification".into(), + object_schema(&["thread"], &[("thread", reference("Thread"))]), + ); + definitions.insert( + "ThreadStatusChangedNotification".into(), + object_schema( + &["threadId", "status"], + &[ + ("threadId", json!({ "type": "string" })), + ("status", reference("ThreadStatus")), + ], + ), + ); + definitions.insert( + "ThreadResumeParams".into(), + object_schema(&["threadId"], &[("threadId", json!({ "type": "string" }))]), + ); + definitions.insert( + "ThreadResumeResponse".into(), + object_schema(&["thread"], &[("thread", reference("Thread"))]), + ); + definitions.insert( + "TurnStartParams".into(), + object_schema( + &["threadId", "input"], + &[ + ("threadId", json!({ "type": "string" })), + ("input", array_of(reference("UserInput"))), + ("clientUserMessageId", json!({ "type": ["string", "null"] })), + ], + ), + ); + definitions.insert( + "TurnSteerParams".into(), + object_schema( + &["threadId", "expectedTurnId", "input"], + &[ + ("threadId", json!({ "type": "string" })), + ("expectedTurnId", json!({ "type": "string" })), + ("input", array_of(reference("UserInput"))), + ("clientUserMessageId", json!({ "type": ["string", "null"] })), + ], + ), + ); + definitions.insert( + "TurnStartResponse".into(), + object_schema(&["turn"], &[("turn", reference("Turn"))]), + ); + definitions.insert( + "TurnSteerResponse".into(), + object_schema(&["turnId"], &[("turnId", json!({ "type": "string" }))]), + ); + definitions.insert( + "ThreadLoadedListResponse".into(), + object_schema( + &["data"], + &[("data", array_of(json!({ "type": "string" })))], + ), + ); + definitions.insert( + "HooksListParams".into(), + object_schema(&[], &[("cwds", array_of(json!({ "type": "string" })))]), + ); + definitions.insert( + "HooksListResponse".into(), + object_schema( + &["data"], + &[("data", array_of(reference("HooksListEntry")))], + ), + ); + definitions.insert( + "HooksListEntry".into(), + object_schema( + &["hooks"], + &[("hooks", array_of(reference("HookMetadata")))], + ), + ); + definitions.insert( + "HookMetadata".into(), + object_schema( + &["currentHash", "isManaged", "key", "trustStatus"], + &[ + ("currentHash", json!({ "type": "string" })), + ("isManaged", json!({ "type": "boolean" })), + ("key", json!({ "type": "string" })), + ("trustStatus", reference("HookTrustStatus")), + ], + ), + ); + definitions.insert( + "HookTrustStatus".into(), + json!({ + "type": "string", + "enum": ["managed", "modified", "trusted", "untrusted"] + }), + ); + CodexProtocolSchemas { + protocol: json!({ "definitions": definitions }), + client_requests: method_schema(REQUIRED_CODEX_CLIENT_REQUESTS), + client_notifications: method_schema(REQUIRED_CODEX_CLIENT_NOTIFICATIONS), + server_requests: method_schema(&["currentTime/read"]), + server_notifications: method_schema(REQUIRED_CODEX_SERVER_NOTIFICATIONS), + } +} + +fn write_fake_codex( + root: &Path, + name: &str, + version: &str, + schemas: &CodexProtocolSchemas, +) -> PathBuf { + let fixture = root.join(format!("{name}-schemas")); + fs::create_dir(&fixture).unwrap(); + for (filename, schema) in [ + ( + "codex_app_server_protocol.v2.schemas.json", + &schemas.protocol, + ), + ("ClientRequest.json", &schemas.client_requests), + ("ClientNotification.json", &schemas.client_notifications), + ("ServerRequest.json", &schemas.server_requests), + ("ServerNotification.json", &schemas.server_notifications), + ] { + fs::write(fixture.join(filename), serde_json::to_vec(schema).unwrap()).unwrap(); + } + let path = root.join(name); + fs::write( + &path, + format!( + "#!/bin/sh\nif [ \"$1\" = \"--version\" ]; then printf '%s\\n' '{version}'; exit 0; fi\nout=\nwhile [ \"$#\" -gt 0 ]; do if [ \"$1\" = \"--out\" ]; then out=$2; break; fi; shift; done\n[ -n \"$out\" ] || exit 2\ncp '{fixture}/'*.json \"$out/\"\n", + fixture = fixture.display() + ), + ) + .unwrap(); + fs::set_permissions(&path, fs::Permissions::from_mode(0o755)).unwrap(); + path +} + +#[test] +fn protocol_schema_gate_accepts_a_compatible_release_and_rejects_shape_drift() { + let tmp = tempfile::tempdir().unwrap(); + let compatible = compatible_protocol_schemas(); + let patch = write_fake_codex( + tmp.path(), + "codex-compatible-patch", + "codex-cli 0.150.0", + &compatible, + ); + ensure_supported_protocol(patch.to_str().unwrap()).unwrap(); + + let mut incompatible = compatible_protocol_schemas(); + incompatible + .protocol + .pointer_mut("/definitions/ThreadActiveFlag/enum") + .unwrap() + .as_array_mut() + .unwrap() + .push(Value::String("waitingOnFutureInput".into())); + let incompatible = write_fake_codex( + tmp.path(), + "codex-incompatible-schema", + "codex-cli 0.150.1", + &incompatible, + ); + let error = ensure_supported_protocol(incompatible.to_str().unwrap()).unwrap_err(); + assert!(format!("{error:#}").contains("ThreadActiveFlag changed")); +} + +#[test] +fn protocol_schema_gate_accepts_additive_items_and_server_requests() { + let mut schemas = compatible_protocol_schemas(); + schemas + .server_requests + .get_mut("oneOf") + .unwrap() + .as_array_mut() + .unwrap() + .push( + method_schema(&["future/request"]) + .get_mut("oneOf") + .unwrap() + .as_array_mut() + .unwrap() + .remove(0), + ); + schemas + .protocol + .pointer_mut("/definitions/ThreadItem/oneOf") + .unwrap() + .as_array_mut() + .unwrap() + .push(tagged_variant("futureItem", &[], &[])); + + verify_codex_protocol_schemas(&schemas).unwrap(); +} + +/// The classifier reads one word out of `Turn.error.codexErrorInfo` and depends on it being +/// distinct from the quota words. A release that dropped or merged it must refuse the launch +/// rather than let st2 report an exhausted allowance as a rejected credential. +#[test] +fn protocol_schema_gate_requires_the_distinct_credential_and_quota_error_words() { + let mut schemas = compatible_protocol_schemas(); + let words = schemas + .protocol + .pointer_mut("/definitions/CodexErrorInfo/oneOf/0/enum") + .unwrap() + .as_array_mut() + .unwrap(); + words.retain(|word| word.as_str() != Some("unauthorized")); + let error = verify_codex_protocol_schemas(&schemas).unwrap_err(); + assert!( + format!("{error:#}").contains("CodexErrorInfo has no 'unauthorized' word"), + "{error:#}" + ); + + let mut merged = compatible_protocol_schemas(); + merged + .protocol + .pointer_mut("/definitions/CodexErrorInfo/oneOf/0/enum") + .unwrap() + .as_array_mut() + .unwrap() + .retain(|word| word.as_str() != Some("rateLimitExceeded")); + let error = verify_codex_protocol_schemas(&merged).unwrap_err(); + assert!( + format!("{error:#}").contains("CodexErrorInfo has no 'rateLimitExceeded' word"), + "{error:#}" + ); + + let mut untyped = compatible_protocol_schemas(); + untyped + .protocol + .pointer_mut("/definitions/Turn/properties") + .unwrap() + .as_object_mut() + .unwrap() + .remove("error"); + let error = verify_codex_protocol_schemas(&untyped).unwrap_err(); + assert!( + format!("{error:#}").contains("Turn has no error property"), + "{error:#}" + ); +} + +#[test] +fn protocol_rejection_reaches_the_declared_supervisor_once() { + let tmp = tempfile::tempdir().unwrap(); + let worker = tmp.path().join("agents/h/worker/agent.kdl"); + let supervisor = tmp.path().join("agents/h/cos/agent.kdl"); + fs::create_dir_all(worker.parent().unwrap()).unwrap(); + fs::create_dir_all(supervisor.parent().unwrap()).unwrap(); + fs::write( + &worker, + r#"agent "worker" { + host "h" + supervisor "h.cos" + command "true" +} +"#, + ) + .unwrap(); + fs::write( + &supervisor, + r#"agent "cos" { + host "h" + command "true" +} +"#, + ) + .unwrap(); + let mut incompatible = compatible_protocol_schemas(); + incompatible + .protocol + .pointer_mut("/definitions/ThreadActiveFlag/enum") + .unwrap() + .as_array_mut() + .unwrap() + .push(Value::String("waitingOnFutureInput".into())); + let codex = write_fake_codex( + tmp.path(), + "codex-rejected", + "codex-cli 0.150.1", + &incompatible, + ); + let argv = vec![codex.display().to_string()]; + + for _ in 0..2 { + let error = run_controlled( + tmp.path(), + "h.worker".into(), + "h.worker".into(), + argv.clone(), + ) + .unwrap_err(); + assert!(format!("{error:#}").contains("ThreadActiveFlag changed")); + } + + let inbox = message::list_inbox(&message::inbox_dir(supervisor.parent().unwrap())).unwrap(); + assert_eq!(inbox.len(), 1, "the rejection report was not idempotent"); + assert_eq!(inbox[0].from.as_deref(), Some("h.worker")); + assert_eq!( + inbox[0].subject.as_deref(), + Some("Codex protocol rejected: h.worker") + ); + assert!(inbox[0].body.contains("Native delivery did not start")); + assert!(inbox[0].body.contains("ThreadActiveFlag changed")); +} + +#[test] +fn unknown_thread_status_remains_a_hold_not_a_terminal_system_error() { + let mut state = subscribed_state(CodexObservedState::Idle); + state.observe_thread_status("futureStatus", None); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::UnknownStatus, + turn_id: None, + } + ); + state.observe_turn_completed("turn-future", CodexTurnOutcome::Indeterminate); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::UnknownStatus, + turn_id: None, + } + ); +} + +#[test] +fn tui_loaded_deadline_precedes_the_outer_binding_deadline() { + assert!(TUI_LOADED_TIMEOUT < STARTUP_TIMEOUT); +} + +/// An agent directory with a parent to stage into, and a producer over it carrying a fixed +/// incarnation so the record's provenance is assertable. +fn context_producer(root: &Path) -> (PathBuf, CodexContextProducer) { + let agent_dir = root.join("agents/h/worker"); + fs::create_dir_all(&agent_dir).unwrap(); + let writer = + harness_context::Writer::new(&agent_dir, "h.worker", harness_context::Harness::Codex) + .unwrap() + .with_session("codex-incarnation"); + (agent_dir, CodexContextProducer::new(writer)) +} + +fn context_record(agent_dir: &Path) -> Option { + harness_context::read(&harness_context::harness_context_path(agent_dir)) +} + +fn token_usage_frame(last_total: i64, window: Value) -> Value { + json!({ + "method": "thread/tokenUsage/updated", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "tokenUsage": { + "last": { "totalTokens": last_total }, + "total": { "totalTokens": last_total }, + "modelContextWindow": window + } + } + }) +} + +fn compaction_item_frame(method: &str, turn_id: &str, item_id: &str) -> Value { + json!({ + "method": method, + "params": { + "threadId": "thread-main", + "turnId": turn_id, + "item": { "id": item_id, "type": "contextCompaction" } + } + }) +} + +/// HC-R13's Codex fixture. The frames are a transposition, and the comment says which half came +/// from where: the SHAPE is codex-cli 0.151.0's own app-server schema dump +/// (`ThreadTokenUsageUpdatedNotification`, `AccountRateLimitsUpdatedNotification`), while the +/// NUMBERS are verbatim from a real rollout captured on 2026-08-29 from a 0.150.1 session +/// (`session_meta.payload.cli_version = "0.150.1"`) — its first and last `token_count` events +/// and the `rate_limits` snapshot riding them. Fields the capture elided are omitted rather +/// than invented; this producer reads three numbers and must not need the rest. +/// +/// What must fail here when a codex bump moves something: the 12,000 baseline (the percent +/// changes), the numerator (`total` reads 100 and `last.inputTokens` without the baseline reads +/// 36 against this very capture, both asserted below), and the version literal itself, which is +/// the only thing tying this arithmetic to a build whose source was actually read. +#[test] +fn codex_context_recomputes_the_captured_reading_and_pins_its_verified_version() { + assert_eq!(CODEX_CONTEXT_VERIFIED_VERSION, "0.151.0"); + assert_eq!(CODEX_BASELINE_TOKENS, 12_000); + + let frames = include_str!("../../tests/fixtures/codex_token_usage_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + assert_eq!(frames.len(), 3); + + let tmp = tempfile::tempdir().unwrap(); + let (agent_dir, mut producer) = context_producer(tmp.path()); + + // The session's FIRST reading: 32,237 of 258,400 with the baseline normalized out is 8%, + // and no rate-limit notification has arrived yet, so both windows are honestly absent. + assert!(producer.observe(&frames[0], "thread-main").unwrap()); + let first = context_record(&agent_dir).unwrap(); + assert_eq!(first.used_tokens, Some(32_237)); + assert_eq!(first.window_tokens, Some(258_400)); + assert_eq!(first.used_percent, Some(8.0)); + assert_eq!(first.rate_limits, harness_context::RateLimits::default()); + + // The account-scoped snapshot carries no occupancy, so it writes nothing on its own and is + // held for the next reading (HC-T06). + assert!(!producer.observe(&frames[1], "thread-main").unwrap()); + let mut unchanged = context_record(&agent_dir).unwrap(); + // `age_ms` is derived at read time, not stored, so it moves between two reads of one + // record. Everything the record itself carries — including `observed_at_ms`, which is what + // proves no write happened — must be identical. + assert!(unchanged.age_ms >= first.age_ms); + unchanged.age_ms = first.age_ms; + assert_eq!(unchanged, first); + + assert!(producer.observe(&frames[2], "thread-main").unwrap()); + let observed = context_record(&agent_dir).unwrap(); + assert_eq!(observed.harness, harness_context::Harness::Codex); + assert_eq!(observed.used_tokens, Some(92_283)); + assert_eq!(observed.window_tokens, Some(258_400)); + // 100 − Codex's displayed "67% context left" for this exact capture. + assert_eq!(observed.used_percent, Some(33.0)); + assert_eq!(observed.session_total_tokens, Some(2_235_329)); + // The channel carries neither: `Thread` has `modelProvider` and no model identifier, and + // Codex reports no session cost anywhere in the protocol. + assert_eq!(observed.model, None); + assert_eq!(observed.cost_usd, None); + // Only the seven-day window was ever captured on this harness; the five-hour leg is not + // inferred from a field name (see `observe_rate_limits`). + assert_eq!( + observed.rate_limits, + harness_context::RateLimits { + five_hour: None, + seven_day: Some(44.0), + } + ); + assert_eq!(observed.compactions, 0); + assert_eq!(observed.last_compaction_ms, None); + + // The trap, asserted rather than described: the cumulative session total is 2,235,329 + // against a 258,400-token window. A producer that used it as the numerator would publish a + // saturated 100 for a window that is a third full. + assert_eq!(codex_used_percent(Some(258_400), 2_235_329), Some(100.0)); + assert_ne!( + codex_used_percent(Some(258_400), 2_235_329), + observed.used_percent + ); + // And the baseline-free percent over the same operands is 36 — close enough to look right. + let baseline_free = (92_283.0_f64 / 258_400.0 * 100.0).round(); + assert_eq!(baseline_free, 36.0); + assert_ne!(Some(baseline_free), observed.used_percent); + + // Mirroring is not the same function as rounding the used percentage: at an exact half + // they disagree. Effective window 200, used 101 — Codex displays 50% left, so st2 publishes + // 50; rounding `used/effective` would publish 51. + assert_eq!(codex_used_percent(Some(12_200), 12_101), Some(50.0)); + assert_eq!((101.0_f64 / 200.0 * 100.0).round(), 51.0); +} + +/// HC-R02/HC-R03: the operands are the harness's and are published as they arrive; only the +/// percent is withheld, and only where Codex's own normalization cannot run. A window at or +/// below the baseline is the sharp case — Codex itself returns "0% remaining" there, which +/// mirrored blindly would publish a fabricated 100% used. +#[test] +fn a_missing_or_unnormalizable_window_withholds_the_percent_but_not_the_operands() { + for (window, expected_window) in [ + (Value::Null, None), + (json!(12_000), Some(12_000)), + (json!(0), None), + ] { + let tmp = tempfile::tempdir().unwrap(); + let (agent_dir, mut producer) = context_producer(tmp.path()); + assert!( + producer + .observe(&token_usage_frame(92_283, window.clone()), "thread-main") + .unwrap() + ); + let observed = context_record(&agent_dir).unwrap(); + assert_eq!(observed.used_tokens, Some(92_283), "window {window}"); + assert_eq!(observed.window_tokens, expected_window, "window {window}"); + assert_eq!(observed.used_percent, None, "window {window}"); + } + + // A window key that is absent rather than null reads the same way. + let tmp = tempfile::tempdir().unwrap(); + let (agent_dir, mut producer) = context_producer(tmp.path()); + assert!( + producer + .observe( + &json!({ + "method": "thread/tokenUsage/updated", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "tokenUsage": { + "last": { "totalTokens": 92_283 }, + "total": { "totalTokens": 92_283 } + } + } + }), + "thread-main", + ) + .unwrap() + ); + let observed = context_record(&agent_dir).unwrap(); + assert_eq!(observed.window_tokens, None); + assert_eq!(observed.used_percent, None); +} + +/// Codex speaks once per model response — roughly 10-15 times a turn, and again on resume or +/// re-attach. The core's quantization is the ONLY thing deciding what lands (HC-R09): this +/// producer holds no reading of its own and imposes no cadence. The shape that catches a second +/// guard is the last frame here — a bucket crossing arriving immediately after two suppressed +/// readings, which any time floor in the producer would swallow. +#[test] +fn every_reading_reaches_the_core_guard_and_the_producer_imposes_no_cadence_of_its_own() { + let tmp = tempfile::tempdir().unwrap(); + let (agent_dir, mut producer) = context_producer(tmp.path()); + let window = json!(258_400); + + assert!( + producer + .observe(&token_usage_frame(92_283, window.clone()), "thread-main") + .unwrap() + ); + assert_eq!(context_record(&agent_dir).unwrap().used_percent, Some(33.0)); + + // Both still round to 33% used, so both sit in the written bucket and neither lands. + for moved in [93_000, 94_000] { + assert_eq!(codex_used_percent(Some(258_400), moved), Some(33.0)); + assert!( + !producer + .observe(&token_usage_frame(moved, window.clone()), "thread-main") + .unwrap() + ); + assert_eq!( + context_record(&agent_dir).unwrap().used_tokens, + Some(92_283) + ); + } + + // The crossing lands at once, with no elapsed time behind it. + assert!( + producer + .observe(&token_usage_frame(95_000, window.clone()), "thread-main") + .unwrap() + ); + let observed = context_record(&agent_dir).unwrap(); + assert_eq!(observed.used_percent, Some(34.0)); + assert_eq!(observed.used_tokens, Some(95_000)); + + // A reading for another thread is not this seat's. + assert!( + !producer + .observe(&token_usage_frame(200_000, window), "thread-other") + .unwrap() + ); + assert_eq!( + context_record(&agent_dir).unwrap().used_tokens, + Some(95_000) + ); +} + +/// HC-R12: one compaction is one count, however many of its spellings arrive. Codex publishes +/// the live edge as an `item/started` AND an `item/completed` over the same +/// `ContextCompactionThreadItem` id, and the protocol still carries a deprecated +/// `thread/compacted` notification for the same event that names only the turn. +#[test] +fn one_compaction_is_counted_once_across_every_spelling_of_its_edge() { + let tmp = tempfile::tempdir().unwrap(); + let (agent_dir, mut producer) = context_producer(tmp.path()); + + assert!( + producer + .observe( + &compaction_item_frame("item/started", "turn-1", "item-a"), + "thread-main" + ) + .unwrap() + ); + let first = context_record(&agent_dir).unwrap(); + assert_eq!(first.compactions, 1); + assert_eq!( + first.last_compaction_trigger, + Some(harness_context::CompactionTrigger::Unknown), + "the item carries an id and a type and no reason at all" + ); + assert!(first.last_compaction_ms.is_some()); + + // The same compaction's closing edge, and the deprecated notification for the same event. + assert!( + !producer + .observe( + &compaction_item_frame("item/completed", "turn-1", "item-a"), + "thread-main" + ) + .unwrap() + ); + assert!( + !producer + .observe( + &json!({ + "method": "thread/compacted", + "params": { "threadId": "thread-main", "turnId": "turn-1" } + }), + "thread-main", + ) + .unwrap() + ); + assert_eq!(context_record(&agent_dir).unwrap().compactions, 1); + + // A genuinely second compaction inside the same turn is a second count. + assert!( + producer + .observe( + &compaction_item_frame("item/started", "turn-1", "item-b"), + "thread-main" + ) + .unwrap() + ); + assert_eq!(context_record(&agent_dir).unwrap().compactions, 2); + + // Interleaved lifecycles: two starts before either completion still count exactly two, so + // the dedupe cannot be a single last-key memory. + for (method, item) in [ + ("item/started", "item-c"), + ("item/started", "item-d"), + ("item/completed", "item-c"), + ("item/completed", "item-d"), + ] { + producer + .observe( + &compaction_item_frame(method, "turn-2", item), + "thread-main", + ) + .unwrap(); + } + assert_eq!(context_record(&agent_dir).unwrap().compactions, 4); + + // The deprecated notification arriving FIRST also claims the compaction, so the item that + // follows it does not count a second time. + assert!( + producer + .observe( + &json!({ + "method": "thread/compacted", + "params": { "threadId": "thread-main", "turnId": "turn-3" } + }), + "thread-main", + ) + .unwrap() + ); + assert!( + !producer + .observe( + &compaction_item_frame("item/started", "turn-3", "item-e"), + "thread-main" + ) + .unwrap() + ); + assert_eq!(context_record(&agent_dir).unwrap().compactions, 5); + + // Another thread's compaction is not this seat's, and a non-compaction item is not an edge. + assert!( + !producer + .observe( + &compaction_item_frame("item/started", "turn-9", "item-z"), + "thread-other" + ) + .unwrap() + ); + assert!( + !producer + .observe( + &json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-4", + "item": { "id": "item-y", "type": "agentMessage" } + } + }), + "thread-main", + ) + .unwrap() + ); + assert_eq!(context_record(&agent_dir).unwrap().compactions, 5); +} + +/// The producer runs beside a live delivery loop and sees every frame that loop sees. Replaying +/// the captured #263 session — 23 real inbound frames, none of them a token count — must leave +/// no record at all: absence here is "never observed", and a producer that manufactured a +/// reading from a turn boundary would break exactly the HC-R03 rule the record exists for. +#[test] +fn captured_delivery_frames_carrying_no_token_count_publish_no_record() { + let frames = include_str!("../../tests/fixtures/codex_usage_limit_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + assert_eq!(frames.len(), 23); + + let tmp = tempfile::tempdir().unwrap(); + let (agent_dir, mut producer) = context_producer(tmp.path()); + for frame in &frames { + assert!( + !producer.observe(frame, "thread-main").unwrap(), + "no captured delivery frame carries a context reading: {frame}" + ); + } + assert!(context_record(&agent_dir).is_none()); +} + +fn delivery_config(root: &Path) -> CodexDeliveryConfig { + let agent_dir = root.join("agents/h/worker"); + CodexDeliveryConfig { + catalog_root: root.to_path_buf(), + inbox: message::inbox_dir(&agent_dir), + agent_dir, + identity: "h.worker".into(), + this_host: "h".into(), + supervisor: None, + producer_version: Some("codex-cli 0.153.0".into()), + } +} + +fn subscribed_state(observed: CodexObservedState) -> CodexControlState { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state.subscribed = true; + state.observed = observed; + state +} + +fn inbox_delivery(root: &Path, config: CodexDeliveryConfig) -> CodexInboxDelivery { + CodexInboxDelivery::new( + config, + root.join("state").join(delivery_ledger::LEDGER_FILE), + CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(), + ) + .unwrap() +} + +/// Read the ledger back through its own loader and the real correlation derivation: a test +/// that read the bytes directly would not notice a record the pump itself would refuse. +fn ledger_entry(root: &Path, filename: &str) -> Option { + delivery_ledger::Ledger::open( + &root.join("state").join(delivery_ledger::LEDGER_FILE), + delivery_ledger::Harness::Codex.profile(), + "h.worker", + "h.worker", + |thread, file| stable_client_user_message_id("h.worker", thread, file), + ) + .entry(filename) + .cloned() +} + +fn acknowledge_tui_thread_loaded(events: &Receiver) { + let ControlEvent::TuiThreadLoaded(acknowledge) = + events.recv_timeout(Duration::from_secs(10)).unwrap() + else { + panic!("control did not report the TUI-loaded gate"); + }; + acknowledge.send(()).unwrap(); +} + +#[test] +fn delivery_request_uses_typed_start_and_exact_turn_steer() { + let start = codex_delivery_request( + 2, + "thread-main", + "st2:client", + "notice", + &CodexDeliveryMethod::Start, + ); + assert_eq!(start["method"], "turn/start"); + assert_eq!(start["params"]["threadId"], "thread-main"); + assert_eq!(start["params"]["clientUserMessageId"], "st2:client"); + assert_eq!(start["params"]["input"][0]["type"], "text"); + assert_eq!(start["params"]["input"][0]["text"], "notice"); + assert!(start["params"].get("expectedTurnId").is_none()); + + let steer = codex_delivery_request( + 3, + "thread-main", + "st2:client", + "notice", + &CodexDeliveryMethod::Steer { + turn_id: "turn-current".into(), + }, + ); + assert_eq!(steer["method"], "turn/steer"); + assert_eq!(steer["params"]["expectedTurnId"], "turn-current"); + assert!(steer["params"].get("model").is_none()); + assert!(steer["params"].get("approvalPolicy").is_none()); +} + +/// Behavioral oracle for the #268 §B projection: a projection that withheld every row — or +/// that reported the two misclassified rows as indeterminate — fails here, because each +/// emitting row is asserted positively. +#[test] +fn harness_projection_is_faithful_and_withholds_only_unprovable_rows() { + use crate::harness_state::{Activity, Ask, BlockedOn, InputBuffer}; + let held = |reason| CodexObservedState::Held { + reason, + turn_id: None, + }; + + // Rows with no provable observation are withheld — and no absence may derive idle. + for state in [ + CodexObservedState::AwaitingStatus, + held(CodexHoldReason::NotLoaded), + held(CodexHoldReason::SystemError), + ] { + assert_eq!(state.harness_observation(), None, "{state:?}"); + } + + // Codex positively reported work: active, even where st2 cannot name a steerable turn + // (the two rows a naive steerability decomposition reported as unknown) or where the + // delivery gate holds. + for state in [ + CodexObservedState::Active { + turn_id: "turn-current".into(), + }, + held(CodexHoldReason::ActiveWithoutTurn), + held(CodexHoldReason::ConflictingTurn), + held(CodexHoldReason::Compaction), + // Review's edges are model-emitted items inside a running turn: plain activity, + // no human, no ask — the delivery hold is a separate axis. + held(CodexHoldReason::Review), + ] { + let observation = state + .harness_observation() + .unwrap_or_else(|| panic!("{state:?} must emit")); + assert_eq!(observation.state, Activity::Active, "{state:?}"); + assert_eq!(observation.blocked_on, BlockedOn::None, "{state:?}"); + assert_eq!(observation.input_buffer, InputBuffer::Unknown, "{state:?}"); + } + + // The holds a human resolves set the blocked axis instead of disappearing into active, + // and each names its machine-readable ask kind so consumers never branch on `reason`. + for (reason, ask) in [ + (CodexHoldReason::WaitingOnApproval, Ask::Permission), + (CodexHoldReason::WaitingOnUserInput, Ask::Question), + ] { + let observation = held(reason) + .harness_observation() + .unwrap_or_else(|| panic!("{reason:?} must emit")); + assert_eq!(observation.state, Activity::Active, "{reason:?}"); + assert_eq!(observation.blocked_on, BlockedOn::Human, "{reason:?}"); + assert_eq!(observation.ask, ask, "{reason:?}"); + } + + let idle = CodexObservedState::Idle.harness_observation().unwrap(); + assert_eq!(idle.state, Activity::Idle); + assert_eq!(idle.blocked_on, BlockedOn::None); + + let ended = CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + .harness_observation() + .unwrap(); + assert_eq!(ended.state, Activity::Ended); + assert_eq!(ended.reason.as_deref(), Some("systemError")); +} + +#[test] +#[cfg(unix)] +fn a_failed_transition_write_is_retried_before_any_heartbeat() { + use crate::harness_state::{self, Activity}; + use std::os::unix::fs::PermissionsExt as _; + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let agent_dir = config.agent_dir.clone(); + let record_path = harness_state::harness_state_path(&agent_dir); + let mut delivery = inbox_delivery(tmp.path(), config); + + delivery.observe_harness(&CodexObservedState::Active { + turn_id: "turn-current".into(), + }); + assert_eq!( + harness_state::read(&record_path, None).unwrap().state, + Activity::Active + ); + + // The transition to idle fails to land: the agent dir is briefly unwritable. + let live = fs::metadata(&agent_dir).unwrap().permissions(); + fs::set_permissions(&agent_dir, fs::Permissions::from_mode(0o555)).unwrap(); + delivery.observe_harness(&CodexObservedState::Idle); + fs::set_permissions(&agent_dir, live).unwrap(); + assert_eq!( + harness_state::read(&record_path, None).unwrap().state, + Activity::Active, + "the failed write cannot have landed" + ); + + // No heartbeat may re-stamp the contradicted on-disk state; the retry lands the pending + // transition on the NEXT pump pass — deliberately without advancing the presence + // cadence, which gates only heartbeats. + let stale_active = fs::read(&record_path).unwrap(); + delivery.next_presence_refresh = Instant::now() + status::STATUS_REFRESH; + delivery.refresh_if_due().unwrap(); + let after = harness_state::read(&record_path, None).unwrap(); + assert_eq!(after.state, Activity::Idle, "pending transition retried"); + assert_ne!(fs::read(&record_path).unwrap(), stale_active); +} + +#[test] +fn pump_publishes_observations_and_stops_heartbeating_on_evidence_loss() { + use crate::harness_state::{self, Activity}; + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let agent_dir = config.agent_dir.clone(); + let record_path = harness_state::harness_state_path(&agent_dir); + let mut delivery = inbox_delivery(tmp.path(), config); + + delivery.observe_harness(&CodexObservedState::Active { + turn_id: "turn-current".into(), + }); + let observed = harness_state::read(&record_path, None).expect("record written"); + assert_eq!(observed.state, Activity::Active); + assert_eq!(observed.harness.as_deref(), Some("codex")); + + // An indeterminate projection writes nothing and stops the heartbeat: the presence + // refresh still runs, but the record's bytes stay untouched and age toward unknown. + delivery.observe_harness(&CodexObservedState::Held { + reason: CodexHoldReason::NotLoaded, + turn_id: None, + }); + let before = fs::read(&record_path).unwrap(); + delivery.refresh_if_due().unwrap(); + assert!( + status::read_state(&status::status_path(&agent_dir)) != status::State::Offline, + "presence refresh must still run" + ); + assert_eq!( + fs::read(&record_path).unwrap(), + before, + "no heartbeat without evidence" + ); + + // Evidence returning resumes both observation and heartbeat. + delivery.observe_harness(&CodexObservedState::Idle); + assert_eq!( + harness_state::read(&record_path, None).unwrap().state, + Activity::Idle + ); + delivery.next_presence_refresh = Instant::now(); + delivery.refresh_if_due().unwrap(); + assert_ne!( + fs::read(&record_path).unwrap(), + before, + "heartbeat resumes with evidence" + ); +} + +#[test] +fn evidence_loss_marks_the_stream_discontinuous_for_a_restated_state() { + use crate::harness_state::{self, Activity}; + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let record_path = harness_state::harness_state_path(&config.agent_dir); + let mut delivery = inbox_delivery(tmp.path(), config); + + delivery.observe_harness(&CodexObservedState::Active { + turn_id: "turn-a".into(), + }); + let before = fs::read(&record_path).unwrap(); + + // The same tuple restated across an unproven interval must not coalesce into the + // pre-gap record — continuity was not observed, so a fresh transition opens. + delivery.observe_harness(&CodexObservedState::Held { + reason: CodexHoldReason::SystemError, + turn_id: None, + }); + delivery.observe_harness(&CodexObservedState::Active { + turn_id: "turn-a".into(), + }); + assert_ne!( + fs::read(&record_path).unwrap(), + before, + "a restated state after an evidence gap must open a fresh transition" + ); + assert_eq!( + harness_state::read(&record_path, None).unwrap().state, + Activity::Active + ); +} + +#[test] +fn delivery_client_id_is_stable_and_binds_every_identity_component() { + let id = + stable_client_user_message_id("h.worker", "thread-main", "1786380000000-abc123.md"); + assert_eq!( + id, + stable_client_user_message_id("h.worker", "thread-main", "1786380000000-abc123.md") + ); + assert!(id.starts_with("st2:")); + assert_ne!( + id, + stable_client_user_message_id("h.other", "thread-main", "1786380000000-abc123.md") + ); + assert_ne!( + id, + stable_client_user_message_id("h.worker", "thread-other", "1786380000000-abc123.md") + ); + assert_ne!( + id, + stable_client_user_message_id("h.worker", "thread-main", "1786380000000-def456.md") + ); +} + +#[test] +fn review_compaction_and_dnd_hold_the_unread_fifo_head() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + for reason in [CodexHoldReason::Review, CodexHoldReason::Compaction] { + let state = subscribed_state(CodexObservedState::Held { + reason, + turn_id: Some("turn-current".into()), + }); + assert_eq!(delivery.maybe_request(&state).unwrap(), None); + assert!(config.inbox.join(&filename).is_file()); + } + + status::set_state(&status::status_path(&config.agent_dir), status::State::Dnd).unwrap(); + delivery.next_inbox_refresh = Instant::now(); + assert_eq!( + delivery + .maybe_request(&subscribed_state(CodexObservedState::Idle)) + .unwrap(), + None + ); + assert_eq!(message::list_inbox(&config.inbox).unwrap().len(), 1); +} + +#[test] +fn failed_turn_without_idle_allows_next_native_delivery_and_preserves_system_error() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + message::send_to_inbox( + &config.inbox, + "h.sender", + Some("after error"), + None, + &[], + "body", + ) + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config); + let mut state = subscribed_state(CodexObservedState::Idle); + + state + .observe(&json!({ + "method": "turn/started", + "params": { + "threadId": "thread-main", + "turn": { "id": "turn-failed" } + } + })) + .unwrap(); + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "systemError" } + } + })) + .unwrap(); + state + .observe(&json!({ + "method": "turn/completed", + "params": { + "threadId": "thread-main", + "turn": { "id": "turn-failed", "status": "failed" } + } + })) + .unwrap(); + + assert_eq!( + state.observed(), + &CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + ); + + let request = delivery + .maybe_request(&state) + .unwrap() + .expect("a terminal system error must not block the next native delivery"); + assert_eq!(request["method"], "turn/start"); +} + +#[test] +fn captured_usage_limit_boundary_allows_next_native_delivery() { + // This fixture is a payload-minimized projection of all 23 inbound frames from the + // #263 trivial capture. It preserves their order and methods while removing fields this + // observer never reads. The second capture has the same method sequence. The recorder + // stops at turn completion, so this test pins the boundary state only. The provider + // source establishes that no later idle notification follows the system error. + let frames = include_str!("../../tests/fixtures/codex_usage_limit_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + assert_eq!(frames.len(), 23); + + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + message::send_to_inbox( + &config.inbox, + "h.sender", + Some("after capture"), + None, + &[], + "body", + ) + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config); + let mut state = subscribed_state(CodexObservedState::AwaitingStatus); + + for frame in &frames { + state.observe(frame).unwrap(); + } + + assert_eq!( + frames + .last() + .and_then(|frame| frame.get("method")) + .and_then(Value::as_str), + Some("turn/completed") + ); + assert_eq!( + state.observed(), + &CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + ); + let request = delivery + .maybe_request(&state) + .unwrap() + .expect("a captured terminal system error must permit the next native delivery"); + assert_eq!(request["method"], "turn/start"); +} + +/// The credential class and the quota class arrive through the SAME frame sequence, differing +/// only in one word of `Turn.error.codexErrorInfo`. This replays the auth-rejected shape and +/// asserts the fork: `providerAuth` on the observed record, a native-driver diagnostic, and +/// delivery still permitted — while the captured usage-limit fixture beside it keeps reading +/// `systemError` with no diagnostic at all. +#[test] +fn a_rejected_codex_credential_reads_provider_auth_while_a_quota_failure_does_not() { + let rejected = include_str!("../../tests/fixtures/codex_provider_auth_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + let quota = include_str!("../../tests/fixtures/codex_usage_limit_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let agent_dir = config.agent_dir.clone(); + message::send_to_inbox( + &config.inbox, + "h.sender", + Some("after rejection"), + None, + &[], + "body", + ) + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config); + let mut state = subscribed_state(CodexObservedState::AwaitingStatus); + for frame in &rejected { + state.observe(frame).unwrap(); + delivery.observe_provider_auth(frame, "thread-main"); + } + + assert_eq!( + state.observed(), + &CodexObservedState::TerminalError { + reason: CodexTerminalError::ProviderAuthRejected, + } + ); + let observation = state.observed().harness_observation().unwrap(); + assert_eq!(observation.state, harness_state::Activity::Ended); + assert_eq!( + observation.reason.as_deref(), + Some("providerAuth"), + "the same word OpenCode's ProviderAuthError already publishes" + ); + let record = driver_diagnostic::path(&agent_dir); + let driver_diagnostic::Observed::Failure(failure) = driver_diagnostic::read(&record) else { + panic!("a rejected credential must publish a native-driver diagnostic") + }; + assert_eq!(failure.driver, driver_diagnostic::Driver::Codex); + assert_eq!(failure.stage, driver_diagnostic::Stage::ProviderAuth); + assert_eq!( + failure.reason, + driver_diagnostic::Reason::ProviderAuthRejected + ); + assert_eq!(failure.source, driver_diagnostic::Source::TurnResult); + assert_eq!( + failure.producer_version.as_deref(), + Some("codex-cli 0.153.0") + ); + assert_eq!(failure.support, driver_diagnostic::Support::Supported); + let request = delivery + .maybe_request(&state) + .unwrap() + .expect("a rejected credential must not block the next native delivery"); + assert_eq!(request["method"], "turn/start"); + + // A turn that reaches its ordinary end is the recovery edge. + delivery.observe_provider_auth( + &json!({ + "method": "turn/completed", + "params": { + "threadId": "thread-main", + "turn": { "id": "turn-ok", "status": "completed" } + } + }), + "thread-main", + ); + assert_eq!( + driver_diagnostic::read(&record), + driver_diagnostic::Observed::Absent + ); + + // The quota capture walks the same methods and must stay unclassified. + let quota_tmp = tempfile::tempdir().unwrap(); + let quota_config = delivery_config(quota_tmp.path()); + let quota_agent_dir = quota_config.agent_dir.clone(); + let mut quota_delivery = inbox_delivery(quota_tmp.path(), quota_config); + let mut quota_state = subscribed_state(CodexObservedState::AwaitingStatus); + for frame in "a { + quota_state.observe(frame).unwrap(); + quota_delivery.observe_provider_auth(frame, "thread-main"); + } + assert_eq!( + quota_state.observed(), + &CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + ); + assert_eq!( + driver_diagnostic::read(&driver_diagnostic::path("a_agent_dir)), + driver_diagnostic::Observed::Absent, + "an exhausted allowance is not a rejected credential" + ); +} + +#[test] +fn idle_session_refreshes_stale_presence_without_inbox_activity() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let presence = status::status_path(&config.agent_dir); + std::fs::create_dir_all(&config.agent_dir).unwrap(); + std::fs::write(&presence, "available\n").unwrap(); + std::fs::File::open(&presence) + .unwrap() + .set_modified(SystemTime::now() - status::STATUS_STALE - Duration::from_secs(1)) + .unwrap(); + assert_eq!(status::read_state(&presence), status::State::Unknown); + + let mut delivery = inbox_delivery(tmp.path(), config); + delivery.refresh_if_due().unwrap(); + + assert_eq!(status::read_state(&presence), status::State::Available); + assert!( + std::fs::read_to_string(&presence) + .unwrap() + .contains("\nv1 ") + ); + assert!(delivery.head.is_none()); +} + +#[test] +fn inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let presence = status::status_path(&config.agent_dir); + status::set_state(&presence, status::State::Available).unwrap(); + let before = std::fs::read_to_string(&presence).unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config); + delivery.next_inbox_refresh = Instant::now(); + delivery.next_presence_refresh = Instant::now() + status::STATUS_REFRESH; + + delivery.refresh_if_due().unwrap(); + + assert_eq!(std::fs::read_to_string(&presence).unwrap(), before); +} + +#[test] +fn a_rejected_exact_steer_has_no_fallback_and_remains_retryable_after_state_changes() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("retry"), None, &[], "body") + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + let active = subscribed_state(CodexObservedState::Active { + turn_id: "turn-current".into(), + }); + let steer = delivery.maybe_request(&active).unwrap().unwrap(); + assert_eq!(steer["method"], "turn/steer"); + assert_eq!(steer["params"]["expectedTurnId"], "turn-current"); + let request_id = steer["id"].clone(); + let client_id = steer["params"]["clientUserMessageId"].clone(); + + assert!( + !delivery + .accept_response( + &json!({ + "id": request_id, + "method": "item/commandExecution/requestApproval", + "params": {} + }), + active.observed(), + ) + .unwrap() + ); + assert!(delivery + .accept_response( + &json!({ "id": request_id, "error": { "code": -32600, "message": "stale turn" } }), + active.observed(), + ) + .unwrap()); + assert_eq!(delivery.maybe_request(&active).unwrap(), None); + assert!(config.inbox.join(&filename).is_file()); + + let retry = delivery + .maybe_request(&subscribed_state(CodexObservedState::Idle)) + .unwrap() + .unwrap(); + assert_eq!(retry["method"], "turn/start"); + assert_eq!(retry["params"]["clientUserMessageId"], client_id); + assert!(config.inbox.join(&filename).is_file()); +} + +#[test] +fn a_success_response_is_only_an_attempt_and_does_not_archive_the_message() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = message::send_to_inbox( + &config.inbox, + "h.sender", + Some("submitted"), + None, + &[], + "body", + ) + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + let idle = subscribed_state(CodexObservedState::Idle); + let request = delivery.maybe_request(&idle).unwrap().unwrap(); + assert_eq!( + delivery.ledger.entry(&filename).unwrap().phase, + delivery_ledger::Phase::Attempted, + "submission ownership is durable before transport" + ); + assert!( + delivery + .accept_response( + &json!({ "id": request["id"], "result": { "turn": { "id": "turn-new" } } }), + idle.observed(), + ) + .unwrap() + ); + assert_eq!( + delivery.ledger.entry(&filename).unwrap().phase, + delivery_ledger::Phase::TransportAccepted, + "a well-formed JSON result is transport, never typed acceptance" + ); + assert_eq!(delivery.maybe_request(&idle).unwrap(), None); + assert!(config.inbox.join(&filename).is_file()); +} + +#[test] +fn only_a_completed_matching_user_message_persists_acceptance() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = message::send_to_inbox( + &config.inbox, + "h.sender", + Some("receipt"), + None, + &[], + "body", + ) + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + let mut idle = CodexControlState::new(&delivery.runtime, "thread-main".into()); + idle.subscribed = true; + idle.observed = CodexObservedState::Idle; + let request = delivery.maybe_request(&idle).unwrap().unwrap(); + let client_id = request["params"]["clientUserMessageId"] + .as_str() + .unwrap() + .to_string(); + + assert!( + !delivery + .accept_typed_receipt( + &json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-delivery", + "item": { "type": "userMessage", "clientId": client_id } + } + }), + &idle, + ) + .unwrap(), + "item/started is progress, not acceptance" + ); + assert!( + !delivery + .accept_typed_receipt( + &json!({ + "method": "item/completed", + "params": { + "threadId": "thread-other", + "turnId": "turn-delivery", + "item": { "type": "userMessage", "clientId": client_id } + } + }), + &idle, + ) + .unwrap(), + "another thread cannot acknowledge this delivery" + ); + assert!( + delivery + .accept_typed_receipt( + &json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-delivery", + "item": { "type": "userMessage", "clientId": client_id } + } + }), + &idle, + ) + .unwrap() + ); + assert_eq!( + ledger_entry(tmp.path(), &filename).unwrap().phase, + delivery_ledger::Phase::Consumed + ); + assert!(config.inbox.join(&filename).is_file()); + + drop(delivery); + let mut replacement = inbox_delivery(tmp.path(), config.clone()); + assert_eq!( + replacement.maybe_request(&idle).unwrap(), + None, + "a fresh runtime incarnation restores accepted duplicate control" + ); + + message::archive_msg( + &config.inbox, + &message::archive_dir(&config.agent_dir), + &filename, + ) + .unwrap(); + replacement.next_inbox_refresh = Instant::now(); + assert_eq!(replacement.maybe_request(&idle).unwrap(), None); + assert!( + ledger_entry(tmp.path(), &filename).is_none(), + "archive precedence — the recipient agent's own act — releases the ledger entry" + ); +} + +#[test] +fn an_ambiguous_attempt_reconciles_resume_history_before_retry() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = message::send_to_inbox( + &config.inbox, + "h.sender", + Some("reconcile"), + None, + &[], + "body", + ) + .unwrap(); + let idle = subscribed_state(CodexObservedState::Idle); + let mut first = inbox_delivery(tmp.path(), config.clone()); + let request = first.maybe_request(&idle).unwrap().unwrap(); + let client_id = request["params"]["clientUserMessageId"] + .as_str() + .unwrap() + .to_string(); + drop(first); + + let mut recovered = inbox_delivery(tmp.path(), config.clone()); + assert_eq!(recovered.maybe_request(&idle).unwrap(), None); + recovered + .reconcile_resume( + &json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "result": { + "thread": { + "id": "thread-main", + "turns": [{ + "id": "turn-delivery", + "items": [{ + "type": "userMessage", + "id": "item-delivery", + "clientId": client_id, + "content": [] + }] + }] + } + } + }), + &idle, + ) + .unwrap(); + assert_eq!( + recovered.ledger.entry(&filename).unwrap().phase, + delivery_ledger::Phase::Consumed, + "a resumed history carrying the client ID is the same typed receipt, found late" + ); + assert_eq!(recovered.maybe_request(&idle).unwrap(), None); + assert!(config.inbox.join(&filename).is_file()); + + // An authoritative resumed history WITHOUT the client ID proves the pre-crash attempt + // never landed. Only that absence may re-authorize the same stable ID — so it needs its + // own scenario, because the delivery above is settled and can never be un-settled. + let absent_tmp = tempfile::tempdir().unwrap(); + let absent_config = delivery_config(absent_tmp.path()); + let absent_filename = message::send_to_inbox( + &absent_config.inbox, + "h.sender", + Some("absent"), + None, + &[], + "body", + ) + .unwrap(); + let mut attempted = inbox_delivery(absent_tmp.path(), absent_config.clone()); + let absent_client_id = attempted.maybe_request(&idle).unwrap().unwrap() + ["params"]["clientUserMessageId"] + .as_str() + .unwrap() + .to_string(); + drop(attempted); + + let mut replacement = inbox_delivery(absent_tmp.path(), absent_config); + assert_eq!( + replacement.maybe_request(&idle).unwrap(), + None, + "an ambiguous attempt is held and surfaced, never replayed on its own" + ); + replacement + .reconcile_resume( + &json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "result": { "thread": { "id": "thread-main", "turns": [] } } + }), + &idle, + ) + .unwrap(); + assert_eq!( + replacement + .ledger + .entry(&absent_filename) + .unwrap() + .negative, + Some(delivery_ledger::NegativeReceipt::Absent), + "the absence is retained as evidence, not erased" + ); + let retry = replacement.maybe_request(&idle).unwrap().unwrap(); + assert_eq!(retry["params"]["clientUserMessageId"], absent_client_id); +} + + +#[test] +fn subscribed_control_pump_delivers_a_typed_reference_to_the_real_fifo_head() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("wired"), None, &[], "body") + .unwrap(); + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let server_filename = filename.clone(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + stream + // Parallel Darwin test runs can deschedule the in-process peer + // for longer than the Linux-oriented two-second budget. + .set_read_timeout(Some(Duration::from_secs(10))) + .unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialize" + ); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialized" + ); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/started", + "params": { "thread": { "id": "thread-main", "status": { "type": "idle" } } } + }), + ) + .unwrap(); + let delivery = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(delivery["id"], FIRST_DELIVERY_REQUEST_ID); + assert_eq!(delivery["method"], "turn/start"); + assert_eq!(delivery["params"]["threadId"], "thread-main"); + let head_id = server_filename + .trim_end_matches(".md") + .rsplit_once('-') + .unwrap() + .1; + assert!( + delivery["params"]["input"][0]["text"] + .as_str() + .unwrap() + .contains(head_id), + "the transport payload must identify the actionable FIFO head" + ); + assert_eq!( + delivery["params"]["clientUserMessageId"], + stable_client_user_message_id("h.worker", "thread-main", &server_filename) + ); + let client_id = delivery["params"]["clientUserMessageId"] + .as_str() + .unwrap() + .to_string(); + write_json_message( + &mut websocket, + &json!({ + "id": FIRST_DELIVERY_REQUEST_ID, + "result": { "turn": { "id": "turn-delivery" } } + }), + ) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-delivery", + "item": { + "type": "userMessage", + "id": "item-delivery", + "clientId": client_id, + "content": [] + } + } + }), + ) + .unwrap(); + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let binding_path = tmp.path().join("state/binding.json"); + let control_state_path = tmp.path().join("state/control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let runtime_for_pump = runtime.clone(); + let binding_for_pump = binding_path.clone(); + let control_state_for_pump = control_state_path.clone(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_for_pump, + &control_state_for_pump, + &runtime_for_pump, + None, + Some(config), + tx, + ) + }); + assert!(matches!( + rx.recv_timeout(Duration::from_secs(10)).unwrap(), + ControlEvent::Bound + )); + server.join().unwrap(); + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + assert!(delivery_config(tmp.path()).inbox.join(&filename).is_file()); + assert_eq!( + ledger_entry(tmp.path(), &filename).unwrap().phase, + delivery_ledger::Phase::Consumed + ); +} + +/// The wiring, not the arithmetic: a `thread/tokenUsage/updated` arriving on the real control +/// socket reaches the record. Every other context test drives the producer directly, so all of +/// them would stay green if the pump stopped handing it frames — which is exactly how a +/// producer silently stops producing. +#[test] +fn the_control_pump_publishes_a_context_reading_from_a_live_token_usage_notification() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let config = delivery_config(tmp.path()); + let agent_dir = config.agent_dir.clone(); + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(10))) + .unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialize" + ); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialized" + ); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/started", + "params": { "thread": { "id": "thread-main", "status": { "type": "idle" } } } + }), + ) + .unwrap(); + write_json_message(&mut websocket, &token_usage_frame(92_283, json!(258_400))).unwrap(); + // Hold the connection open until the reading has landed: closing here would race the + // pump's read of the frame just written. Bounded, so a pump that stopped handing + // frames to the producer fails this test instead of hanging it. + let deadline = Instant::now() + Duration::from_secs(10); + while harness_context::read(&harness_context::harness_context_path(&agent_dir)) + .is_none() + && Instant::now() < deadline + { + std::thread::sleep(Duration::from_millis(10)); + } + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let binding_path = tmp.path().join("state/binding.json"); + let control_state_path = tmp.path().join("state/control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let binding_for_pump = binding_path.clone(); + let control_state_for_pump = control_state_path.clone(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_for_pump, + &control_state_for_pump, + &runtime, + None, + Some(config), + tx, + ) + }); + assert!(matches!( + rx.recv_timeout(Duration::from_secs(10)).unwrap(), + ControlEvent::Bound + )); + server.join().unwrap(); + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + + let observed = context_record(&tmp.path().join("agents/h/worker")) + .expect("the pump published nothing"); + assert_eq!(observed.harness, harness_context::Harness::Codex); + assert_eq!(observed.used_tokens, Some(92_283)); + assert_eq!(observed.window_tokens, Some(258_400)); + assert_eq!(observed.used_percent, Some(33.0)); +} + + +#[test] +fn control_initializes_before_recording_the_first_thread_only() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + let initialize = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(initialize["method"], "initialize"); + assert_eq!(initialize["params"]["clientInfo"]["name"], "st2"); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + let initialized = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(initialized["method"], "initialized"); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/started", + "params": { "thread": { "id": "thread-main", "status": { "type": "idle" } } } + }), + ) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/status/changed", + "params": { "threadId": "thread-main", "status": { "type": "idle" } } + }), + ) + .unwrap(); + // JSON-RPC request IDs are per direction. A server request may reuse the client's + // subscription ID and must not be consumed as a client response. + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "method": "item/commandExecution/requestApproval", + "params": {} + }), + ) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/started", + "params": { "thread": { "id": "thread-review", "status": { "type": "idle" } } } + }), + ) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-main" } } + }), + ) + .unwrap(); + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let state = tmp.path().join("state"); + let binding_path = state.join("binding.json"); + let control_state_path = state.join("control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let runtime_for_pump = runtime.clone(); + let binding_for_pump = binding_path.clone(); + let control_state_for_pump = control_state_path.clone(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_for_pump, + &control_state_for_pump, + &runtime_for_pump, + None, + None, + tx, + ) + }); + let first_event = rx.recv_timeout(Duration::from_secs(2)).unwrap(); + assert!( + matches!(first_event, ControlEvent::Bound), + "first control event: {first_event:?}" + ); + server.join().unwrap(); + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + + let binding = load_current_binding(&binding_path, &runtime) + .unwrap() + .unwrap(); + assert_eq!(binding.thread_id(), "thread-main"); + let state = + load_current_control_state(&state.join("control-state.json"), &runtime, &binding) + .unwrap() + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-main".into() + } + ); + assert!(state.subscribed()); +} + +#[test] +fn expected_resume_waits_for_tui_loaded_thread_and_binds_from_control_response() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let (pre_gate_checked_tx, pre_gate_checked_rx) = mpsc::channel(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + stream + .set_read_timeout(Some(Duration::from_millis(100))) + .unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + let initialize = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(initialize["method"], "initialize"); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + let initialized = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(initialized["method"], "initialized"); + assert!(matches!( + poll_json_message(&mut websocket).unwrap(), + ControlRead::Timeout + )); + pre_gate_checked_tx.send(()).unwrap(); + websocket + .get_mut() + .set_read_timeout(Some(Duration::from_millis(500))) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/started", + "params": { + "thread": { "id": "thread-unrelated", "status": { "type": "idle" } } + } + }), + ) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-unrelated", + "status": { "type": "active", "activeFlags": [] } + } + }), + ) + .unwrap(); + let first_loaded = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(first_loaded["method"], "thread/loaded/list"); + assert_eq!(first_loaded["id"], CONTROL_TUI_LOADED_REQUEST_ID); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_TUI_LOADED_REQUEST_ID, + "result": { "data": ["thread-unrelated"] } + }), + ) + .unwrap(); + let second_loaded = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(second_loaded["method"], "thread/loaded/list"); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_TUI_LOADED_REQUEST_ID, + "result": { "data": ["thread-unrelated", "thread-prior"] } + }), + ) + .unwrap(); + let subscribe = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(subscribe["method"], "thread/resume"); + assert_eq!(subscribe["params"]["threadId"], "thread-prior"); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "result": { + "thread": { "id": "thread-prior", "status": { "type": "idle" } } + } + }), + ) + .unwrap(); + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let binding_path = tmp.path().join("state/binding.json"); + let control_state_path = tmp.path().join("state/control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); + let runtime_for_pump = runtime.clone(); + let binding_for_pump = binding_path.clone(); + let control_state_for_pump = control_state_path.clone(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_for_pump, + &control_state_for_pump, + &runtime_for_pump, + Some(ControlResume { + thread_id: "thread-prior", + ready: resume_ready_rx, + tui_loaded_timeout: TUI_LOADED_TIMEOUT, + }), + None, + tx, + ) + }); + pre_gate_checked_rx + .recv_timeout(Duration::from_secs(2)) + .unwrap(); + resume_ready_tx.send(()).unwrap(); + acknowledge_tui_thread_loaded(&rx); + assert!(matches!( + rx.recv_timeout(Duration::from_secs(2)).unwrap(), + ControlEvent::Bound + )); + server.join().unwrap(); + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + + let binding = load_current_binding(&binding_path, &runtime) + .unwrap() + .unwrap(); + assert_eq!(binding.thread_id(), "thread-prior"); + let state = load_current_control_state(&control_state_path, &runtime, &binding) + .unwrap() + .unwrap(); + assert!(state.subscribed()); + assert_eq!(state.observed(), &CodexObservedState::Idle); +} + +/// A resumed thread still holds its context, and the app-server replays +/// `thread/tokenUsage/updated` to the newly attached connection — before the resume response, +/// which the binding handshake otherwise discards along with every other notification. The +/// construction that resumed this seat has already removed the predecessor's record, so a +/// dropped replay leaves a resumed-and-idle seat reading `null` against a full window with +/// nothing to correct it until its next model response. +#[test] +fn a_token_usage_replayed_before_the_resume_response_still_reaches_the_record() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let config = delivery_config(tmp.path()); + let agent_dir = config.agent_dir.clone(); + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let agent_dir_for_server = agent_dir.clone(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + stream + .set_read_timeout(Some(Duration::from_secs(10))) + .unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialize" + ); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialized" + ); + let loaded = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(loaded["method"], "thread/loaded/list"); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_TUI_LOADED_REQUEST_ID, + "result": { "data": ["thread-prior"] } + }), + ) + .unwrap(); + let subscribe = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(subscribe["method"], "thread/resume"); + assert_eq!(subscribe["params"]["threadId"], "thread-prior"); + // The replay, ahead of the response the handshake is waiting for. + write_json_message( + &mut websocket, + &json!({ + "method": "thread/tokenUsage/updated", + "params": { + "threadId": "thread-prior", + "turnId": "turn-prior", + "tokenUsage": { + "last": { "totalTokens": 92_283 }, + "total": { "totalTokens": 2_235_329 }, + "modelContextWindow": 258_400 + } + } + }), + ) + .unwrap(); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "result": { + "thread": { "id": "thread-prior", "status": { "type": "idle" } } + } + }), + ) + .unwrap(); + let deadline = Instant::now() + Duration::from_secs(10); + while harness_context::read(&harness_context::harness_context_path( + &agent_dir_for_server, + )) + .is_none() + && Instant::now() < deadline + { + std::thread::sleep(Duration::from_millis(10)); + } + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let binding_path = tmp.path().join("state/binding.json"); + let control_state_path = tmp.path().join("state/control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); + let binding_for_pump = binding_path.clone(); + let control_state_for_pump = control_state_path.clone(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_for_pump, + &control_state_for_pump, + &runtime, + Some(ControlResume { + thread_id: "thread-prior", + ready: resume_ready_rx, + tui_loaded_timeout: TUI_LOADED_TIMEOUT, + }), + Some(config), + tx, + ) + }); + resume_ready_tx.send(()).unwrap(); + acknowledge_tui_thread_loaded(&rx); + assert!(matches!( + rx.recv_timeout(Duration::from_secs(10)).unwrap(), + ControlEvent::Bound + )); + server.join().unwrap(); + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + + let observed = + context_record(&agent_dir).expect("the replayed reading never reached the record"); + assert_eq!(observed.used_percent, Some(33.0)); + assert_eq!(observed.used_tokens, Some(92_283)); + assert_eq!(observed.session_total_tokens, Some(2_235_329)); +} + +#[test] +fn tui_loaded_timeout_reports_the_specific_failure_before_outer_binding_timeout() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialize" + ); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialized" + ); + let loaded = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(loaded["method"], "thread/loaded/list"); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_TUI_LOADED_REQUEST_ID, + "result": { "data": [] } + }), + ) + .unwrap(); + thread::sleep(Duration::from_millis(250)); + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let binding_path = tmp.path().join("state/binding.json"); + let control_state_path = tmp.path().join("state/control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_path, + &control_state_path, + &runtime, + Some(ControlResume { + thread_id: "thread-prior", + ready: resume_ready_rx, + tui_loaded_timeout: Duration::from_millis(50), + }), + None, + tx, + ) + }); + resume_ready_tx.send(()).unwrap(); + let ControlEvent::Failed(error) = rx.recv_timeout(Duration::from_secs(2)).unwrap() else { + panic!("inner TUI-loaded deadline did not report its specific failure"); + }; + assert!( + error.contains( + "controlled Codex TUI did not load preserved thread thread-prior before control resume" + ), + "unexpected control failure: {error}" + ); + + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + server.join().unwrap(); +} + +#[test] +fn missing_saved_rollout_fails_without_rebinding_the_incarnation() { + let tmp = tempfile::tempdir().unwrap(); + let _stop_exclusive = stop_flag_tests(); + let binding_path = tmp.path().join("state/binding.json"); + let control_state_path = tmp.path().join("state/control-state.json"); + let prior_runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let prior_binding = CodexThreadBinding::new(&prior_runtime, "thread-prior".into()); + atomic_json(&binding_path, &prior_binding).unwrap(); + + let socket = tmp.path().join("server.sock"); + let listener = UnixListener::bind(&socket).unwrap(); + let server = thread::spawn(move || { + let (stream, _) = listener.accept().unwrap(); + let mut websocket = tungstenite::accept(stream).unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialize" + ); + write_json_message( + &mut websocket, + &json!({ "id": 0, "result": { "userAgent": "fake" } }), + ) + .unwrap(); + assert_eq!( + read_json_message(&mut websocket).unwrap().unwrap()["method"], + "initialized" + ); + let loaded = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(loaded["method"], "thread/loaded/list"); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_TUI_LOADED_REQUEST_ID, + "result": { "data": ["thread-prior"] } + }), + ) + .unwrap(); + let resume = read_json_message(&mut websocket).unwrap().unwrap(); + assert_eq!(resume["method"], "thread/resume"); + assert_eq!(resume["params"]["threadId"], "thread-prior"); + write_json_message( + &mut websocket, + &json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "error": { + "code": -32600, + "message": "no rollout found for thread id thread-prior" + } + }), + ) + .unwrap(); + }); + + let stream = UnixStream::connect(&socket).unwrap(); + let shutdown = stream.try_clone().unwrap(); + let websocket = initialize_control(stream) + .unwrap() + .expect("no stop raised in tests"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let (tx, rx) = mpsc::channel(); + let (resume_ready_tx, resume_ready_rx) = mpsc::channel(); + let runtime_for_pump = runtime.clone(); + let binding_for_pump = binding_path.clone(); + let control_state_for_pump = control_state_path.clone(); + let pump = thread::spawn(move || { + pump_control( + websocket, + &binding_for_pump, + &control_state_for_pump, + &runtime_for_pump, + Some(ControlResume { + thread_id: "thread-prior", + ready: resume_ready_rx, + tui_loaded_timeout: TUI_LOADED_TIMEOUT, + }), + None, + tx, + ) + }); + resume_ready_tx.send(()).unwrap(); + acknowledge_tui_thread_loaded(&rx); + let ControlEvent::Failed(error) = rx.recv_timeout(Duration::from_secs(2)).unwrap() else { + panic!("missing saved rollout did not fail closed"); + }; + assert!(error.contains("saved Codex resume binding has no persisted rollout")); + + server.join().unwrap(); + let _ = shutdown.shutdown(Shutdown::Both); + pump.join().unwrap(); + assert_eq!( + serde_json::from_slice::(&fs::read(&binding_path).unwrap()) + .unwrap(), + prior_binding + ); + assert!(!control_state_path.exists()); +} + +#[test] +fn a_binding_from_another_runtime_incarnation_is_rejected() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("binding.json"); + let prior = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let current = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + atomic_json( + &path, + &CodexThreadBinding::new(&prior, "thread-prior".into()), + ) + .unwrap(); + assert_eq!( + load_resume_thread(&path, "h.worker", "h.worker").unwrap(), + Some("thread-prior".into()), + "a validated prior binding may select resume but must not become current ownership" + ); + let error = load_current_binding(&path, ¤t).unwrap_err(); + assert!(error.to_string().contains("different runtime incarnation")); +} + +#[test] +fn watcher_holds_without_an_exact_turn_and_tracks_one_unmatched_lifecycle() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + + assert!( + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "active", "activeFlags": [] } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::ActiveWithoutTurn, + turn_id: None, + } + ); + + assert!( + state + .observe(&json!({ + "method": "turn/started", + "params": { + "threadId": "thread-main", + "turn": { "id": "turn-1" } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-1".into() + } + ); + + assert!( + !state + .observe(&json!({ + "method": "turn/started", + "params": { + "threadId": "thread-other", + "turn": { "id": "turn-other" } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-1".into() + } + ); + + assert!( + state + .observe(&json!({ + "method": "turn/completed", + "params": { + "threadId": "thread-main", + "turn": { "id": "turn-1" } + } + })) + .unwrap() + ); + assert_eq!(state.observed(), &CodexObservedState::Idle); +} + +#[test] +fn watcher_holds_review_compaction_and_conflicting_turns_until_safe() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + state + .observe(&json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "item": { "type": "enteredReviewMode" } + } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-1".into()), + } + ); + + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "active", "activeFlags": [] } + } + })) + .unwrap(); + assert!(matches!( + state.observed(), + CodexObservedState::Held { + reason: CodexHoldReason::Review, + .. + } + )); + + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-2" } } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-2".into()), + } + ); + + // Codex can complete the preparatory review item after the reviewer turn starts. That + // duplicate review event keeps the typed hold bound to the newer turn. + assert!( + !state + .observe(&json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "item": { "type": "enteredReviewMode" } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-2".into()), + } + ); + + // The review hold also survives the stale turn completion. Only an idle thread releases + // it. + assert!( + !state + .observe(&json!({ + "method": "turn/completed", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-2".into()), + } + ); + + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { "threadId": "thread-main", "status": { "type": "idle" } } + })) + .unwrap(); + assert_eq!(state.observed(), &CodexObservedState::Idle); + + // A real review can start its reviewer turn before Codex reports the preparatory turn's + // typed review item. The typed non-steerable event refines that generic conflict. + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-late-1" } } + })) + .unwrap(); + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-late-2" } } + })) + .unwrap(); + assert!(matches!( + state.observed(), + CodexObservedState::Held { + reason: CodexHoldReason::ConflictingTurn, + .. + } + )); + state + .observe(&json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-late-1", + "item": { "type": "enteredReviewMode" } + } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-late-1".into()), + } + ); + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { "threadId": "thread-main", "status": { "type": "idle" } } + })) + .unwrap(); + + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-3" } } + })) + .unwrap(); + state + .observe(&json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-3", + "item": { "type": "contextCompaction" } + } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Compaction, + turn_id: Some("turn-3".into()), + } + ); + assert!( + !state + .observe(&json!({ + "method": "turn/completed", + "params": { "threadId": "thread-main", "turn": { "id": "turn-3" } } + })) + .unwrap() + ); + assert!(matches!( + state.observed(), + CodexObservedState::Held { + reason: CodexHoldReason::Compaction, + .. + } + )); +} + +#[test] +fn exiting_review_mode_mid_turn_restores_the_steerable_turn() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state.subscribed = true; + + // An inline review runs as its own turn on the reviewed thread, so the hold binds to the + // reviewer turn that `exitedReviewMode` later reports. + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-review" } } + })) + .unwrap(); + state + .observe(&json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-review", + "item": { "type": "enteredReviewMode", "id": "item-1", "review": "review" } + } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-review".into()), + } + ); + assert_eq!(delivery.maybe_request(&state).unwrap(), None); + + // Review ends while the turn keeps running: the typed exit item is the only signal, and it + // must restore the exact turn the hold carried instead of waiting for the next idle. + assert!( + state + .observe(&json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-review", + "item": { "type": "exitedReviewMode", "id": "item-2", "review": "review" } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-review".into(), + } + ); + + // Codex reports both lifecycle edges of the same item; the second one changes nothing. + assert!( + !state + .observe(&json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-review", + "item": { "type": "exitedReviewMode", "id": "item-2", "review": "review" } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-review".into(), + } + ); + + // The payoff: native delivery steers the still-running turn instead of waiting for idle. + let request = delivery.maybe_request(&state).unwrap().unwrap(); + assert_eq!(request["method"], "turn/steer"); + assert_eq!(request["params"]["threadId"], "thread-main"); + assert_eq!(request["params"]["expectedTurnId"], "turn-review"); + assert!(config.inbox.join(&filename).is_file()); +} + +#[test] +fn delivery_irrelevant_items_and_foreign_turn_review_exits_keep_the_observed_state() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + + // Most item types say nothing about steerability. They are ignored on purpose, not by + // omission: the observed state and the changed flag both stay put. + for item_type in ["agentMessage", "commandExecution", "webSearch"] { + assert!( + !state + .observe(&json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "item": { "type": item_type, "id": "item-1" } + } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-1".into(), + } + ); + } + + // A review exit reporting a turn the hold does not carry proves nothing about the held + // turn, so the hold survives exactly as it did before typed exits were observed. + state.observed = CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-2".into()), + }; + let stale_exit = json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "item": { "type": "exitedReviewMode", "id": "item-2", "review": "review" } + } + }); + assert!(!state.observe(&stale_exit).unwrap()); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::Review, + turn_id: Some("turn-2".into()), + } + ); + + // A review exit never invents a turn on an idle thread and never releases another hold. + for observed in [ + CodexObservedState::Idle, + CodexObservedState::AwaitingStatus, + CodexObservedState::Held { + reason: CodexHoldReason::Compaction, + turn_id: Some("turn-1".into()), + }, + CodexObservedState::Held { + reason: CodexHoldReason::ConflictingTurn, + turn_id: None, + }, + ] { + state.observed = observed.clone(); + assert!( + !state + .observe(&json!({ + "method": "item/started", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "item": { + "type": "exitedReviewMode", + "id": "item-3", + "review": "review" + } + } + })) + .unwrap() + ); + assert_eq!(state.observed(), &observed); + } +} + +#[test] +fn an_unclassified_item_holds_until_the_next_idle_status() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + + assert!( + state + .observe(&json!({ + "method": "item/completed", + "params": { + "threadId": "thread-main", + "turnId": "turn-1", + "item": { "type": "futureBlockingItem", "id": "item-1" } + } + })) + .unwrap() + ); + assert!(matches!( + state.observed(), + CodexObservedState::Held { + reason: CodexHoldReason::UnknownProtocol, + turn_id: Some(turn_id), + } if turn_id == "turn-1" + )); + + assert!( + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "idle" } + } + })) + .unwrap() + ); + assert_eq!(state.observed(), &CodexObservedState::Idle); +} + +#[test] +fn an_unclassified_server_request_holds_until_the_next_idle_status() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + + assert!( + !state + .observe(&json!({ + "id": 1, + "method": "item/commandExecution/requestApproval", + "params": {} + })) + .unwrap() + ); + assert!(matches!( + state.observed(), + CodexObservedState::Active { .. } + )); + + assert!( + state + .observe(&json!({ + "id": 2, + "method": "future/request", + "params": {} + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::UnknownProtocol, + turn_id: Some("turn-1".into()), + } + ); + + assert!( + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "idle" } + } + })) + .unwrap() + ); + assert_eq!(state.observed(), &CodexObservedState::Idle); +} + +#[test] +fn an_errored_turn_completes_into_the_named_error_not_a_conflicting_turn() { + // Replays the captured terminal-error ordering (#264): a usage limit emits + // `thread/status/changed -> systemError` immediately before the failed turn's + // `turn/completed`. That completion reports one turn's lifecycle and carries no thread + // status, so it is not evidence the thread recovered, and it is not evidence of a second + // live turn either. The honest resolution is the condition the thread itself reported. + for (status, reason) in [ + ("systemError", CodexHoldReason::SystemError), + ("notLoaded", CodexHoldReason::NotLoaded), + ] { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state.subscribed = true; + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + assert!( + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { "threadId": "thread-main", "status": { "type": status } } + })) + .unwrap() + ); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason, + turn_id: None + } + ); + + // Completion makes a reported system error terminal. It preserves `notLoaded`, whose + // owner is the later thread status that proves the thread loaded again. + let changed = state + .observe(&json!({ + "method": "turn/completed", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + + if reason == CodexHoldReason::SystemError { + assert!(changed); + assert_eq!( + state.observed(), + &CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + ); + } else { + assert!(!changed); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason, + turn_id: None, + } + ); + } + + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + if reason == CodexHoldReason::SystemError { + let request = delivery + .maybe_request(&state) + .unwrap() + .expect("a terminal system error must permit the next turn"); + assert_eq!(request["method"], "turn/start"); + } else { + assert_eq!(delivery.maybe_request(&state).unwrap(), None); + assert!( + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "idle" } + } + })) + .unwrap() + ); + assert_eq!(state.observed(), &CodexObservedState::Idle); + assert!(delivery.maybe_request(&state).unwrap().is_some()); + } + assert!(config.inbox.join(&filename).is_file()); + } + + // The next provider turn replaces the terminal diagnostic with the exact live turn. + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state.subscribed = true; + for message in [ + json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + }), + json!({ + "method": "thread/status/changed", + "params": { "threadId": "thread-main", "status": { "type": "systemError" } } + }), + json!({ + "method": "turn/completed", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + }), + ] { + state.observe(&message).unwrap(); + } + assert_eq!( + state.observed(), + &CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + ); + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "active", "activeFlags": [] } + } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::ActiveWithoutTurn, + turn_id: None, + } + ); + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-2" } } + })) + .unwrap(); + assert_eq!( + state.observed(), + &CodexObservedState::Active { + turn_id: "turn-2".into(), + } + ); +} + +#[test] +fn persisted_control_state_is_bound_to_the_exact_runtime_incarnation() { + let tmp = tempfile::tempdir().unwrap(); + let path = tmp.path().join("control-state.json"); + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let binding = CodexThreadBinding::new(&runtime, "thread-main".into()); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + state.observed = CodexObservedState::Active { + turn_id: "turn-1".into(), + }; + atomic_json(&path, &state).unwrap(); + let persisted: Value = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); + assert_eq!(persisted["observed"]["turnId"], "turn-1"); + assert!(persisted["observed"].get("turn_id").is_none()); + + assert_eq!( + load_current_control_state(&path, &runtime, &binding) + .unwrap() + .unwrap(), + state + ); + + let replacement = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let replacement_binding = CodexThreadBinding::new(&replacement, "thread-main".into()); + let error = + load_current_control_state(&path, &replacement, &replacement_binding).unwrap_err(); + assert!(error.to_string().contains("different runtime binding")); +} + +#[test] +fn subscription_waits_for_a_rollout_without_claiming_success() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + let acceptance = state + .accept_subscription(&json!({ + "id": CONTROL_SUBSCRIBE_REQUEST_ID, + "error": { + "code": -32600, + "message": "no rollout found for thread id thread-main" + } + })) + .unwrap(); + + assert!(matches!(acceptance, SubscriptionAcceptance::Deferred)); + assert!(!state.subscribed()); + assert_eq!(state.observed(), &CodexObservedState::AwaitingStatus); +} + +#[test] +fn app_server_receives_only_its_supported_global_configuration() { + let authored = vec![ + "-c".into(), + "projects={\"/workspace\"={trust_level=\"trusted\"}}".into(), + "--model".into(), + "gpt-test".into(), + "--enable".into(), + "one".into(), + "--disable=two".into(), + "--strict-config".into(), + "--dangerously-bypass-approvals-and-sandbox".into(), + "--dangerously-bypass-hook-trust".into(), + "boot".into(), + ]; + + assert_eq!( + controlled_app_server_args("unix:///server.sock", &authored).unwrap(), + [ + "app-server", + "-c", + "projects={\"/workspace\"={trust_level=\"trusted\"}}", + "--enable", + "one", + "--disable=two", + "--strict-config", + "--listen", + "unix:///server.sock", + ] + ); +} + +#[test] +fn remote_resume_projects_exact_hook_hashes_without_persisted_state() { + let tmp = tempfile::tempdir().unwrap(); + let cwd = fs::canonicalize(tmp.path()).unwrap(); + let source = cwd.join(".codex/hooks.json"); + let untrusted_key = format!("{}:session_start:0:0", source.display()); + let modified_key = format!("{}:stop:1:0", source.display()); + let response = json!({ + "id": HOOK_TRUST_PREFLIGHT_REQUEST_ID, + "result": { + "data": [{ + "cwd": cwd, + "hooks": [ + { + "key": untrusted_key, + "currentHash": "sha256:one", + "trustStatus": "untrusted", + "isManaged": false, + "enabled": true + }, + { + "key": modified_key, + "currentHash": "sha256:two", + "trustStatus": "modified", + "isManaged": false, + "enabled": false + }, + { + "key": "already-trusted", + "currentHash": "sha256:three", + "trustStatus": "trusted", + "isManaged": false, + "enabled": true + }, + { + "key": "managed", + "currentHash": "sha256:four", + "trustStatus": "managed", + "isManaged": true, + "enabled": true + } + ] + }] + } + }); + + let projection = hook_trust_projection_from_response(&response, &cwd) + .unwrap() + .unwrap(); + assert_eq!(projection.count, 2); + let parsed: toml::Value = toml::from_str(&projection.override_value).unwrap(); + let state = parsed + .get("hooks") + .and_then(|hooks| hooks.get("state")) + .and_then(toml::Value::as_table) + .unwrap(); + assert_eq!( + state[&untrusted_key]["trusted_hash"].as_str(), + Some("sha256:one") + ); + assert_eq!( + state[&modified_key]["trusted_hash"].as_str(), + Some("sha256:two") + ); + assert!(!state.contains_key("already-trusted")); + assert!(!state.contains_key("managed")); + + let mut args = controlled_app_server_args( + "unix:///server.sock", + &["--dangerously-bypass-hook-trust".into(), "boot".into()], + ) + .unwrap(); + insert_app_server_config_override(&mut args, projection.override_value).unwrap(); + assert_eq!(args[args.len() - 4], "-c"); + assert!(args[args.len() - 3].starts_with("hooks.state=")); + assert_eq!(&args[args.len() - 2..], ["--listen", "unix:///server.sock"]); +} + +#[test] +fn hook_trust_projection_fails_closed_on_provider_shape_drift() { + let tmp = tempfile::tempdir().unwrap(); + let cwd = fs::canonicalize(tmp.path()).unwrap(); + let response = json!({ + "result": { + "data": [{ + "cwd": cwd, + "hooks": [{ + "key": "hook", + "currentHash": "not-a-provider-hash", + "trustStatus": "untrusted", + "isManaged": false + }] + }] + } + }); + let error = hook_trust_projection_from_response(&response, &cwd).unwrap_err(); + assert!(error.to_string().contains("typed currentHash")); + + let response = json!({ + "result": { + "data": [{ + "cwd": cwd, + "hooks": [{ + "key": "hook", + "currentHash": "sha256:value", + "trustStatus": "future-status", + "isManaged": false + }] + }] + } + }); + let error = hook_trust_projection_from_response(&response, &cwd).unwrap_err(); + assert!(error.to_string().contains("unknown trustStatus")); +} + +#[test] +fn hook_preflight_uses_the_explicit_controlled_workspace() { + let tmp = tempfile::tempdir().unwrap(); + let explicit = tmp.path().join("workspace"); + fs::create_dir(&explicit).unwrap(); + assert_eq!( + controlled_hook_cwd(&[ + "--dangerously-bypass-hook-trust".into(), + "--cd".into(), + explicit.display().to_string(), + "boot".into(), + ]) + .unwrap(), + fs::canonicalize(explicit).unwrap() + ); + assert!( + authored_bypasses_hook_trust(&[ + "--dangerously-bypass-hook-trust".into(), + "boot".into() + ]) + .unwrap() + ); + assert!( + !authored_bypasses_hook_trust(&["--".into(), "--dangerously-bypass-hook-trust".into()]) + .unwrap() + ); +} + +#[test] +fn process_group_cleanup_reaps_a_native_launcher_descendant() { + let temporary = tempfile::tempdir().unwrap(); + let descendant_pidfile = temporary.path().join("descendant.pid"); + let mut command = Command::new("sh"); + command + .arg("-c") + .arg(r#"sh -c 'printf "%s" "$$" > "$DESCENDANT_PIDFILE"; exec sleep 60' & sleep 60"#) + .env("DESCENDANT_PIDFILE", &descendant_pidfile) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let mut launcher = spawn_process_group(&mut command, None).unwrap(); + let mut foreign_command = Command::new("/bin/sh"); + foreign_command + .arg("-c") + .arg("sleep 60") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let mut foreign_owner = spawn_process_group(&mut foreign_command, None).unwrap(); + let foreign_pid = foreign_owner.id() as i32; + let deadline = Instant::now() + Duration::from_secs(1); + // The shell's `>` redirection creates an empty pidfile before `printf` + // writes, so wait for parsable content, not mere file existence. + let mut descendant = None; + while descendant.is_none() && Instant::now() < deadline { + if let Ok(content) = std::fs::read_to_string(&descendant_pidfile) { + descendant = content.trim().parse::().ok(); + } + if descendant.is_none() { + std::thread::sleep(Duration::from_millis(10)); + } + } + let descendant = descendant.expect("the launcher did not create its native descendant"); + assert!( + process_can_retain_cleanup_resources(descendant), + "the native descendant was not alive before cleanup" + ); + + launcher.terminate(); + assert!( + process_can_retain_cleanup_resources(foreign_pid), + "cleanup killed a different live owner" + ); + foreign_owner.terminate(); + let deadline = Instant::now() + Duration::from_secs(1); + while process_can_retain_cleanup_resources(descendant) && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + let survived = process_can_retain_cleanup_resources(descendant); + if survived { + unsafe { + libc::kill(descendant, libc::SIGKILL); + } + } + assert!( + !survived, + "native descendant {descendant} survived process-group cleanup" + ); +} + +#[test] +fn dropping_a_process_group_owner_reaps_the_group_and_socket() { + let temporary = tempfile::tempdir().unwrap(); + let socket_path = temporary.path().join("app-server.sock"); + let _listener = UnixListener::bind(&socket_path).unwrap(); + let mut command = Command::new("/bin/sh"); + command + .arg("-c") + .arg("sleep 60") + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let launcher = spawn_process_group(&mut command, Some(&socket_path)).unwrap(); + let launcher_pid = launcher.id() as i32; + assert!(process_can_retain_cleanup_resources(launcher_pid)); + + drop(launcher); + let deadline = Instant::now() + Duration::from_secs(1); + while (process_can_retain_cleanup_resources(launcher_pid) || socket_path.exists()) + && Instant::now() < deadline + { + std::thread::sleep(Duration::from_millis(10)); + } + assert!( + !process_can_retain_cleanup_resources(launcher_pid), + "the app-server survived owner cleanup" + ); + assert!( + !socket_path.exists(), + "the app-server socket survived owner cleanup" + ); +} + +#[test] +fn a_live_socket_refuses_a_second_control_owner() { + let temporary = tempfile::tempdir().unwrap(); + let socket_path = temporary.path().join("app-server.sock"); + let listener = UnixListener::bind(&socket_path).unwrap(); + + let error = prepare_socket_for_launch(&socket_path).unwrap_err(); + + assert!( + error + .to_string() + .contains("refusing a second control owner") + ); + assert!(socket_path.exists(), "the live owner socket was removed"); + assert!( + UnixStream::connect(&socket_path).is_ok(), + "the first owner stopped accepting connections" + ); + drop(listener); +} + +#[test] +fn a_dead_socket_is_removed_before_launch() { + let temporary = tempfile::tempdir().unwrap(); + let socket_path = temporary.path().join("app-server.sock"); + let listener = UnixListener::bind(&socket_path).unwrap(); + drop(listener); + assert!(socket_path.exists()); + + prepare_socket_for_launch(&socket_path).unwrap(); + + assert!(!socket_path.exists(), "the dead socket was not removed"); +} + +#[test] +fn a_killed_wrapper_reaps_its_app_server_and_the_next_launch_recovers_its_socket() { + const TEST_NAME: &str = "codex_app_server::tests::a_killed_wrapper_reaps_its_app_server_and_the_next_launch_recovers_its_socket"; + const ROLE: &str = "ST2_CODEX_ORPHAN_TEST_ROLE"; + const SOCKET_PATH: &str = "ST2_CODEX_ORPHAN_TEST_SOCKET"; + const PID_PATH: &str = "ST2_CODEX_ORPHAN_TEST_PID"; + const READY_PATH: &str = "ST2_CODEX_ORPHAN_TEST_READY"; + + match std::env::var(ROLE).as_deref() { + Ok("server") => { + let socket_path = PathBuf::from(std::env::var_os(SOCKET_PATH).unwrap()); + let ready_path = PathBuf::from(std::env::var_os(READY_PATH).unwrap()); + let _listener = UnixListener::bind(socket_path).unwrap(); + fs::write(ready_path, b"ready").unwrap(); + loop { + std::thread::sleep(Duration::from_secs(60)); + } + } + Ok("wrapper") => { + let pid_path = PathBuf::from(std::env::var_os(PID_PATH).unwrap()); + let socket_path = PathBuf::from(std::env::var_os(SOCKET_PATH).unwrap()); + let mut command = Command::new(std::env::current_exe().unwrap()); + command + .arg("--exact") + .arg(TEST_NAME) + .arg("--nocapture") + .env(ROLE, "server") + .env(SOCKET_PATH, std::env::var_os(SOCKET_PATH).unwrap()) + .env(READY_PATH, std::env::var_os(READY_PATH).unwrap()) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()); + let server = spawn_process_group(&mut command, Some(&socket_path)).unwrap(); + fs::write(pid_path, server.id().to_string()).unwrap(); + loop { + std::thread::sleep(Duration::from_secs(60)); + } + } + Ok(role) => panic!("unknown orphan test role {role}"), + Err(_) => {} + } + + let temporary = tempfile::tempdir().unwrap(); + let socket_path = temporary.path().join("app-server.sock"); + let pid_path = temporary.path().join("app-server.pid"); + let ready_path = temporary.path().join("app-server.ready"); + let mut wrapper = Command::new(std::env::current_exe().unwrap()) + .arg("--exact") + .arg(TEST_NAME) + .arg("--nocapture") + .env(ROLE, "wrapper") + .env(SOCKET_PATH, &socket_path) + .env(PID_PATH, &pid_path) + .env(READY_PATH, &ready_path) + .stdin(Stdio::null()) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .spawn() + .unwrap(); + let deadline = Instant::now() + Duration::from_secs(2); + while (!pid_path.is_file() || !ready_path.is_file()) && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + if !pid_path.is_file() || !ready_path.is_file() { + let _ = wrapper.kill(); + let _ = wrapper.wait(); + panic!("the wrapper did not start its app-server"); + } + let server_pid = fs::read_to_string(&pid_path) + .expect("the wrapper did not report its app-server PID") + .parse::() + .unwrap(); + assert!( + process_can_retain_cleanup_resources(server_pid), + "the app-server was not alive before the wrapper died" + ); + assert!( + fs::symlink_metadata(&socket_path) + .unwrap() + .file_type() + .is_socket(), + "the app-server did not bind its socket" + ); + + unsafe { + libc::kill(wrapper.id() as i32, libc::SIGKILL); + } + let _ = wrapper.wait(); + let deadline = Instant::now() + Duration::from_secs(2); + while process_can_retain_cleanup_resources(server_pid) && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + let server_survived = process_can_retain_cleanup_resources(server_pid); + if server_survived { + unsafe { + libc::kill(server_pid, libc::SIGKILL); + } + } + assert!(!server_survived, "the app-server survived its wrapper"); + assert!( + socket_path.exists(), + "the app-server did not leave the expected recoverable socket" + ); + let refusal_deadline = Instant::now() + Duration::from_secs(2); + let refusal = loop { + match UnixStream::connect(&socket_path) { + Ok(stream) if Instant::now() < refusal_deadline => { + drop(stream); + std::thread::sleep(Duration::from_millis(10)); + } + Ok(_) => panic!("the residual socket still had a live listener"), + Err(error) => break error, + } + }; + assert_eq!(refusal.kind(), std::io::ErrorKind::ConnectionRefused); + + prepare_socket_for_launch(&socket_path) + .expect("the next launch did not recover the residual socket"); + assert!( + !socket_path.exists(), + "the next launch did not remove the residual socket" + ); + let replacement = UnixListener::bind(&socket_path) + .expect("the next app-server could not bind the recovered socket"); + assert!( + UnixStream::connect(&socket_path).is_ok(), + "the replacement app-server socket did not accept a connection" + ); + drop(replacement); +} + +#[test] +fn app_server_configuration_extraction_fails_closed_at_ambiguous_boundaries() { + let missing = + controlled_app_server_args("unix:///server.sock", &["-c".into()]).unwrap_err(); + assert!(missing.to_string().contains("has no value")); + + let unknown = controlled_app_server_args( + "unix:///server.sock", + &["--future-option".into(), "value".into(), "boot".into()], + ) + .unwrap_err(); + assert!(unknown.to_string().contains("unknown Codex option")); + + let sensitive = controlled_app_server_args( + "unix:///server.sock", + &["--future-token=do-not-log-this".into(), "boot".into()], + ) + .unwrap_err(); + assert!(sensitive.to_string().contains("--future-token")); + assert!(!sensitive.to_string().contains("do-not-log-this")); + + assert_eq!( + controlled_app_server_args( + "unix:///server.sock", + &[ + "--config=projects.x.trust_level=\"trusted\"".into(), + "resume".into(), + "thread-explicit".into(), + ], + ) + .unwrap(), + [ + "app-server", + "--config=projects.x.trust_level=\"trusted\"", + "--listen", + "unix:///server.sock", + ] + ); +} + +#[test] +fn controlled_tui_resumes_a_prior_binding_without_overriding_authored_selection() { + let authored = vec!["--model".into(), "gpt-test".into(), "boot".into()]; + assert_eq!( + controlled_tui_args("unix:///server.sock", &authored, None).unwrap(), + [ + "--remote", + "unix:///server.sock", + "--model", + "gpt-test", + "boot" + ] + ); + assert_eq!( + controlled_tui_args("unix:///server.sock", &authored, Some("thread-prior")).unwrap(), + [ + "--remote", + "unix:///server.sock", + "resume", + "--model", + "gpt-test", + "thread-prior", + "boot" + ] + ); + assert_eq!( + controlled_tui_args( + "unix:///server.sock", + &["resume".into(), "thread-explicit".into()], + Some("thread-prior") + ) + .unwrap(), + [ + "--remote", + "unix:///server.sock", + "resume", + "thread-explicit" + ] + ); + assert_eq!( + expected_resume_thread( + &["resume".into(), "thread-explicit".into()], + Some("thread-prior") + ) + .unwrap(), + None + ); + + let fork = vec![ + "--dangerously-bypass-hook-trust".into(), + "fork".into(), + "thread-explicit".into(), + ]; + assert_eq!( + controlled_tui_args("unix:///server.sock", &fork, Some("thread-prior")).unwrap(), + [ + "--remote", + "unix:///server.sock", + "--dangerously-bypass-hook-trust", + "fork", + "thread-explicit" + ] + ); + assert_eq!( + expected_resume_thread(&fork, Some("thread-prior")).unwrap(), + None + ); + assert_eq!( + expected_resume_thread(&authored, Some("thread-prior")).unwrap(), + Some("thread-prior") + ); +} + +#[test] +fn controlled_tui_resume_fails_closed_at_ambiguous_option_boundaries() { + let unknown = controlled_tui_args( + "unix:///server.sock", + &["--future-option".into(), "value".into(), "prompt".into()], + Some("thread-prior"), + ) + .unwrap_err(); + assert!(unknown.to_string().contains("unknown Codex option")); + + let image = controlled_tui_args( + "unix:///server.sock", + &["--image".into(), "one.png".into(), "prompt".into()], + Some("thread-prior"), + ) + .unwrap_err(); + assert!(image.to_string().contains("explicit `--`")); + + assert_eq!( + controlled_tui_args( + "unix:///server.sock", + &[ + "--image".into(), + "one.png".into(), + "--".into(), + "prompt".into(), + ], + Some("thread-prior"), + ) + .unwrap(), + [ + "--remote", + "unix:///server.sock", + "resume", + "--image", + "one.png", + "thread-prior", + "--", + "prompt" + ] + ); +} + +#[test] +fn state_key_is_path_and_identity_specific_without_embedding_either() { + let base = Path::new("/state"); + let first = state_dir_in(base, Path::new("/catalog/a"), "h.worker"); + let second = state_dir_in(base, Path::new("/catalog/b"), "h.worker"); + assert_ne!(first, second); + assert!(first.starts_with("/state/st2/codex")); + assert!(!first.display().to_string().contains("worker")); + assert!(!first.display().to_string().contains("catalog/a")); +} + +#[test] +fn wrapper_diagnostics_keep_one_bounded_run_without_authored_input() { + let tmp = tempfile::tempdir().unwrap(); + let state = tmp.path().join("state"); + secure_dir(&state).unwrap(); + + { + let mut diagnostics = WrapperDiagnostics::open(&state, "h.worker", "h.worker").unwrap(); + diagnostics.record("ownerAcquired", json!({})).unwrap(); + diagnostics + .record("failed", json!({ "error": "control socket was not ready" })) + .unwrap(); + } + let path = state.join("wrapper.log"); + let first = fs::read_to_string(&path).unwrap(); + let entries = first + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + assert_eq!(entries.len(), 2); + assert_eq!(entries[0]["schema"], WRAPPER_DIAGNOSTIC_SCHEMA); + assert_eq!(entries[0]["agent"], "h.worker"); + assert_eq!(entries[1]["stage"], "failed"); + assert!(first.contains("control socket was not ready")); + assert!(!first.contains("prompt")); + + { + let mut replacement = WrapperDiagnostics::open(&state, "h.worker", "h.worker").unwrap(); + replacement.record("ownerAcquired", json!({})).unwrap(); + } + let replacement = fs::read_to_string(&path).unwrap(); + assert_eq!(replacement.lines().count(), 1); + assert!(!replacement.contains("control socket was not ready")); + assert_eq!( + fs::metadata(path).unwrap().permissions().mode() & 0o777, + 0o600 + ); +} + +#[test] +fn runtime_owner_lock_is_nonblocking_and_released_on_close() { + let tmp = tempfile::tempdir().unwrap(); + let first = acquire_owner_lock(tmp.path()).unwrap(); + let error = acquire_owner_lock(tmp.path()).unwrap_err(); + assert!(error.to_string().contains("already has an owner")); + drop(first); + acquire_owner_lock(tmp.path()).unwrap(); +} + +#[test] +fn waiting_on_a_human_holds_the_exact_turn_and_releases_it_when_the_flag_clears() { + let runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); + let mut state = CodexControlState::new(&runtime, "thread-main".into()); + let status_changed = |flags: Value| { + json!({ + "method": "thread/status/changed", + "params": { + "threadId": "thread-main", + "status": { "type": "active", "activeFlags": flags } + } + }) + }; + let active_turn_1 = CodexObservedState::Active { + turn_id: "turn-1".into(), + }; + + state.observe(&status_changed(json!([]))).unwrap(); + state + .observe(&json!({ + "method": "turn/started", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + assert_eq!(state.observed(), &active_turn_1); + + for (flags, reason) in [ + ( + json!(["waitingOnApproval"]), + CodexHoldReason::WaitingOnApproval, + ), + ( + json!(["waitingOnUserInput"]), + CodexHoldReason::WaitingOnUserInput, + ), + ( + json!(["conversationHandoff", "waitingOnApproval"]), + CodexHoldReason::WaitingOnApproval, + ), + ] { + assert!(state.observe(&status_changed(flags)).unwrap()); + assert_eq!( + state.observed(), + &CodexObservedState::Held { + reason, + turn_id: Some("turn-1".into()), + } + ); + // Clearing the flag releases the same turn: no `turn/started` repeats mid-turn. + assert!(state.observe(&status_changed(json!([]))).unwrap()); + assert_eq!(state.observed(), &active_turn_1); + } + + // An unknown future flag value degrades to plain `active` instead of failing the frame. + assert!(!state.observe(&status_changed(json!(["handoff"]))).unwrap()); + assert_eq!(state.observed(), &active_turn_1); + + // The same field is carried by `thread/started`, before any turn is known. + let mut resumed = CodexControlState::new(&runtime, "thread-main".into()); + assert!( + resumed + .observe(&json!({ + "method": "thread/started", + "params": { + "thread": { + "id": "thread-main", + "status": { + "type": "active", + "activeFlags": ["waitingOnUserInput"] + } + } + } + })) + .unwrap() + ); + assert_eq!( + resumed.observed(), + &CodexObservedState::Held { + reason: CodexHoldReason::WaitingOnUserInput, + turn_id: None, + } + ); + + // A turn that ends while still flagged stays unsteerable and is released by the next + // idle status. `observe_turn_completed` is not modified here; this pins only that the + // flagged hold cannot decay into a steerable turn. + state + .observe(&status_changed(json!(["waitingOnApproval"]))) + .unwrap(); + state + .observe(&json!({ + "method": "turn/completed", + "params": { "threadId": "thread-main", "turn": { "id": "turn-1" } } + })) + .unwrap(); + assert!(matches!(state.observed(), CodexObservedState::Held { .. })); + + // A status arm without `activeFlags` keeps reading exactly as before. + assert!( + state + .observe(&json!({ + "method": "thread/status/changed", + "params": { "threadId": "thread-main", "status": { "type": "idle" } } + })) + .unwrap() + ); + assert_eq!(state.observed(), &CodexObservedState::Idle); + + // Delivery declines to steer a session that is waiting on a human, and retains the head. + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") + .unwrap(); + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + for reason in [ + CodexHoldReason::WaitingOnApproval, + CodexHoldReason::WaitingOnUserInput, + ] { + let blocked = subscribed_state(CodexObservedState::Held { + reason, + turn_id: Some("turn-1".into()), + }); + assert_eq!(delivery.maybe_request(&blocked).unwrap(), None); + assert!(config.inbox.join(&filename).is_file()); + } + let released = delivery + .maybe_request(&subscribed_state(active_turn_1.clone())) + .unwrap() + .expect("the retained head steers once the human has answered"); + assert_eq!(released["method"], "turn/steer"); + assert_eq!(released["params"]["expectedTurnId"], "turn-1"); +} diff --git a/src/ding/mod.rs b/src/ding/mod.rs index fa055d26..ef7403a6 100644 --- a/src/ding/mod.rs +++ b/src/ding/mod.rs @@ -1276,2753 +1276,4 @@ fn drain(rx: &Receiver<()>) { } #[cfg(test)] -mod tests { - use super::*; - use crate::message::{archive_dir, archive_msg, inbox_dir, send_to_inbox}; - use std::sync::Mutex; - use std::sync::atomic::AtomicUsize; - - fn msg(filename: &str, from: &str, subject: Option<&str>) -> Message { - Message { - filename: filename.to_string(), - ts_ms: filename - .split_once('-') - .and_then(|(timestamp, _)| timestamp.parse().ok()) - .unwrap_or_default(), - from: Some(from.to_string()), - subject: subject.map(str::to_string), - in_reply_to: None, - tags: vec![], - priority: None, - idempotency_key: None, - stream: None, - event_id: None, - event_key: None, - body: String::new(), - } - } - - fn declare_agent(root: &Path, host: &str, identity: &str, supervisor: Option<&str>) { - let directory = root.join(host).join(identity); - std::fs::create_dir_all(&directory).unwrap(); - let supervisor = supervisor - .map(|value| format!(" supervisor {value:?}\n")) - .unwrap_or_default(); - std::fs::write( - directory.join("agent.kdl"), - format!( - "agent {identity:?} {{\n identity {identity:?}\n host {host:?}\n{supervisor} type \"service\"\n pty \"agent\" {{ command \"x\" }}\n}}\n" - ), - ) - .unwrap(); - } - - fn render_without_catalog(message: &Message) -> String { - let catalog = tempfile::tempdir().unwrap(); - poke_text(catalog.path(), "h", "h.recipient", message) - } - - fn notice_text_without_catalog(notice: &PendingNotice) -> String { - let catalog = tempfile::tempdir().unwrap(); - notice.text( - DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }, - &mut None, - ) - } - - fn flush_without_catalog( - status_path: Option<&Path>, - pending: &mut VecDeque, - poker: &dyn Poker, - ) { - let catalog = tempfile::tempdir().unwrap(); - flush_pending( - DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }, - status_path, - pending, - poker, - ); - } - - /// Block until `ready` holds, reporting whether it did. The ceiling bounds a DING loop that - /// made no progress at all; it is not the behaviour under test, so it is far larger than any - /// plausible scheduling delay and a loaded host cannot turn it into a failure. Callers report - /// the result after their scope ends rather than panicking inside it, because an unwind from a - /// scoped thread leaves `run_ding` without the `stop` flag that ends it. - fn await_ding_progress(ready: impl Fn() -> bool) -> bool { - let deadline = Instant::now() + Duration::from_secs(30); - while !ready() { - if Instant::now() >= deadline { - return false; - } - std::thread::sleep(Duration::from_millis(2)); - } - true - } - - #[derive(Default)] - struct RecordingPoker { - alive: AtomicBool, - defer: AtomicBool, - probes: AtomicUsize, - failures: Mutex, - calls: Mutex>, - } - - impl RecordingPoker { - fn live() -> Self { - Self { - alive: AtomicBool::new(true), - ..Default::default() - } - } - } - - impl Poker for RecordingPoker { - fn poke(&self, text: &str) -> anyhow::Result { - self.calls.lock().unwrap().push(text.to_string()); - if self.defer.load(Ordering::SeqCst) { - return Ok(PokeOutcome::Deferred(DeferralReason::NoMaintainedComposer)); - } - let mut failures = self.failures.lock().unwrap(); - if *failures > 0 { - *failures -= 1; - anyhow::bail!("injected send failure"); - } - Ok(PokeOutcome::Delivered) - } - - fn session_alive(&self) -> bool { - self.probes.fetch_add(1, Ordering::SeqCst); - self.alive.load(Ordering::SeqCst) - } - } - - struct OwnershipPoker { - pokes: Mutex>, - retries: Mutex>, - poke_outcomes: Mutex>, - retry_outcomes: Mutex>, - } - - impl Poker for OwnershipPoker { - fn poke(&self, text: &str) -> anyhow::Result { - self.pokes.lock().unwrap().push(text.to_string()); - Ok(self - .poke_outcomes - .lock() - .unwrap() - .pop_front() - .expect("unexpected fresh poke")) - } - - fn retry_staged(&self, text: &str) -> anyhow::Result { - self.retries.lock().unwrap().push(text.to_string()); - Ok(self - .retry_outcomes - .lock() - .unwrap() - .pop_front() - .expect("unexpected staged retry")) - } - - fn session_alive(&self) -> bool { - true - } - } - - #[test] - fn poke_id_extracts_rand6() { - assert_eq!(poke_id("1785070000000-abc123.md"), "abc123"); - assert_eq!(poke_id("notes.md"), "notes"); - } - - #[test] - fn ancestor_depth_is_encoded_in_the_marker_run() { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "root", None); - declare_agent(catalog.path(), "h", "middle", Some("h.root")); - declare_agent(catalog.path(), "h", "recipient", Some("middle")); - - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-abc123.md", "h.middle", Some("direct")) - ), - "[DING] ↓ h.middle: direct [id:abc123]" - ); - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-def456.md", "h.root", Some("skip-level")) - ), - "[DING] ↓↓ h.root: skip-level [id:def456]" - ); - } - - #[test] - fn relationship_markers_cover_descendant_peer_and_missing_sender_spec() { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "root", None); - declare_agent(catalog.path(), "h", "recipient", Some("root")); - declare_agent(catalog.path(), "h", "child", Some("recipient")); - declare_agent(catalog.path(), "h", "peer", Some("root")); - - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-abc123.md", "h.child", Some("report")) - ), - "[DING] ↑ h.child: report [id:abc123]" - ); - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-def456.md", "h.peer", Some("note")) - ), - "[DING] ← h.peer: note [id:def456]" - ); - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-ghi789.md", "nightly-timer", Some("check")) - ), - "[DING] ? nightly-timer: check [id:ghi789]" - ); - } - - #[test] - fn dangling_catalog_entry_makes_relationship_unknown() { - use std::os::unix::fs::symlink; - - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "root", None); - declare_agent(catalog.path(), "h", "recipient", Some("root")); - symlink( - catalog.path().join("missing.kdl"), - catalog.path().join("dangling.kdl"), - ) - .unwrap(); - - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-abc123.md", "h.root", Some("dangling")) - ), - "[DING] ? h.root: dangling [id:abc123]" - ); - } - - #[test] - fn self_addressed_message_uses_identity_marker() { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "recipient", None); - - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.recipient", - &msg("1785070000000-abc123.md", "h.recipient", Some("self")) - ), - "[DING] ↺ h.recipient: self [id:abc123]" - ); - } - - #[test] - fn supervisor_cycle_renders_unknown_and_still_delivers() { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "recipient", Some("loop")); - declare_agent(catalog.path(), "h", "loop", Some("recipient")); - let message = msg("1785070000000-abc123.md", "h.loop", Some("cycle")); - let expected = "[DING] ? h.loop: cycle [id:abc123]"; - let resolver = RelationshipResolver::read(catalog.path()); - let recipient = resolve_spec(&resolver.specs, "h.recipient", "h").unwrap(); - - assert_eq!( - chain_bus_ids(&resolver.specs, recipient, "h"), - Err(crate::supervisor_chain::SupervisorChainError::Cycle), - "cycle detection must be distinct from the independent depth limit" - ); - - assert_eq!( - poke_text(catalog.path(), "h", "h.recipient", &message), - expected - ); - - let mut pending = VecDeque::from([PendingNotice::message(message)]); - let poker = RecordingPoker::live(); - flush_pending( - DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }, - None, - &mut pending, - &poker, - ); - - assert!(pending.is_empty()); - assert_eq!(poker.calls.lock().unwrap().as_slice(), [expected]); - } - - #[test] - fn malformed_catalog_with_resolvable_endpoints_renders_unknown_and_still_delivers() { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "root", None); - declare_agent(catalog.path(), "h", "recipient", Some("root")); - let malformed = catalog.path().join("h/broken/agent.kdl"); - std::fs::create_dir_all(malformed.parent().unwrap()).unwrap(); - std::fs::write(&malformed, "agent this is not valid KDL {").unwrap(); - let message = msg("1785070000000-abc123.md", "h.sender", Some("fallback")); - let message = Message { - from: Some("h.root".to_string()), - ..message - }; - let expected = "[DING] ? h.root: fallback [id:abc123]"; - let mut pending = VecDeque::from([PendingNotice::message(message)]); - let poker = RecordingPoker::live(); - - flush_pending( - DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }, - None, - &mut pending, - &poker, - ); - - assert!(pending.is_empty()); - assert_eq!(poker.calls.lock().unwrap().as_slice(), [expected]); - } - - #[test] - fn supervisor_depth_limit_fails_soft() { - let catalog = tempfile::tempdir().unwrap(); - for depth in 0..=SUPERVISOR_CHAIN_LIMIT { - let identity = format!("agent-{depth}"); - let supervisor = - (depth < SUPERVISOR_CHAIN_LIMIT).then(|| format!("agent-{}", depth + 1)); - declare_agent(catalog.path(), "h", &identity, supervisor.as_deref()); - } - - assert_eq!( - poke_text( - catalog.path(), - "h", - "h.agent-0", - &msg("1785070000000-abc123.md", "h.agent-64", Some("too deep")) - ), - "[DING] ? h.agent-64: too deep [id:abc123]" - ); - } - - #[test] - fn poke_text_normalizes_and_bounds_untrusted_fields() { - assert_eq!( - render_without_catalog(&msg("1785070000000-abc123.md", "alice", Some("deploy?"))), - "[DING] ? alice: deploy? [id:abc123]" - ); - assert_eq!( - render_without_catalog(&Message { - from: None, - subject: None, - ..msg("1785070000000-def456.md", "", None) - }), - "[DING] ? unknown: (no subject) [id:def456]" - ); - - let subject = format!("{}\nignored", "s".repeat(SUBJECT_MAX_CHARS + 20)); - let sender = format!("{}\tignored", "f".repeat(SENDER_MAX_CHARS + 20)); - let text = render_without_catalog(&msg("1785070000000-ghi789.md", &sender, Some(&subject))); - assert!(text.contains(&"s".repeat(SUBJECT_MAX_CHARS))); - assert!(!text.contains(&"s".repeat(SUBJECT_MAX_CHARS + 1))); - assert!(text.contains(&"f".repeat(SENDER_MAX_CHARS))); - assert!(!text.contains(&"f".repeat(SENDER_MAX_CHARS + 1))); - assert!(text.ends_with("[id:ghi789]")); - assert!(text.contains("[id:ghi789]")); - } - - #[test] - fn malicious_controls_cannot_escape_the_single_paste_frame() { - let message = msg( - "1785070000000-k0ygwh.md", - "attacker\x1b[201~\r\u{009b}2J", - Some("line one\n\tline two\x1b[201~key:return"), - ); - let text = render_without_catalog(&message); - assert!(!text.chars().any(char::is_control)); - assert!(!text.contains(" ")); - assert!(text.contains("[id:k0ygwh]")); - assert!(text.ends_with("[id:k0ygwh]")); - - let direct = format!("{text}\x1b[201~\nsecond line"); - let args = pty_stage_args("seat", &direct); - let framed = &args[3]; - assert_eq!(framed.matches(BRACKETED_PASTE_START).count(), 1); - assert_eq!(framed.matches(BRACKETED_PASTE_END).count(), 1); - let inner = framed - .strip_prefix(BRACKETED_PASTE_START) - .unwrap() - .strip_suffix(BRACKETED_PASTE_END) - .unwrap(); - assert!(!inner.chars().any(char::is_control)); - assert!(inner.ends_with("[201~ second line")); - } - - #[test] - fn pty_stage_and_submit_are_separate_exact_sequences() { - assert_eq!( - pty_stage_args("my-session", "hello\nworld"), - vec![ - "send", - "my-session", - "--seq", - "\x1b[200~hello world\x1b[201~", - ] - ); - assert_eq!( - pty_submit_args("my-session"), - vec!["send", "my-session", "--seq", "key:return"] - ); - assert!(!pty_stage_args("my-session", "hello").contains(&"key:return".to_string())); - } - - #[test] - fn pty_delivery_uses_face607_delay_order_and_seconds() { - assert_eq!( - pty_delivery_args("s", "hello"), - vec![ - "send", - "s", - "--with-delay", - "0.5", - "--seq", - "\x1b[200~hello\x1b[201~", - "--seq", - "key:return" - ] - ); - } - - fn idle_codex_screen_with_footer(footer: &str) -> String { - format!( - "\x1b[1m›\x1b[1C\x1b[22;2mFind and fix a bug in @filename\r\n\r\n\ - \x1b[2C\x1b[0m{footer}" - ) - } - - fn idle_codex_screen() -> String { - idle_codex_screen_with_footer("gpt-5.6-sol xhigh · /workspace") - } - - fn idle_codex_screen_with_home_relative_cwd() -> String { - idle_codex_screen().replace(" · /workspace", " · ~/Code/st2") - } - - fn staged_codex_screen_with_footer(text: &str, footer: &str) -> String { - let rendered = text.replace(' ', "\x1b[1C"); - format!( - "\x1b[1m›\x1b[1C\x1b[0m{rendered}\r\n\r\n\ - \x1b[2C\x1b[0m{footer}" - ) - } - - fn staged_codex_screen(text: &str) -> String { - staged_codex_screen_with_footer(text, "gpt-5.6-sol xhigh · /workspace") - } - - fn staged_wrapped_codex_screen() -> (&'static str, String) { - let text = "[DING] ↓ supervisor: a deliberately long synthetic notification with enough content to reach another renderer boundary before the final words [id:abc123]"; - let composer = concat!( - "[DING] ↓ supervisor: a deliberately long synthetic notification with enough", - "\x1b[3X\r\n", - " content to reach another renderer boundary before the final words [id:abc123]", - ); - ( - text, - format!( - "\x1b[1m›\x1b[22m {composer}\r\n\r\n\x1b[2C\x1b[0mgpt-5.6-sol low · Context 14% used" - ), - ) - } - - fn human_codex_screen() -> String { - staged_codex_screen("please keep my half-written draft") - } - - fn accepted_codex_screen(text: &str) -> String { - format!( - "{}\r\n\r\n{}", - staged_codex_screen(text), - idle_codex_screen() - ) - } - - fn queued_codex_screen(text: &str) -> String { - format!( - "Messages to be submitted after next tool call:\r\n{text}\r\n\r\n{}", - idle_codex_screen() - ) - } - - fn claude_rule() -> String { - "─".repeat(80) - } - - fn idle_claude_screen() -> String { - let rule = claude_rule(); - format!( - "Claude Code v2.1.220\r\n{rule}\r\n❯\u{00a0}Try \"write a test for validate.rs\"\r\n\ - {rule}\r\n ⏵⏵ bypass permissions on (shift+tab to cycle) · ← for agents" - ) - } - - fn idle_claude_screen_without_hint() -> String { - idle_claude_screen().replace(" (shift+tab to cycle)", "") - } - - fn staged_claude_screen(text: &str) -> String { - assert!(text.is_ascii()); - let rule = claude_rule(); - let composer = if text.len() <= 77 { - format!("❯\u{00a0}{text}") - } else { - let (first, continuation) = text.split_at(77); - format!("❯\u{00a0}{first}\r\n {continuation}") - }; - format!( - "Claude Code v2.1.220\r\n{rule}\r\n{composer}\r\n{rule}\r\n\ - ⏵⏵ bypass permissions on (shift+tab to cycle)" - ) - } - - /// A pane that has been used: the startup banner has scrolled out of the peeked viewport, the - /// composer is empty rather than showing a rotating placeholder, and the footer omits the - /// conditional `(shift+tab to cycle)` hint while keeping the permission-mode indicator. - fn mature_claude_screen(composer: &str) -> String { - let rule = claude_rule(); - let footer = " ⏵⏵ bypass permissions on · PR #42 · 2 shells · ← 1 agent"; - format!(" an earlier turn\r\n{rule}\r\n{composer}\r\n{rule}\r\n{footer}") - } - - fn mature_idle_claude_screen() -> String { - mature_claude_screen("❯\u{00a0}") - } - - fn mature_idle_claude_screen_with_hint() -> String { - mature_idle_claude_screen().replace( - "⏵⏵ bypass permissions on", - "⏵⏵ bypass permissions on (shift+tab to cycle)", - ) - } - - /// The same used pane in accept-edits mode. `permissions on` is specific to the bypass footer, - /// so no accept-edits or auto pane is positively idle to this classifier. - fn mature_idle_accept_edits_claude_screen() -> String { - mature_idle_claude_screen().replace("⏵⏵ bypass permissions on", "⏵⏵ accept edits on") - } - - fn mature_staged_claude_screen(text: &str) -> String { - mature_claude_screen(&format!("❯\u{00a0}{text}")) - } - - /// Selection region captured with `pty peek --plain st2-codex-selection-fixture` from a - /// dedicated 80x24 Codex CLI 0.145.0 pane on 2026-08-04. Codex rendered this built-in trust - /// selection after `codex --no-alt-screen -a untrusted -s read-only` launched in a fresh - /// untrusted directory. The command used no bypass flag. The leading working-directory row is - /// intentionally omitted; every retained row is verbatim from the captured selection. - const CAPTURED_CODEX_TRUST_SELECTION: &str = " - Do you trust the contents of this directory? Working with untrusted contents - comes with higher risk of prompt injection. Trusting the directory allows - project-local config, hooks, and exec policies to load. - -› 1. Yes, continue - 2. No, quit - - Press enter to continue"; - - /// Exact public-safe suffix from default raw `pty peek` while Codex CLI 0.145.0's `/model` - /// picker was open. It begins at the picker heading and preserves every emitted byte through - /// the end of stdout. - const CAPTURED_CODEX_MODEL_PICKER: &str = - include_str!("fixtures/codex-model-picker-default-to-end.bin"); - - const CODEX_LATENCY_RETRY_NOTICE: &str = - "Our systems are thinking a bit more\r\n\r\nRetry with a faster model"; - - /// Form region captured with `pty peek --plain st2-claude-form-fixture-clean` from a dedicated - /// 80x24 Claude Code 2.1.220 pane in safe/manual mode on 2026-08-03. Claude rendered this after - /// an `AskUserQuestion` call. The host-local banner and working directory are intentionally - /// omitted; every retained row is verbatim from the captured form. - const CAPTURED_CLAUDE_QUESTION_FORM: &str = " -──────────────────────────────────────────────────────────────────────────────── -☐ Fixture - -Which fixture should this test use? - -❯ 1. Captured form - Use a captured form fixture - 2. Plain composer - Use a plain composer fixture\u{20} - 3. Type something. -──────────────────────────────────────────────────────────────────────────────── - 4. Chat about this - -Enter to select · ↑/↓ to navigate · Esc to cancel"; - - fn captured_claude_question_form_with_selection(option: usize) -> String { - assert!((1..=4).contains(&option)); - if option == 1 { - return CAPTURED_CLAUDE_QUESTION_FORM.to_string(); - } - CAPTURED_CLAUDE_QUESTION_FORM - .replacen("❯ 1.", " 1.", 1) - .replacen(&format!(" {option}."), &format!("❯ {option}."), 1) - } - - /// The captured form reduced only by deleting its third and fourth option rows. This pins the - /// real two-option shape without adding any screen layout that was not present in the capture. - fn captured_two_option_claude_question_form() -> String { - CAPTURED_CLAUDE_QUESTION_FORM.replace( - &format!( - " 3. Type something.\n{}\n 4. Chat about this\n\n", - claude_rule() - ), - "", - ) - } - - /// The same used pane with an in-flight turn: a spinner status line above the composer. Every - /// frame below was observed on a real 2.1.220 pane; the glyph animates and the elapsed timer is - /// not always rendered, so both variations appear here. - fn mid_turn_claude_screen(status: &str, composer: &str) -> String { - let rule = claude_rule(); - let footer = " ⏵⏵ bypass permissions on · PR #42 · 2 shells · ← 1 agent"; - format!(" an earlier turn\r\n{status}\r\n{rule}\r\n{composer}\r\n{rule}\r\n{footer}") - } - - /// Status lines for an ACTIVE turn — Return must never be sent. - const ACTIVE_TURN_STATUS: [&str; 4] = [ - "✻ Frolicking… (3m 35s · ↓ 6.9k tokens)", - "✽ Schlepping…", - "· Metamorphosing…", - "✶ Schlepping… (9s · ↓ 296 tokens · thinking with high effort)", - ]; - - /// Status lines for a FINISHED turn — these sit above every genuinely idle composer, so - /// treating them as blocked would stop delivery entirely. - const FINISHED_TURN_STATUS: [&str; 4] = [ - "✻ Brewed for 5s", - "✻ Crunched for 7s", - "✻ Cogitated for 11s · 1 shell still running", - "✻ Baked for 3s · 1 shell still running", - ]; - - /// A live Claude composer with a stale Codex composer above it in scrollback, preceded by - /// escape-heavy output. The escapes inflate the Codex byte offset far past the Claude - /// composer's row, which is what makes the two locators' units observably disagree. - fn live_claude_below_escape_heavy_codex_transcript() -> String { - let padding = - "\x1b[1;32m\x1b[38;5;204mpadding with lots of escapes\x1b[0m\x1b[0m\r\n".repeat(10); - let codex = staged_codex_screen("a stale pasted codex draft"); - let filler = "\x1b[1;32mmore padding\x1b[0m\r\n".repeat(6); - format!( - "{padding}{codex}\r\n{filler}{}", - mature_idle_claude_screen() - ) - } - - /// A Codex pane whose scrollback holds a captured Claude screen — two ruled lines around a `❯` - /// row plus a Claude idle footer — above the live, ANSI-detected Codex composer. Capturing and - /// pasting pane text is routine, so this shape is not exotic. - fn codex_screen_below_claude_transcript(transcript_row: &str, codex: &str) -> String { - let rule = claude_rule(); - format!( - " scrollback: a pasted Claude pane\r\n{rule}\r\n❯\u{00a0}{transcript_row}\r\n{rule}\r\n\ - \u{0020} ⏵⏵ bypass permissions on (shift+tab to cycle)\r\n\r\n{codex}" - ) - } - - /// An in-flight Claude turn ships no interrupt hint, so the composer being empty is not proof - /// that Return is safe — the pane is working. The finished-turn line looks almost identical and - /// sits above every genuinely idle composer, so both directions are pinned here. - #[test] - fn an_in_flight_turn_blocks_return_but_a_finished_one_does_not() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - - for status in ACTIVE_TURN_STATUS { - // Empty composer mid-turn: positively empty, but not safe. - assert_ne!( - classify_composer(&mid_turn_claude_screen(status, "❯\u{00a0}"), expected), - ComposerState::EmptySafe, - "active turn must not be EmptySafe: {status}" - ); - // The notice already staged mid-turn: exact, but Return is still withheld. - assert_eq!( - classify_composer( - &mid_turn_claude_screen(status, &format!("❯\u{00a0}{expected}")), - expected - ), - ComposerState::ExactBlocked, - "active turn must be ExactBlocked: {status}" - ); - } - - // A finished turn is the normal idle screen. Blocking on it would stop delivery forever. - for status in FINISHED_TURN_STATUS { - assert_eq!( - classify_composer(&mid_turn_claude_screen(status, "❯\u{00a0}"), expected), - ComposerState::EmptySafe, - "finished turn must stay deliverable: {status}" - ); - assert_eq!( - classify_composer( - &mid_turn_claude_screen(status, &format!("❯\u{00a0}{expected}")), - expected - ), - ComposerState::ExactSafe, - "finished turn must stay submittable: {status}" - ); - } - } - - #[test] - fn claude_question_form_blocks_while_an_ordinary_idle_composer_stays_deliverable() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - - assert!(composer::looks_like_choice_menu( - CAPTURED_CLAUDE_QUESTION_FORM - )); - assert_eq!( - classify_composer( - &format!( - "{CAPTURED_CLAUDE_QUESTION_FORM}\r\n{}", - mature_staged_claude_screen(expected) - ), - expected - ), - ComposerState::ExactBlocked - ); - - assert_eq!( - classify_composer(&mature_idle_claude_screen(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&mature_staged_claude_screen(expected), expected), - ComposerState::ExactSafe - ); - } - - #[test] - fn codex_trust_selection_blocks_while_an_ordinary_idle_composer_stays_deliverable() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - - assert!(composer::looks_like_choice_menu( - CAPTURED_CODEX_TRUST_SELECTION - )); - assert_eq!( - classify_composer(CAPTURED_CODEX_TRUST_SELECTION, expected), - ComposerState::Ambiguous - ); - assert_eq!( - classify_composer( - &format!( - "{CAPTURED_CODEX_TRUST_SELECTION}\r\n{}", - staged_codex_screen(expected) - ), - expected - ), - ComposerState::ExactBlocked - ); - - assert_eq!( - classify_composer(&idle_codex_screen(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&staged_codex_screen(expected), expected), - ComposerState::ExactSafe - ); - } - - #[test] - fn captured_codex_model_picker_is_recognized_as_a_choice_menu() { - let plain = composer::strip_ansi(CAPTURED_CODEX_MODEL_PICKER); - - assert!(composer::looks_like_choice_menu(&plain)); - assert_eq!( - classify_composer(CAPTURED_CODEX_MODEL_PICKER, "unused notice"), - ComposerState::Ambiguous - ); - - let without_selected_option = plain.replacen("› 1.", " 1.", 1); - assert!(!composer::looks_like_choice_menu(&without_selected_option)); - } - - #[test] - fn fresh_delivery_does_not_return_after_idle_changes_to_model_picker() { - use std::os::unix::fs::PermissionsExt as _; - - let text = "[DING] unread st2 messages remain; check your inbox"; - let temp = tempfile::tempdir().unwrap(); - let bin = temp.path().join("pty"); - let idle = temp.path().join("idle.bin"); - let picker = temp.path().join("picker.bin"); - let changed = temp.path().join("changed"); - let audit = temp.path().join("audit.log"); - std::fs::write(&idle, idle_codex_screen()).unwrap(); - std::fs::write(&picker, CAPTURED_CODEX_MODEL_PICKER).unwrap(); - std::fs::write( - &bin, - format!( - "#!/bin/sh\nprintf '%s\\n' \"$*\" >> '{}'\n\ - if [ \"$1\" = peek ]; then\n\ - if [ -e '{}' ]; then cat '{}'; else cat '{}'; fi\n\ - exit 0\n\ - fi\n\ - if [ \"$1\" = send ]; then : > '{}'; exit 0; fi\n\ - exit 1\n", - audit.display(), - changed.display(), - picker.display(), - idle.display(), - changed.display(), - ), - ) - .unwrap(); - std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).unwrap(); - - let poker = PtyPoker { - bin: bin.to_string_lossy().into_owned(), - session: "fixture".to_string(), - }; - assert_eq!( - poker.poke_with(text, &mut || Ok(())).unwrap(), - PokeOutcome::Staged - ); - - let audit = std::fs::read_to_string(audit).unwrap(); - assert_eq!(audit.lines().next(), Some("peek fixture")); - assert!( - !audit.contains("key:return"), - "Return must be withheld after the staged paste exposes a picker: {audit:?}" - ); - } - - #[test] - fn codex_latency_retry_notice_blocks_without_choice_menu_structure() { - let expected = "[DING] ? cos: latency control [id:abc123]"; - - assert!(!composer::looks_like_choice_menu( - CODEX_LATENCY_RETRY_NOTICE - )); - assert_eq!( - classify_composer( - &format!( - "{CODEX_LATENCY_RETRY_NOTICE}\r\n{}", - staged_codex_screen(expected) - ), - expected, - ), - ComposerState::ExactBlocked - ); - } - - #[test] - fn claude_question_form_blocks_after_selection_moves_to_second_or_last_option() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - - for option in [2, 4] { - let form = captured_claude_question_form_with_selection(option); - assert!( - composer::looks_like_choice_menu(&form), - "selection on option {option} must remain modal" - ); - assert_eq!( - classify_composer( - &format!("{form}\r\n{}", mature_staged_claude_screen(expected)), - expected - ), - ComposerState::ExactBlocked, - "selection on option {option} must block Return" - ); - } - } - - #[test] - fn two_option_claude_question_form_blocks_return() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - let form = captured_two_option_claude_question_form(); - - assert!(!form.contains(" 3.")); - assert!(!form.contains(" 4.")); - assert!(composer::looks_like_choice_menu(&form)); - assert_eq!( - classify_composer( - &format!("{form}\r\n{}", mature_staged_claude_screen(expected)), - expected - ), - ComposerState::ExactBlocked - ); - } - - #[test] - fn legacy_numbered_choice_menu_still_blocks_return() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - let legacy_menu = "› 1. Continue\r\n 2. Cancel"; - - assert!(composer::looks_like_choice_menu(legacy_menu)); - assert_eq!( - classify_composer( - &format!("{legacy_menu}\r\n{}", mature_staged_claude_screen(expected)), - expected - ), - ComposerState::ExactBlocked - ); - } - - /// The two locators do not natively work in the same units. Codex is matched with `rfind` over - /// the raw screen, so it reports a **byte offset** inflated by every escape sequence above it; - /// Claude is matched over stripped lines, so it reports a **row**. Comparing those directly - /// picks Codex almost always, since an offset dwarfs a row — including when the live composer - /// is Claude's and the Codex match is stale scrollback. Both must be normalized to a row. - /// - /// On the screen below, measured: the Codex composer sits at byte offset 560 but row 10, while - /// the live Claude composer is row 20. Comparing row against offset picks Codex, so this would - /// classify from a pasted draft instead of the real composer. - #[test] - fn composer_positions_are_compared_as_rows_not_raw_byte_offsets() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - assert_eq!( - classify_composer(&live_claude_below_escape_heavy_codex_transcript(), expected), - ComposerState::EmptySafe - ); - } - - /// Normalizing the Codex offset means counting newlines in the *stripped* prefix, which is only - /// faithful if stripping preserves them. It does for well-formed input. It does not for an - /// unterminated sequence: the CSI scanner runs until a byte in `0x40..=0x7e` and `\n` is `0x0a`, - /// so it eats newlines, and an unterminated OSC consumes to the end of input. Both are recorded - /// here so a future change to `strip_ansi` cannot silently shift every row. - #[test] - fn stripping_preserves_newlines_for_well_formed_sequences_only() { - let nl = |text: &str| composer::strip_ansi(text).matches('\n').count(); - - assert_eq!( - nl("\x1b[1;32mone\x1b[0m\r\n\x1b[2Ctwo\x1b[0m\r\n\x1b[1mthree\x1b[0m\r\n"), - 3 - ); - // Unterminated CSI: the newline is consumed while hunting for a final byte. - assert_eq!(nl("before\r\n\x1b[999999\r\nafter\r\n"), 2); - // Unterminated OSC: everything to the end of input is consumed. - assert_eq!(nl("before\r\n\x1b]0;no terminator\r\nafter\r\n"), 1); - } - - /// Scrollback that merely looks like a composer must never outrank the live one. The paste and - /// the Return always go to the pane's real bottom composer, so misreading transcript text as - /// "idle" or "already staged" is a wrong positive: it can type into, or submit, a human draft. - #[test] - fn transcript_composers_never_outrank_the_live_bottom_composer() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - - // The live Codex composer holds a human draft in both cases, so both must stay `Changed`. - // An empty transcript row would otherwise read as positively-empty and allow the paste. - assert_eq!( - classify_composer( - &codex_screen_below_claude_transcript("", &human_codex_screen()), - expected - ), - ComposerState::Changed - ); - // A transcript row holding the exact notice is the worse case: it would otherwise satisfy - // the two adjacent exact observations and send a bare Return to the draft. - assert_eq!( - classify_composer( - &codex_screen_below_claude_transcript(expected, &human_codex_screen()), - expected - ), - ComposerState::Changed - ); - - // The rule is positional, not a Codex preference: a genuine Claude pane whose scrollback - // shows a captured Codex composer still classifies from its own live Claude composer. - assert_eq!( - classify_composer( - &format!( - "{}\r\n{}", - staged_codex_screen("a stale pasted codex draft"), - mature_idle_claude_screen() - ), - expected - ), - ComposerState::EmptySafe - ); - } - - #[test] - fn maintained_composer_classifiers_require_exact_idle_state() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - assert_eq!( - classify_composer(&idle_codex_screen(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&idle_codex_screen_with_home_relative_cwd(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&staged_codex_screen(expected), expected), - ComposerState::ExactSafe - ); - assert_eq!( - classify_composer(&human_codex_screen(), expected), - ComposerState::Changed - ); - assert_eq!( - classify_composer( - &format!("Create a plan?\r\n{}", staged_codex_screen(expected)), - expected - ), - ComposerState::ExactBlocked - ); - - assert_eq!( - classify_composer(&idle_claude_screen(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&idle_claude_screen_without_hint(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&staged_claude_screen(expected), expected), - ComposerState::ExactSafe - ); - assert_eq!( - classify_composer(&staged_claude_screen("a changed human composer"), expected), - ComposerState::Changed - ); - assert_eq!( - classify_composer( - &format!("Esc to interrupt\r\n{}", staged_claude_screen(expected)), - expected - ), - ComposerState::ExactBlocked - ); - // A used pane must classify exactly like a fresh one: neither the scrolled-away banner, the - // empty composer, nor the missing cycle hint is evidence that Return is unsafe. - assert_eq!( - classify_composer(&mature_idle_claude_screen(), expected), - ComposerState::EmptySafe - ); - assert_eq!( - classify_composer(&mature_idle_claude_screen_with_hint(), expected), - ComposerState::EmptySafe - ); - // Only the bypass footer carries `permissions on`, so an otherwise identical accept-edits - // pane is never positively idle. It stays unsubmitted rather than being proven safe. - assert_eq!( - classify_composer(&mature_idle_accept_edits_claude_screen(), expected), - ComposerState::Changed - ); - assert_eq!( - classify_composer(&mature_staged_claude_screen(expected), expected), - ComposerState::ExactSafe - ); - // The same pane shape must still fail closed on a human draft and on an active turn. - assert_eq!( - classify_composer( - &mature_staged_claude_screen("a changed human composer"), - expected - ), - ComposerState::Changed - ); - assert_eq!( - classify_composer( - &format!( - "Esc to interrupt\r\n{}", - mature_staged_claude_screen(expected) - ), - expected - ), - ComposerState::ExactBlocked - ); - - assert_eq!( - classify_composer("unknown terminal pixels", expected), - ComposerState::Ambiguous - ); - } - - #[test] - fn maintained_codex_context_footers_are_narrow_and_position_bound() { - let expected = "[DING] ? cos: exact observation [id:abc123]"; - - for footer in [ - "gpt-5.6-sol xhigh · ding-fix · Context 73% left", - "gpt-5.6-sol xhigh · Context 0% used", - "gpt-5.6-sol xhigh · Context 100% left", - ] { - assert_eq!( - classify_composer(&idle_codex_screen_with_footer(footer), expected), - ComposerState::EmptySafe, - "maintained empty composer footer: {footer}" - ); - assert_eq!( - classify_composer(&staged_codex_screen_with_footer(expected, footer), expected), - ComposerState::ExactSafe, - "maintained staged composer footer: {footer}" - ); - } - - for footer in [ - "gpt-5.6-sol xhigh · ding-fix", - "gpt-5.6-sol xhigh · Contextual 73% left", - "gpt-5.6-sol xhigh · Context 73 left", - "gpt-5.6-sol xhigh · Context 101% left", - "gpt-5.6-sol xhigh · Context left 73%", - "gpt-5.6-sol xhigh · Context 73% remaining", - ] { - assert_eq!( - classify_composer(&idle_codex_screen_with_footer(footer), expected), - ComposerState::Ambiguous, - "unsupported empty composer footer: {footer}" - ); - assert_eq!( - classify_composer(&staged_codex_screen_with_footer(expected, footer), expected), - ComposerState::ExactBlocked, - "unsupported staged composer footer: {footer}" - ); - } - - let maintained_footer = "gpt-5.6-sol xhigh · ding-fix · Context 73% left"; - let branch_only_footer = "gpt-5.6-sol xhigh · ding-fix"; - assert_eq!( - classify_composer( - &format!( - "{maintained_footer}\r\n{}", - staged_codex_screen_with_footer(expected, branch_only_footer) - ), - expected - ), - ComposerState::ExactBlocked, - "a valid-looking transcript row above the live composer is not its footer" - ); - assert_eq!( - classify_composer( - &staged_codex_screen_with_footer( - "please keep my half-written draft", - maintained_footer - ), - expected - ), - ComposerState::Changed - ); - for footer in [ - format!("{maintained_footer}\r\n{maintained_footer}"), - format!("{maintained_footer}\r\n{branch_only_footer}"), - format!("{maintained_footer}\r\nunknown trailing chrome"), - ] { - assert_eq!( - classify_composer(&idle_codex_screen_with_footer(&footer), expected), - ComposerState::Ambiguous, - "duplicated or trailing footer chrome must not prove an empty composer idle: {footer}" - ); - assert_eq!( - classify_composer( - &staged_codex_screen_with_footer(expected, &footer), - expected - ), - ComposerState::ExactBlocked, - "duplicated or trailing footer chrome must not prove Return safe: {footer}" - ); - } - for blocking_chrome in [ - "Esc to interrupt", - "Create a plan?", - "› 1. Allow\r\n 2. Deny", - ] { - assert_eq!( - classify_composer( - &format!( - "{blocking_chrome}\r\n{}", - staged_codex_screen_with_footer(expected, maintained_footer) - ), - expected - ), - ComposerState::ExactBlocked, - "blocking chrome: {blocking_chrome}" - ); - } - } - - #[test] - fn codex_renderer_wraps_preserve_possible_inter_word_spaces() { - let (expected, screen) = staged_wrapped_codex_screen(); - let token_split = expected.replace("enough content", "enoughcontent"); - let changed = expected.replace("synthetic", "different"); - - assert_eq!( - classify_composer(&screen, expected), - ComposerState::ExactSafe - ); - assert_eq!( - classify_receipt(&screen, expected), - ReceiptState::RetainedSafe - ); - assert_eq!( - classify_composer(&screen, &token_split), - ComposerState::ExactSafe - ); - assert_eq!(classify_composer(&screen, &changed), ComposerState::Changed); - assert_eq!( - classify_receipt(&screen, &changed), - ReceiptState::NotRetained - ); - } - - #[test] - fn codex_word_wraps_preserve_short_rows_before_continuations() { - let expected = "[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source consistency complete (from dev3.compoundingtech.st2.message-sent.orchestration); check your inbox"; - let screen = "\x1b[1m›\x1b[1C\x1b[0m[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source\r\n consistency complete (from dev3.compoundingtech.st2.message-\r\n sent.orchestration); check your inbox\r\n\r\n\x1b[2C\x1b[0mgpt-5.6-sol xhigh · /workspace"; - - assert_eq!( - classify_composer(screen, expected), - ComposerState::ExactSafe - ); - assert_eq!( - classify_receipt(screen, expected), - ReceiptState::RetainedSafe - ); - - let human_draft = screen.replace("light-work receipt", "human draft"); - assert_eq!( - classify_composer(&human_draft, expected), - ComposerState::Changed - ); - assert_eq!( - classify_receipt(&human_draft, expected), - ReceiptState::NotRetained - ); - - let unfamiliar_shape = screen.replace("\r\n consistency", "\r\nconsistency"); - assert_eq!( - classify_composer(&unfamiliar_shape, expected), - ComposerState::Ambiguous - ); - assert_eq!( - classify_receipt(&unfamiliar_shape, expected), - ReceiptState::Unproven - ); - } - - #[test] - fn codex_hard_newline_shape_is_knowingly_admitted_until_issue_250() { - let expected = "[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source consistency complete (from dev3.compoundingtech.st2.message-sent.orchestration); check your inbox"; - // A buffer containing hard newlines plus two literal spaces renders identically to this - // soft wrap. The screen-only ambiguity is accepted until https://github.com/compoundingtech/st2/issues/250. - let screen = "\x1b[1m›\x1b[1C\x1b[0m[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source\r\n consistency complete (from dev3.compoundingtech.st2.message-\r\n sent.orchestration); check your inbox\r\n\r\n\x1b[2C\x1b[0mgpt-5.6-sol xhigh · /workspace"; - - assert_eq!( - classify_composer(screen, expected), - ComposerState::ExactSafe - ); - assert_eq!( - classify_receipt(screen, expected), - ReceiptState::RetainedSafe - ); - } - - #[test] - fn claude_word_wraps_preserve_short_rows_before_continuations() { - let expected = "[DING] ? dev3.dotfiles.st2.claude-composer.worker: inspect the composer word wrapping behavior (from dev3.dotfiles.st2.main.orchestration); check your inbox [id:abc123]"; - let rule = claude_rule(); - let screen = format!( - "Claude Code v2.1.220\r\n{rule}\r\n\ - ❯\u{00a0}[DING] ? dev3.dotfiles.st2.claude-composer.worker: inspect the\r\n \ - composer word wrapping behavior (from\r\n \ - dev3.dotfiles.st2.main.orchestration); check your inbox\r\n \ - [id:abc123]\r\n{rule}\r\n\ - ⏵⏵ bypass permissions on (shift+tab to cycle)" - ); - - assert_eq!( - classify_composer(&screen, expected), - ComposerState::ExactSafe - ); - assert_eq!( - classify_receipt(&screen, expected), - ReceiptState::RetainedSafe - ); - - let human_draft = screen.replace("composer word", "human draft"); - assert_eq!( - classify_composer(&human_draft, expected), - ComposerState::Changed - ); - assert_eq!( - classify_receipt(&human_draft, expected), - ReceiptState::NotRetained - ); - - let unfamiliar_shape = screen.replace("\r\n composer", "\r\ncomposer"); - assert_eq!( - classify_composer(&unfamiliar_shape, expected), - ComposerState::Ambiguous - ); - assert_eq!( - classify_receipt(&unfamiliar_shape, expected), - ReceiptState::Unproven - ); - } - - #[test] - fn preserved_clean_codex_context_footer_is_idle() { - // Composer-to-end slice of clean zero-turn capture bb1eddba2725672287e4a73b0aaf50652250611566d9826977dc3c6639d70360. - let screen = "\x1b[1m›\x1b[22m \x1b[2mImprove documentation in @filename\r\n\x1b[22m \r\n\x1b[0m\x1b[2X\x1b[2C\x1b[38;2;246;226;183mgpt-5.6-sol low\x1b[39;2m · \x1b[38;2;242;181;144;22mContext 0% used\x1b[45X\x1b[2A\x1b[33D\x1b[0m\x1b[?2004h\x1b[?1004h\x1b[?1049l\x1b[?1l\x1b[?7h\x1b[?6l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1004l\x1b[?1006l\x1b[?25h\x1b[?2004l\x1b[4l\x1b[r\x1b[0m\x1b[0 q\x1b>\x1b(B\x1b[<99u\x1b[999;1H\n"; - - assert_eq!( - classify_composer(screen, "generic staged notice"), - ComposerState::EmptySafe - ); - } - - #[test] - fn preserved_post_turn_codex_frame_stays_ambiguous() { - // Composer-to-end slice of post-turn capture f4abd2a65c008db1f52d62fa53f813dedde5c830632133c80a732dfbe12b95fe. - let screen = "\x1b[48;2;30;30;30m \x1b[79X\r\n\x1b[1m›\x1b[22m \x1b[2mFind and fix a bug in @filename\x1b[47X\r\n\x1b[22m \x1b[79X\r\n\x1b[0m \x1b[38;2;246;226;183mgpt-5.6-sol low\x1b[39;2mi· \x1b[38;2;242;181;144;22mContext 3% used\x1b[2A\x1b[33D\x1b[0m\x1b[?2004h\x1b[?1004h\x1b[?1049l\x1b[?1l\x1b[?7h\x1b[?6l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1004l\x1b[?1006l\x1b[?25h\x1b[?2004l\x1b[4l\x1b[r\x1b[0m\x1b[0 q\x1b>\x1b(B\x1b[<99u\x1b[999;1H\n"; - - assert_eq!( - classify_composer(screen, "generic staged notice"), - ComposerState::Ambiguous - ); - } - - #[test] - fn strip_ansi_consumes_designate_g0_charset_sequence() { - assert_eq!( - composer::strip_ansi("Context 0% used\x1b>\x1b(B"), - "Context 0% used" - ); - } - - #[test] - fn startup_adopts_only_an_exact_recovery_or_backlog_composer() { - let recovery = RECOVERY_POKE.to_string(); - let backlog = "[DING] ? cos: seeded [id:abc123]".to_string(); - let candidates = vec![recovery.clone(), backlog.clone()]; - assert_eq!( - exact_staged_candidate(&staged_codex_screen(&backlog), &candidates), - Some(backlog.clone()) - ); - assert_eq!( - exact_staged_candidate(&staged_claude_screen(&recovery), &candidates), - Some(recovery) - ); - assert_eq!( - exact_staged_candidate(&human_codex_screen(), &candidates), - None - ); - } - - #[test] - fn paste_then_two_exact_observations_precede_return() { - use std::cell::RefCell; - - let text = "[DING] ? cos: ordered [id:abc123]"; - let screens = RefCell::new(VecDeque::from([ - idle_codex_screen(), - staged_codex_screen(text), - staged_codex_screen(text), - accepted_codex_screen(text), - ])); - let actions = RefCell::new(Vec::new()); - let outcome = observed_poke_with_window( - text, - &mut || { - actions.borrow_mut().push("peek"); - Ok(screens.borrow_mut().pop_front().unwrap()) - }, - &mut || { - actions.borrow_mut().push("paste"); - Ok(()) - }, - &mut || { - actions.borrow_mut().push("return"); - Ok(()) - }, - &mut || actions.borrow_mut().push("poll"), - &mut || { - actions.borrow_mut().push("receipt"); - Ok(()) - }, - Duration::from_millis(10), - ) - .unwrap(); - assert_eq!(outcome, PokeOutcome::Delivered); - assert_eq!( - *actions.borrow(), - ["peek", "paste", "peek", "peek", "receipt", "return", "peek"] - ); - } - - #[test] - fn changed_modal_ambiguous_and_bounded_timeout_never_return() { - use std::cell::RefCell; - - let text = "[DING] ? cos: guarded [id:abc123]"; - for screen in [ - human_codex_screen(), - format!("Create a plan?\r\n{}", staged_codex_screen(text)), - "unrecognized renderer".to_string(), - ] { - let actions = RefCell::new(Vec::new()); - let outcome = observed_poke_with_window( - text, - &mut || { - actions.borrow_mut().push("peek"); - Ok(screen.clone()) - }, - &mut || { - actions.borrow_mut().push("paste"); - Ok(()) - }, - &mut || { - actions.borrow_mut().push("return"); - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(); - assert_ne!(outcome, PokeOutcome::Delivered); - assert!(!actions.borrow().contains(&"return")); - assert!(!actions.borrow().contains(&"paste")); - } - - let screens = RefCell::new(VecDeque::from([idle_claude_screen(), idle_claude_screen()])); - let actions = RefCell::new(Vec::new()); - let outcome = observed_poke_with_window( - text, - &mut || { - actions.borrow_mut().push("peek"); - Ok(screens.borrow_mut().pop_front().unwrap()) - }, - &mut || { - actions.borrow_mut().push("paste"); - Ok(()) - }, - &mut || { - actions.borrow_mut().push("return"); - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(); - assert_eq!(outcome, PokeOutcome::Staged); - assert_eq!(*actions.borrow(), ["peek", "paste", "peek"]); - } - - #[test] - fn final_observation_change_and_staged_retry_are_fail_closed() { - use std::cell::RefCell; - - let text = "[DING] ? cos: final race [id:abc123]"; - let screens = RefCell::new(VecDeque::from([ - idle_codex_screen(), - staged_codex_screen(text), - human_codex_screen(), - ])); - let actions = RefCell::new(Vec::new()); - let outcome = observed_poke_with_window( - text, - &mut || Ok(screens.borrow_mut().pop_front().unwrap()), - &mut || { - actions.borrow_mut().push("paste"); - Ok(()) - }, - &mut || { - actions.borrow_mut().push("return"); - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::from_millis(10), - ) - .unwrap(); - assert_eq!(outcome, PokeOutcome::Staged); - assert_eq!(*actions.borrow(), ["paste"]); - - let retry_screens = RefCell::new(VecDeque::from([ - staged_claude_screen(text), - staged_claude_screen(text), - format!("❯\u{00a0}{text}\r\n{}", idle_claude_screen()), - ])); - let retry_actions = RefCell::new(Vec::new()); - assert_eq!( - retry_staged_with_window( - text, - &mut || Ok(retry_screens.borrow_mut().pop_front().unwrap()), - &mut || { - retry_actions.borrow_mut().push("return"); - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(), - PokeOutcome::Delivered - ); - assert_eq!(*retry_actions.borrow(), ["return"]); - } - - #[test] - fn successful_transport_with_retained_or_unproven_pixels_is_not_delivered() { - use std::cell::RefCell; - - let text = "[DING] ? cos: receipt truth [id:abc123]"; - for screen in [ - staged_codex_screen(text), - idle_codex_screen(), - human_codex_screen(), - "unknown renderer".to_string(), - ] { - let actions = RefCell::new(Vec::new()); - let outcome = transport_and_observe_with_window( - text, - &mut || { - actions.borrow_mut().push("transport"); - Ok(()) - }, - &mut || { - actions.borrow_mut().push("peek"); - Ok(screen.clone()) - }, - &mut || actions.borrow_mut().push("poll"), - &mut || { - actions.borrow_mut().push("before-submit"); - Ok(()) - }, - Duration::ZERO, - ) - .unwrap(); - assert_eq!(outcome, PokeOutcome::Staged); - assert_eq!( - *actions.borrow(), - ["before-submit", "transport", "peek"], - "transport success alone must never become Delivered" - ); - } - } - - #[test] - fn ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership() { - use std::cell::RefCell; - - let text = "[DING] ? cos: error truth [id:abc123]"; - - let actions = RefCell::new(Vec::new()); - assert_eq!( - transport_and_observe_with_window( - text, - &mut || { - actions.borrow_mut().push("transport"); - anyhow::bail!("ambiguous transport") - }, - &mut || { - actions.borrow_mut().push("peek"); - Ok(idle_codex_screen()) - }, - &mut || {}, - &mut || { - actions.borrow_mut().push("before-submit"); - Ok(()) - }, - Duration::ZERO, - ) - .unwrap(), - PokeOutcome::Staged - ); - assert_eq!(*actions.borrow(), ["before-submit", "transport"]); - - assert_eq!( - transport_and_observe_with_window( - text, - &mut || Ok(()), - &mut || anyhow::bail!("unreadable receipt"), - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(), - PokeOutcome::Staged - ); - - let screens = RefCell::new(VecDeque::from([ - staged_codex_screen(text), - staged_codex_screen(text), - ])); - let submits = RefCell::new(0); - assert_eq!( - retry_staged_with_window( - text, - &mut || Ok(screens.borrow_mut().pop_front().unwrap()), - &mut || { - *submits.borrow_mut() += 1; - anyhow::bail!("ambiguous Return") - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(), - PokeOutcome::Staged - ); - assert_eq!(*submits.borrow(), 1); - } - - #[test] - fn adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt() { - let text = "[DING] ? cos: receipt truth [id:abc123]"; - assert_eq!( - classify_receipt(&queued_codex_screen(text), text), - ReceiptState::Accepted - ); - assert_eq!( - classify_receipt(&accepted_codex_screen(text), text), - ReceiptState::Accepted - ); - assert_eq!( - classify_receipt(&staged_codex_screen(text), text), - ReceiptState::RetainedSafe - ); - assert_eq!( - classify_receipt( - &format!("ordinary transcript: {text}\r\n{}", idle_codex_screen()), - text - ), - ReceiptState::NotRetained, - "notice text outside an adapter-recognized accepted pattern is not a receipt" - ); - assert_eq!( - classify_receipt( - &format!("old receipt: {text}\r\n{}", human_codex_screen()), - text - ), - ReceiptState::NotRetained, - "a parsed changed live composer positively excludes the exact owned notice" - ); - assert_eq!( - classify_receipt(&idle_codex_screen(), text), - ReceiptState::NotRetained - ); - assert_eq!( - classify_receipt("unknown renderer", text), - ReceiptState::Unproven, - "an unrecognized screen cannot prove that the owned notice disappeared" - ); - - assert_eq!( - classify_receipt( - &format!("❯\u{00a0}{text}\r\n{}", idle_claude_screen()), - text - ), - ReceiptState::Accepted - ); - assert_eq!( - classify_receipt(&staged_claude_screen(text), text), - ReceiptState::RetainedSafe - ); - assert_eq!( - classify_receipt( - &format!("ordinary transcript: {text}\r\n{}", idle_claude_screen()), - text - ), - ReceiptState::NotRetained - ); - assert_eq!( - classify_receipt(&idle_claude_screen(), text), - ReceiptState::NotRetained - ); - } - - #[test] - fn soft_wrap_proofs_accept_short_known_continuations() { - let text = "[DING] ? cos: receipt truth [id:abc123]"; - let (first, continuation) = text.split_at(32); - let codex = format!( - "\x1b[1m›\x1b[1C\x1b[0m{first}\r\n {continuation}\r\n\r\n\ - \x1b[2C\x1b[0mgpt-5.6-sol xhigh · /workspace" - ); - let rule = claude_rule(); - let claude = format!( - "Claude Code v2.1.220\r\n{rule}\r\n❯\u{00a0}{first}\r\n {continuation}\r\n\ - {rule}\r\n⏵⏵ bypass permissions on (shift+tab to cycle)" - ); - - assert_eq!( - ( - classify_receipt(&codex, text), - classify_receipt(&claude, text), - ), - (ReceiptState::RetainedSafe, ReceiptState::RetainedSafe), - "both maintained composers prove short rows with their two-cell continuation indent" - ); - assert_eq!( - classify_receipt(&human_codex_screen(), text), - ReceiptState::NotRetained - ); - assert_eq!( - classify_receipt(&staged_claude_screen("a changed human composer"), text), - ReceiptState::NotRetained - ); - } - - #[test] - fn staged_retry_submits_only_retained_safe_and_requires_a_receipt() { - use std::cell::RefCell; - - let text = "[DING] ? cos: retry truth [id:abc123]"; - - let retained = RefCell::new(VecDeque::from([ - staged_codex_screen(text), - staged_codex_screen(text), - staged_codex_screen(text), - ])); - let submits = RefCell::new(0); - let outcome = retry_staged_with_window( - text, - &mut || Ok(retained.borrow_mut().pop_front().unwrap()), - &mut || { - *submits.borrow_mut() += 1; - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(); - assert_eq!(outcome, PokeOutcome::Staged); - assert_eq!(*submits.borrow(), 1); - - for (screen, expected) in [ - ( - format!("Create a plan?\r\n{}", staged_codex_screen(text)), - PokeOutcome::Staged, - ), - (idle_codex_screen(), PokeOutcome::NotRetained), - (human_codex_screen(), PokeOutcome::NotRetained), - ("unknown renderer".to_string(), PokeOutcome::Staged), - ] { - let submits = RefCell::new(0); - let outcome = retry_staged_with_window( - text, - &mut || Ok(screen.clone()), - &mut || { - *submits.borrow_mut() += 1; - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(); - assert_eq!(outcome, expected); - assert_eq!( - *submits.borrow(), - 0, - "blocked or unproven receipts must not receive Return" - ); - } - - let accepted = queued_codex_screen(text); - let submits = RefCell::new(0); - assert_eq!( - retry_staged_with_window( - text, - &mut || Ok(accepted.clone()), - &mut || { - *submits.borrow_mut() += 1; - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(), - PokeOutcome::Delivered - ); - assert_eq!(*submits.borrow(), 0); - } - - #[test] - fn staged_retry_keeps_unproven_and_retained_blocked_owned() { - use std::cell::RefCell; - - let text = "[DING] ? cos: retry truth [id:abc123]"; - for screen in [ - "unknown renderer".to_string(), - format!("Create a plan?\r\n{}", staged_codex_screen(text)), - ] { - let submits = RefCell::new(0); - assert_eq!( - retry_staged_with_window( - text, - &mut || Ok(screen.clone()), - &mut || { - *submits.borrow_mut() += 1; - Ok(()) - }, - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(), - PokeOutcome::Staged - ); - assert_eq!(*submits.borrow(), 0); - } - } - - #[test] - fn pty_commands_have_a_real_outer_timeout() { - let started = Instant::now(); - let error = output_with_timeout( - Command::new("sh").args(["-c", "sleep 2"]), - Duration::from_millis(30), - ) - .unwrap_err(); - assert!(error.to_string().contains("timed out")); - assert!(started.elapsed() < Duration::from_secs(1)); - } - - #[test] - fn session_watch_has_startup_grace_debounce_and_live_reset() { - let mut watch = SessionWatch::default(); - for _ in 0..10 { - assert_eq!(watch.step(false), WatchStep::Poll); - } - assert_eq!(watch.step(true), WatchStep::Poll); - assert_eq!(watch.step(false), WatchStep::Poll); - assert_eq!(watch.step(false), WatchStep::Poll); - assert_eq!(watch.step(true), WatchStep::Poll); - assert_eq!(watch.step(false), WatchStep::Poll); - assert_eq!(watch.step(false), WatchStep::Poll); - assert_eq!(watch.step(false), WatchStep::Gone); - } - - #[test] - fn new_arrivals_is_fifo_and_archive_receipts_prevent_reding() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - let archive = archive_dir(agent.path()); - let first = send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); - std::thread::sleep(Duration::from_millis(2)); - let second = send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); - - let mut seen = HashSet::new(); - let initial = new_arrivals(&inbox, &mut seen); - assert_eq!( - initial - .iter() - .map(|message| message.filename.as_str()) - .collect::>(), - [first.as_str(), second.as_str()] - ); - assert!(new_arrivals(&inbox, &mut seen).is_empty()); - - archive_msg(&inbox, &archive, &first).unwrap(); - std::fs::copy(archive.join(&first), inbox.join(&first)).unwrap(); - assert!( - new_arrivals(&inbox, &mut seen).is_empty(), - "an archive receipt must suppress a restored inbox copy" - ); - - std::thread::sleep(Duration::from_millis(2)); - let third = send_to_inbox(&inbox, "carol", Some("third"), None, &[], "three").unwrap(); - assert_eq!( - new_arrivals(&inbox, &mut seen) - .into_iter() - .map(|message| message.filename) - .collect::>(), - [third] - ); - } - - #[test] - fn staged_ownership_survives_archive_and_never_repastes() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - let archive = archive_dir(agent.path()); - let filename = send_to_inbox(&inbox, "cos", Some("owned"), None, &[], "body").unwrap(); - let message = message::list_inbox(&inbox).unwrap().pop().unwrap(); - let expected = render_without_catalog(&message); - let mut pending = VecDeque::from([PendingNotice::message(message)]); - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), - retry_outcomes: Mutex::new(VecDeque::from([ - PokeOutcome::Staged, - PokeOutcome::Delivered, - ])), - }; - - flush_without_catalog(None, &mut pending, &poker); - assert_eq!(pending.len(), 1); - assert_eq!(pending[0].staged_text(), Some(expected.as_str())); - - archive_msg(&inbox, &archive, &filename).unwrap(); - prune_archived_pending(&inbox, &mut pending); - assert_eq!( - pending.len(), - 1, - "an already-started paste remains inspection-owned across archive" - ); - - flush_without_catalog(None, &mut pending, &poker); - assert_eq!(pending.len(), 1); - flush_without_catalog(None, &mut pending, &poker); - assert!(pending.is_empty()); - assert_eq!(poker.pokes.lock().unwrap().as_slice(), [expected.as_str()]); - assert_eq!( - poker.retries.lock().unwrap().as_slice(), - [expected.as_str(), expected.as_str()] - ); - } - - // ----------------------------------------------------------------------------------------- - // SPIKE — producer-side event supersede against staged payload ownership. - // - // Supersede is the one genuinely new semantic in - // `docs/vrs/.experiments/2026-08-20-pipes-event-model-differentiation.md`, and it archives an - // inbox item *from outside DING*, possibly while DING already owns that item's payload in a - // composer. These two tests are the seam: if either fails, the differentiated event model's - // main earned semantic is unsafe and the design collapses back to unification. - // ----------------------------------------------------------------------------------------- - - /// A real catalog with one agent-owned stream. - fn event_catalog() -> (tempfile::TempDir, PathBuf) { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "hetz", "worker", None); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let declaration = catalog.path().join("hetz/worker/agent.kdl"); - let source = std::fs::read_to_string(&declaration).unwrap(); - std::fs::write( - declaration, - source.replacen("\n}\n", "\n stream \"gh-ci\" {}\n}\n", 1), - ) - .unwrap(); - let inbox = inbox_dir(&catalog.path().join("hetz").join("worker")); - (catalog, inbox) - } - - fn emit_ci(root: &Path, event_id: &str, supersede: bool) -> String { - crate::event::emit( - root, - "hetz", - "hetz.worker", - "gh-ci", - event_id, - Some("pr-42"), - Some(&format!("CI {event_id} on PR #42")), - event_id, - supersede, - ) - .unwrap() - .filename - } - - fn flush_in(root: &Path, pending: &mut VecDeque, poker: &dyn Poker) { - flush_pending( - DingContext { - catalog_root: root, - this_host: "hetz", - recipient: "hetz.worker", - }, - None, - pending, - poker, - ); - } - - /// The race: the producer supersedes event N *while DING owns N's staged payload*. The - /// existing ownership rules must carry it — N is pasted exactly once and never again, the - /// archived-and-not-retained head releases FIFO, and N+1 still delivers. Nothing about - /// `flush_pending` or `prune_archived_pending` changes to make this true. - #[test] - fn a_producer_supersede_of_a_staged_event_never_repastes_and_the_successor_delivers() { - let (catalog, inbox) = event_catalog(); - let root = catalog.path(); - - let failure = emit_ci(root, "failure", true); - let mut seen = HashSet::new(); - let mut pending: VecDeque = new_arrivals(&inbox, &mut seen) - .into_iter() - .map(PendingNotice::message) - .collect(); - let failure_text = pending[0].text( - DingContext { - catalog_root: root, - this_host: "hetz", - recipient: "hetz.worker", - }, - &mut None, - ); - assert!( - failure_text.starts_with("[DING] » hetz.worker/gh-ci:"), - "an event announces itself as a world-event: {failure_text}" - ); - - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([ - // the successor, once ownership of the superseded head is released - PokeOutcome::Delivered, - ])), - retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::NotRetained])), - }; - // DING stages the failure notice and owns it. - let stage_only = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), - retry_outcomes: Mutex::new(VecDeque::new()), - }; - flush_in(root, &mut pending, &stage_only); - assert_eq!(pending[0].staged_text(), Some(failure_text.as_str())); - - // The producer now supersedes: `success` is materialized and `failure` is archived under - // DING's feet. - let success = emit_ci(root, "success", true); - assert!(!inbox.join(&failure).exists(), "the head was retired"); - assert!(inbox.join(&success).exists(), "the successor is unread"); - - pending.extend( - new_arrivals(&inbox, &mut seen) - .into_iter() - .map(PendingNotice::message), - ); - prune_archived_pending(&inbox, &mut pending); - assert_eq!( - pending.len(), - 2, - "the staged-but-archived head keeps ownership; the successor queues behind it" - ); - - flush_in(root, &mut pending, &poker); - - assert!(pending.is_empty(), "FIFO drained, nothing stuck"); - let success_text = poker.pokes.lock().unwrap()[0].clone(); - assert!( - success_text.contains("CI success on PR #42"), - "{success_text}" - ); - assert_eq!( - stage_only.pokes.lock().unwrap().as_slice(), - [failure_text.as_str()], - "the superseded notice was pasted exactly once, ever" - ); - assert_eq!( - poker.pokes.lock().unwrap().len(), - 1, - "and the only fresh paste after supersede is the successor" - ); - assert_eq!( - poker.retries.lock().unwrap().as_slice(), - [failure_text.as_str()], - "the superseded head was released by inspection only, never re-pasted" - ); - } - - /// The pessimistic half of the same race: the adapter still sees the superseded notice in the - /// composer. Ownership is retained, FIFO stays blocked behind it, and the successor is *not* - /// pasted on top of a live payload. Supersede therefore cannot leak a second paste into a - /// composer that is still holding the first. - #[test] - fn a_superseded_but_still_retained_staged_event_keeps_ownership_without_repasting() { - let (catalog, inbox) = event_catalog(); - let root = catalog.path(); - - emit_ci(root, "failure", true); - let mut seen = HashSet::new(); - let mut pending: VecDeque = new_arrivals(&inbox, &mut seen) - .into_iter() - .map(PendingNotice::message) - .collect(); - let failure_text = pending[0].text( - DingContext { - catalog_root: root, - this_host: "hetz", - recipient: "hetz.worker", - }, - &mut None, - ); - - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), - retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), - }; - flush_in(root, &mut pending, &poker); - - emit_ci(root, "success", true); - pending.extend( - new_arrivals(&inbox, &mut seen) - .into_iter() - .map(PendingNotice::message), - ); - prune_archived_pending(&inbox, &mut pending); - flush_in(root, &mut pending, &poker); - - assert_eq!(pending.len(), 2, "later FIFO work remains blocked"); - assert_eq!( - poker.pokes.lock().unwrap().as_slice(), - [failure_text.as_str()], - "the successor is never pasted on top of a retained payload" - ); - assert_eq!( - poker.retries.lock().unwrap().as_slice(), - [failure_text.as_str()], - "the retained superseded notice is retried by inspection only" - ); - } - - /// Platforms that cannot hardlink through the open-file descriptor path (macOS fdescfs - /// rejects linkat(AT_SYMLINK_FOLLOW) on /dev/fd/N with EPERM) fall back to a byte-copy - /// archive receipt. The differentiated supersession semantics must hold unchanged on that - /// path, the receipt must carry exactly the validated bytes, and no staging file may leak - /// into the archive. - #[test] - #[cfg(debug_assertions)] - fn archive_copy_fallback_preserves_supersede_ownership_without_staging_leftovers() { - crate::event::TEST_FORCE_ARCHIVE_RECEIPT_COPY.store(true, Ordering::Relaxed); - struct ResetGuard; - impl Drop for ResetGuard { - fn drop(&mut self) { - crate::event::TEST_FORCE_ARCHIVE_RECEIPT_COPY.store(false, Ordering::Relaxed); - } - } - let _guard = ResetGuard; - - let (catalog, inbox) = event_catalog(); - let root = catalog.path(); - let archive = crate::message::archive_dir(&root.join("hetz").join("worker")); - - let failure_filename = emit_ci(root, "failure", true); - let mut seen = HashSet::new(); - let mut pending: VecDeque = new_arrivals(&inbox, &mut seen) - .into_iter() - .map(PendingNotice::message) - .collect(); - assert_eq!(pending.len(), 1); - let failure_bytes = - std::fs::read(inbox.join(&failure_filename)).expect("staged event bytes"); - let failure_text = pending[0].text( - DingContext { - catalog_root: root, - this_host: "hetz", - recipient: "hetz.worker", - }, - &mut None, - ); - - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), - retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), - }; - flush_in(root, &mut pending, &poker); - - emit_ci(root, "success", true); - pending.extend( - new_arrivals(&inbox, &mut seen) - .into_iter() - .map(PendingNotice::message), - ); - prune_archived_pending(&inbox, &mut pending); - flush_in(root, &mut pending, &poker); - - assert_eq!(pending.len(), 2, "later FIFO work remains blocked"); - assert_eq!( - poker.pokes.lock().unwrap().as_slice(), - [failure_text.as_str()], - "the successor is never pasted on top of a retained payload" - ); - assert!(!inbox.join(&failure_filename).exists(), "head was archived"); - let receipt = std::fs::read(archive.join(&failure_filename)) - .expect("byte-copy archive receipt exists"); - assert_eq!( - receipt, failure_bytes, - "the copy receipt carries exactly the validated bytes" - ); - let staging_leftovers: Vec<_> = std::fs::read_dir(&archive) - .unwrap() - .filter_map(Result::ok) - .map(|entry| entry.file_name().to_string_lossy().into_owned()) - .filter(|name| name.starts_with(".st2-archive-")) - .collect(); - assert!( - staging_leftovers.is_empty(), - "no staging files survive in the archive: {staging_leftovers:?}" - ); - } - - #[test] - fn archived_not_retained_releases_fifo_without_repasting_owned_notice() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - let archive = archive_dir(agent.path()); - let first = send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); - std::thread::sleep(Duration::from_millis(2)); - let second = send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); - let mut pending: VecDeque = message::list_inbox(&inbox) - .unwrap() - .into_iter() - .map(PendingNotice::message) - .collect(); - let first_text = notice_text_without_catalog(&pending[0]); - let second_text = notice_text_without_catalog(&pending[1]); - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([ - PokeOutcome::Staged, - PokeOutcome::Delivered, - ])), - retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::NotRetained])), - }; - - flush_without_catalog(None, &mut pending, &poker); - archive_msg(&inbox, &archive, &first).unwrap(); - prune_archived_pending(&inbox, &mut pending); - flush_without_catalog(None, &mut pending, &poker); - - assert!(pending.is_empty()); - assert_eq!( - poker.pokes.lock().unwrap().as_slice(), - [first_text.as_str(), second_text.as_str()] - ); - assert_eq!(poker.retries.lock().unwrap().as_slice(), [first_text]); - assert!(!inbox.join(first).exists()); - assert!(inbox.join(second).exists()); - } - - #[test] - fn unread_not_retained_keeps_fifo_ownership_without_repasting() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); - std::thread::sleep(Duration::from_millis(2)); - send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); - let mut pending: VecDeque = message::list_inbox(&inbox) - .unwrap() - .into_iter() - .map(PendingNotice::message) - .collect(); - let first_text = notice_text_without_catalog(&pending[0]); - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::from([ - PokeOutcome::Staged, - PokeOutcome::Delivered, - ])), - retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::NotRetained])), - }; - - flush_without_catalog(None, &mut pending, &poker); - flush_without_catalog(None, &mut pending, &poker); - - assert_eq!(pending.len(), 2); - assert_eq!(pending[0].staged_text(), Some(first_text.as_str())); - assert_eq!( - poker.pokes.lock().unwrap().as_slice(), - [first_text.as_str()] - ); - assert_eq!(poker.retries.lock().unwrap().as_slice(), [first_text]); - } - - #[test] - fn pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - let archive = archive_dir(agent.path()); - let status_path = status::status_path(agent.path()); - let first = send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); - std::thread::sleep(Duration::from_millis(2)); - let second = send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); - let mut pending: VecDeque = message::list_inbox(&inbox) - .unwrap() - .into_iter() - .map(PendingNotice::message) - .collect(); - let poker = RecordingPoker::live(); - - archive_msg(&inbox, &archive, &first).unwrap(); - prune_archived_pending(&inbox, &mut pending); - assert_eq!( - pending - .iter() - .filter_map(|notice| match notice { - PendingNotice::Message { message, .. } => Some(message.filename.as_str()), - PendingNotice::Recovery { .. } | PendingNotice::Adopted { .. } => None, - }) - .collect::>(), - [second.as_str()] - ); - - *poker.failures.lock().unwrap() = 1; - status::set_state(&status_path, status::State::Busy).unwrap(); - flush_without_catalog(Some(&status_path), &mut pending, &poker); - assert_eq!(pending.len(), 1, "a failed head remains queued"); - flush_without_catalog(Some(&status_path), &mut pending, &poker); - assert!(pending.is_empty()); - - let third = send_to_inbox(&inbox, "carol", Some("third"), None, &[], "three").unwrap(); - pending.extend( - message::list_inbox(&inbox) - .unwrap() - .into_iter() - .filter(|message| message.filename == third) - .map(PendingNotice::message), - ); - status::set_state(&status_path, status::State::Dnd).unwrap(); - flush_without_catalog(Some(&status_path), &mut pending, &poker); - assert_eq!(pending.len(), 1, "fresh dnd suppresses delivery"); - - let stale = std::time::SystemTime::now() - status::STATUS_STALE - Duration::from_secs(1); - let stale_ms = stale - .duration_since(std::time::UNIX_EPOCH) - .unwrap() - .as_millis(); - std::fs::write(&status_path, format!("dnd\nv1 {stale_ms}\n")).unwrap(); - flush_without_catalog(Some(&status_path), &mut pending, &poker); - assert!( - pending.is_empty(), - "stale dnd reads unknown and no longer suppresses" - ); - - let calls = poker.calls.lock().unwrap(); - assert_eq!(calls.len(), 3); - assert_eq!(calls[0], calls[1], "the failed FIFO head retries first"); - assert!(calls[0].contains("second")); - assert!(calls[2].contains("third")); - } - - #[test] - fn staged_and_recovery_delivery_do_not_traverse_the_catalog() { - let catalog = tempfile::tempdir().unwrap(); - let context = DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }; - - let mut staged = - PendingNotice::message(msg("1785070000000-abc123.md", "h.sender", Some("staged"))); - staged.set_staged_text(Some("immutable staged notice".to_string())); - let mut pending = VecDeque::from([staged]); - let poker = OwnershipPoker { - pokes: Mutex::new(Vec::new()), - retries: Mutex::new(Vec::new()), - poke_outcomes: Mutex::new(VecDeque::new()), - retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Delivered])), - }; - crate::discovery::reset_discovery_walk_count_for_test(); - flush_pending(context, None, &mut pending, &poker); - assert_eq!( - crate::discovery::discovery_walk_count_for_test(), - 0, - "immutable staged delivery must not depend on catalog traversal" - ); - - let mut pending = VecDeque::from([PendingNotice::Recovery { - startup: HashSet::from(["1785070000000-def456.md".to_string()]), - in_inbox: true, - staged_text: None, - }]); - let poker = RecordingPoker::live(); - crate::discovery::reset_discovery_walk_count_for_test(); - flush_pending(context, None, &mut pending, &poker); - assert_eq!( - crate::discovery::discovery_walk_count_for_test(), - 0, - "generic recovery delivery must not depend on catalog traversal" - ); - assert_eq!(poker.calls.lock().unwrap().as_slice(), [RECOVERY_POKE]); - } - - #[test] - fn one_pending_batch_traverses_the_catalog_once() { - let catalog = tempfile::tempdir().unwrap(); - declare_agent(catalog.path(), "h", "recipient", None); - let mut pending = (0..10) - .map(|index| { - PendingNotice::message(msg( - &format!("17850700000{index:02}-abc{index:03}.md"), - "h.sender", - Some("batch"), - )) - }) - .collect::>(); - let poker = RecordingPoker::live(); - - crate::discovery::reset_discovery_walk_count_for_test(); - flush_pending( - DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }, - None, - &mut pending, - &poker, - ); - - assert!(pending.is_empty()); - assert_eq!(poker.calls.lock().unwrap().len(), 10); - assert_eq!( - crate::discovery::discovery_walk_count_for_test(), - 1, - "one pending batch must share one catalog observation" - ); - } - - #[test] - fn startup_recovery_notice_retries_in_memory() { - let poker = RecordingPoker::live(); - *poker.failures.lock().unwrap() = 1; - let mut pending = VecDeque::from([PendingNotice::Recovery { - startup: HashSet::from(["1785070000000-abc123.md".to_string()]), - in_inbox: true, - staged_text: None, - }]); - - flush_without_catalog(None, &mut pending, &poker); - assert_eq!(pending.len(), 1); - flush_without_catalog(None, &mut pending, &poker); - assert!(pending.is_empty()); - - let calls = poker.calls.lock().unwrap(); - assert_eq!(calls.len(), 2); - assert_eq!(calls[0], calls[1], "the failed FIFO head retries first"); - assert_eq!(calls[0], RECOVERY_POKE); - } - - #[test] - fn startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - let status_path = status::status_path(agent.path()); - status::set_state(&status_path, status::State::Available).unwrap(); - send_to_inbox(&inbox, "old", Some("seeded"), None, &[], "old").unwrap(); - - let poker = RecordingPoker::live(); - let stop = AtomicBool::new(false); - let config = DingConfig { - poll: Duration::from_millis(5), - status_refresh: Duration::from_secs(60), - }; - - // Two separate barriers, each with its own generous budget. Sharing one 3s deadline - // across both meant the first wait could spend it: the loop then stopped after a single - // poke and the count assertion failed while naming neither barrier. The ceiling bounds a - // loop that never ran at all, so a loaded host cannot turn it into a failure — and `stop` - // is set on every path, because panicking here would leave `run_ding` looping forever. - let mut polled = false; - let mut poked_twice = false; - std::thread::scope(|scope| { - scope.spawn(|| { - polled = await_ding_progress(|| poker.probes.load(Ordering::SeqCst) > 0); - send_to_inbox(&inbox, "new", Some("post-start"), None, &[], "new").unwrap(); - poked_twice = await_ding_progress(|| poker.calls.lock().unwrap().len() >= 2); - stop.store(true, Ordering::SeqCst); - }); - - run_ding( - DingContext { - catalog_root: agent.path(), - this_host: "h", - recipient: "h.recipient", - }, - &inbox, - Some(&status_path), - &poker, - &config, - &stop, - ) - .unwrap(); - }); - - assert!(polled, "the ding loop never reached its first poll"); - assert!(poked_twice, "the ding loop never delivered both notices"); - let calls = poker.calls.lock().unwrap(); - assert_eq!(calls.len(), 2); - assert_eq!(calls[0], RECOVERY_POKE); - assert!(calls[1].contains("post-start")); - assert!(!calls.iter().any(|call| call.contains("seeded"))); - } - - #[test] - fn deferred_delivery_backoff_bounds_short_lived_pty_attempts() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - let status_path = status::status_path(agent.path()); - status::set_state(&status_path, status::State::Available).unwrap(); - send_to_inbox(&inbox, "alice", Some("active"), None, &[], "active").unwrap(); - - let poker = RecordingPoker::live(); - poker.defer.store(true, Ordering::SeqCst); - let stop = AtomicBool::new(false); - let config = DingConfig { - poll: Duration::from_millis(20), - status_refresh: Duration::from_secs(60), - }; - - // Synchronize on the loop's own progress rather than a wall-clock window. `session_alive` - // runs exactly once per iteration, so `probes` counts polls directly: stop only once the - // first poke landed AND the loop polled several more times, which is precisely when a - // missing backoff would poke again. A fixed window instead ends wherever the host's - // scheduler leaves it — on a loaded machine before the first poll, so the test observed zero - // pokes and failed while asserting nothing about backoff. - const POLLS_AFTER_FIRST_POKE: usize = 5; - let barrier = Duration::from_secs(30); - std::thread::scope(|scope| { - scope.spawn(|| { - let deadline = Instant::now() + barrier; - while poker.calls.lock().unwrap().is_empty() && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(2)); - } - let polled = poker.probes.load(Ordering::SeqCst) + POLLS_AFTER_FIRST_POKE; - while poker.probes.load(Ordering::SeqCst) < polled && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(2)); - } - stop.store(true, Ordering::SeqCst); - }); - run_ding( - DingContext { - catalog_root: agent.path(), - this_host: "h", - recipient: "h.recipient", - }, - &inbox, - Some(&status_path), - &poker, - &config, - &stop, - ) - .unwrap(); - }); - - // Without this the counter-test is vacuous: one poke across one poll proves no backoff. - assert!( - poker.probes.load(Ordering::SeqCst) > POLLS_AFTER_FIRST_POKE, - "the loop did not poll again, so no backoff was exercised" - ); - assert_eq!( - poker.calls.lock().unwrap().len(), - 1, - "an unsafe composer must not respawn pty peek/send children every inbox poll" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn idle_ding_does_not_spin_on_its_own_inbox_reads() { - let agent = tempfile::tempdir().unwrap(); - let inbox = inbox_dir(agent.path()); - std::fs::create_dir_all(&inbox).unwrap(); - let status_path = status::status_path(agent.path()); - status::set_state(&status_path, status::State::Available).unwrap(); - - let poker = RecordingPoker::live(); - let stop = AtomicBool::new(false); - let config = DingConfig { - poll: Duration::from_millis(20), - status_refresh: Duration::from_secs(60), - }; - let started = Instant::now(); - - std::thread::scope(|scope| { - scope.spawn(|| { - std::thread::sleep(Duration::from_millis(150)); - stop.store(true, Ordering::SeqCst); - }); - run_ding( - DingContext { - catalog_root: agent.path(), - this_host: "h", - recipient: "h.recipient", - }, - &inbox, - Some(&status_path), - &poker, - &config, - &stop, - ) - .unwrap(); - }); - - assert!(started.elapsed() >= Duration::from_millis(140)); - assert!( - poker.probes.load(Ordering::SeqCst) <= 20, - "idle DING must sleep at its configured cadence" - ); - } - - #[test] - fn session_liveness_probe_reports_only_positive_evidence() { - use crate::harness_state::SessionLiveness; - let tmp = tempfile::tempdir().unwrap(); - - // An unreadable registry proves nothing: missing or garbled pidfiles downgrade nothing. - assert_eq!( - session_liveness_in(tmp.path(), "absent"), - SessionLiveness::Indeterminate - ); - std::fs::write(tmp.path().join("garbled.pid"), "not-a-pid\n").unwrap(); - assert_eq!( - session_liveness_in(tmp.path(), "garbled"), - SessionLiveness::Indeterminate - ); - - // Our own pid is positive evidence of life. - std::fs::write( - tmp.path().join("live.pid"), - format!("{}\n", std::process::id()), - ) - .unwrap(); - assert_eq!( - session_liveness_in(tmp.path(), "live"), - SessionLiveness::Alive - ); - - // A reaped child is positive evidence of death (ESRCH). - let mut child = std::process::Command::new("true").spawn().unwrap(); - let pid = child.id(); - child.wait().unwrap(); - std::fs::write(tmp.path().join("dead.pid"), format!("{pid}\n")).unwrap(); - assert_eq!( - session_liveness_in(tmp.path(), "dead"), - SessionLiveness::Dead - ); - } - - /// A deferral is the one outcome that both delivers nothing and leaves nothing behind, so it - /// has to carry why. The two causes want different responses: a human drafting in a harness we - /// understand clears on its own, a pane no harness can locate never will. - #[test] - fn a_deferral_names_the_cause_that_produced_it() { - let text = "[DING] ? cos: guarded [id:abc123]"; - for (screen, expected) in [ - ( - human_codex_screen(), - DeferralReason::ComposerChanged { harness: "codex" }, - ), - // Located, but an unrecognised footer proves nothing about the screen. This is a wait, - // not a coverage gap, and must not be reported as an unlocatable pane. - ( - idle_codex_screen_with_footer("Esc to interrupt"), - DeferralReason::ComposerUnproven { harness: "codex" }, - ), - ( - "unrecognized renderer".to_string(), - DeferralReason::NoMaintainedComposer, - ), - ] { - let outcome = observed_poke_with_window( - text, - &mut || Ok(screen.clone()), - &mut || Ok(()), - &mut || Ok(()), - &mut || {}, - &mut || Ok(()), - Duration::ZERO, - ) - .unwrap(); - assert_eq!(outcome, PokeOutcome::Deferred(expected)); - } - } - - /// The deferral has to reach the watch loop, which is the only place that knows the recipient - /// and can decide whether it is worth saying. - #[test] - fn flush_reports_a_deferral_outward() { - let catalog = tempfile::tempdir().unwrap(); - let context = DingContext { - catalog_root: catalog.path(), - this_host: "h", - recipient: "h.recipient", - }; - let notice = || { - VecDeque::from([PendingNotice::message(msg( - "1785070000000-abc123.md", - "h.sender", - Some("hello"), - ))]) - }; - - let poker = RecordingPoker::live(); - poker.defer.store(true, Ordering::SeqCst); - assert_eq!( - flush_pending(context, None, &mut notice(), &poker).deferred, - Some(DeferralReason::NoMaintainedComposer) - ); - - // A delivery that lands has nothing to report. - let poker = RecordingPoker::live(); - assert_eq!( - flush_pending(context, None, &mut notice(), &poker).deferred, - None - ); - } - - /// A pane stuck in one verdict costs one line, not one per retry — but a changed verdict, and - /// the first deferral after delivery resumed, are both news. - #[test] - fn only_a_changed_deferral_verdict_is_worth_reporting() { - let changed = DeferralReason::ComposerChanged { harness: "codex" }; - let unknown = DeferralReason::NoMaintainedComposer; - let mut journal = DeferralJournal::default(); - - assert!(journal.observe(Some(changed))); - assert!(!journal.observe(Some(changed))); - assert!(journal.observe(Some(unknown))); - assert!(!journal.observe(Some(unknown))); - assert!(!journal.observe(None)); - assert!(journal.observe(Some(unknown))); - } -} +mod tests; diff --git a/src/ding/tests.rs b/src/ding/tests.rs new file mode 100644 index 00000000..ab82aca0 --- /dev/null +++ b/src/ding/tests.rs @@ -0,0 +1,2748 @@ +use super::*; +use crate::message::{archive_dir, archive_msg, inbox_dir, send_to_inbox}; +use std::sync::Mutex; +use std::sync::atomic::AtomicUsize; + +fn msg(filename: &str, from: &str, subject: Option<&str>) -> Message { + Message { + filename: filename.to_string(), + ts_ms: filename + .split_once('-') + .and_then(|(timestamp, _)| timestamp.parse().ok()) + .unwrap_or_default(), + from: Some(from.to_string()), + subject: subject.map(str::to_string), + in_reply_to: None, + tags: vec![], + priority: None, + idempotency_key: None, + stream: None, + event_id: None, + event_key: None, + body: String::new(), + } +} + +fn declare_agent(root: &Path, host: &str, identity: &str, supervisor: Option<&str>) { + let directory = root.join(host).join(identity); + std::fs::create_dir_all(&directory).unwrap(); + let supervisor = supervisor + .map(|value| format!(" supervisor {value:?}\n")) + .unwrap_or_default(); + std::fs::write( + directory.join("agent.kdl"), + format!( + "agent {identity:?} {{\n identity {identity:?}\n host {host:?}\n{supervisor} type \"service\"\n pty \"agent\" {{ command \"x\" }}\n}}\n" + ), + ) + .unwrap(); +} + +fn render_without_catalog(message: &Message) -> String { + let catalog = tempfile::tempdir().unwrap(); + poke_text(catalog.path(), "h", "h.recipient", message) +} + +fn notice_text_without_catalog(notice: &PendingNotice) -> String { + let catalog = tempfile::tempdir().unwrap(); + notice.text( + DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }, + &mut None, + ) +} + +fn flush_without_catalog( + status_path: Option<&Path>, + pending: &mut VecDeque, + poker: &dyn Poker, +) { + let catalog = tempfile::tempdir().unwrap(); + flush_pending( + DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }, + status_path, + pending, + poker, + ); +} + +/// Block until `ready` holds, reporting whether it did. The ceiling bounds a DING loop that +/// made no progress at all; it is not the behaviour under test, so it is far larger than any +/// plausible scheduling delay and a loaded host cannot turn it into a failure. Callers report +/// the result after their scope ends rather than panicking inside it, because an unwind from a +/// scoped thread leaves `run_ding` without the `stop` flag that ends it. +fn await_ding_progress(ready: impl Fn() -> bool) -> bool { + let deadline = Instant::now() + Duration::from_secs(30); + while !ready() { + if Instant::now() >= deadline { + return false; + } + std::thread::sleep(Duration::from_millis(2)); + } + true +} + +#[derive(Default)] +struct RecordingPoker { + alive: AtomicBool, + defer: AtomicBool, + probes: AtomicUsize, + failures: Mutex, + calls: Mutex>, +} + +impl RecordingPoker { + fn live() -> Self { + Self { + alive: AtomicBool::new(true), + ..Default::default() + } + } +} + +impl Poker for RecordingPoker { + fn poke(&self, text: &str) -> anyhow::Result { + self.calls.lock().unwrap().push(text.to_string()); + if self.defer.load(Ordering::SeqCst) { + return Ok(PokeOutcome::Deferred(DeferralReason::NoMaintainedComposer)); + } + let mut failures = self.failures.lock().unwrap(); + if *failures > 0 { + *failures -= 1; + anyhow::bail!("injected send failure"); + } + Ok(PokeOutcome::Delivered) + } + + fn session_alive(&self) -> bool { + self.probes.fetch_add(1, Ordering::SeqCst); + self.alive.load(Ordering::SeqCst) + } +} + +struct OwnershipPoker { + pokes: Mutex>, + retries: Mutex>, + poke_outcomes: Mutex>, + retry_outcomes: Mutex>, +} + +impl Poker for OwnershipPoker { + fn poke(&self, text: &str) -> anyhow::Result { + self.pokes.lock().unwrap().push(text.to_string()); + Ok(self + .poke_outcomes + .lock() + .unwrap() + .pop_front() + .expect("unexpected fresh poke")) + } + + fn retry_staged(&self, text: &str) -> anyhow::Result { + self.retries.lock().unwrap().push(text.to_string()); + Ok(self + .retry_outcomes + .lock() + .unwrap() + .pop_front() + .expect("unexpected staged retry")) + } + + fn session_alive(&self) -> bool { + true + } +} + +#[test] +fn poke_id_extracts_rand6() { + assert_eq!(poke_id("1785070000000-abc123.md"), "abc123"); + assert_eq!(poke_id("notes.md"), "notes"); +} + +#[test] +fn ancestor_depth_is_encoded_in_the_marker_run() { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "root", None); + declare_agent(catalog.path(), "h", "middle", Some("h.root")); + declare_agent(catalog.path(), "h", "recipient", Some("middle")); + + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-abc123.md", "h.middle", Some("direct")) + ), + "[DING] ↓ h.middle: direct [id:abc123]" + ); + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-def456.md", "h.root", Some("skip-level")) + ), + "[DING] ↓↓ h.root: skip-level [id:def456]" + ); +} + +#[test] +fn relationship_markers_cover_descendant_peer_and_missing_sender_spec() { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "root", None); + declare_agent(catalog.path(), "h", "recipient", Some("root")); + declare_agent(catalog.path(), "h", "child", Some("recipient")); + declare_agent(catalog.path(), "h", "peer", Some("root")); + + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-abc123.md", "h.child", Some("report")) + ), + "[DING] ↑ h.child: report [id:abc123]" + ); + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-def456.md", "h.peer", Some("note")) + ), + "[DING] ← h.peer: note [id:def456]" + ); + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-ghi789.md", "nightly-timer", Some("check")) + ), + "[DING] ? nightly-timer: check [id:ghi789]" + ); +} + +#[test] +fn dangling_catalog_entry_makes_relationship_unknown() { + use std::os::unix::fs::symlink; + + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "root", None); + declare_agent(catalog.path(), "h", "recipient", Some("root")); + symlink( + catalog.path().join("missing.kdl"), + catalog.path().join("dangling.kdl"), + ) + .unwrap(); + + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-abc123.md", "h.root", Some("dangling")) + ), + "[DING] ? h.root: dangling [id:abc123]" + ); +} + +#[test] +fn self_addressed_message_uses_identity_marker() { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "recipient", None); + + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.recipient", + &msg("1785070000000-abc123.md", "h.recipient", Some("self")) + ), + "[DING] ↺ h.recipient: self [id:abc123]" + ); +} + +#[test] +fn supervisor_cycle_renders_unknown_and_still_delivers() { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "recipient", Some("loop")); + declare_agent(catalog.path(), "h", "loop", Some("recipient")); + let message = msg("1785070000000-abc123.md", "h.loop", Some("cycle")); + let expected = "[DING] ? h.loop: cycle [id:abc123]"; + let resolver = RelationshipResolver::read(catalog.path()); + let recipient = resolve_spec(&resolver.specs, "h.recipient", "h").unwrap(); + + assert_eq!( + chain_bus_ids(&resolver.specs, recipient, "h"), + Err(crate::supervisor_chain::SupervisorChainError::Cycle), + "cycle detection must be distinct from the independent depth limit" + ); + + assert_eq!( + poke_text(catalog.path(), "h", "h.recipient", &message), + expected + ); + + let mut pending = VecDeque::from([PendingNotice::message(message)]); + let poker = RecordingPoker::live(); + flush_pending( + DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }, + None, + &mut pending, + &poker, + ); + + assert!(pending.is_empty()); + assert_eq!(poker.calls.lock().unwrap().as_slice(), [expected]); +} + +#[test] +fn malformed_catalog_with_resolvable_endpoints_renders_unknown_and_still_delivers() { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "root", None); + declare_agent(catalog.path(), "h", "recipient", Some("root")); + let malformed = catalog.path().join("h/broken/agent.kdl"); + std::fs::create_dir_all(malformed.parent().unwrap()).unwrap(); + std::fs::write(&malformed, "agent this is not valid KDL {").unwrap(); + let message = msg("1785070000000-abc123.md", "h.sender", Some("fallback")); + let message = Message { + from: Some("h.root".to_string()), + ..message + }; + let expected = "[DING] ? h.root: fallback [id:abc123]"; + let mut pending = VecDeque::from([PendingNotice::message(message)]); + let poker = RecordingPoker::live(); + + flush_pending( + DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }, + None, + &mut pending, + &poker, + ); + + assert!(pending.is_empty()); + assert_eq!(poker.calls.lock().unwrap().as_slice(), [expected]); +} + +#[test] +fn supervisor_depth_limit_fails_soft() { + let catalog = tempfile::tempdir().unwrap(); + for depth in 0..=SUPERVISOR_CHAIN_LIMIT { + let identity = format!("agent-{depth}"); + let supervisor = + (depth < SUPERVISOR_CHAIN_LIMIT).then(|| format!("agent-{}", depth + 1)); + declare_agent(catalog.path(), "h", &identity, supervisor.as_deref()); + } + + assert_eq!( + poke_text( + catalog.path(), + "h", + "h.agent-0", + &msg("1785070000000-abc123.md", "h.agent-64", Some("too deep")) + ), + "[DING] ? h.agent-64: too deep [id:abc123]" + ); +} + +#[test] +fn poke_text_normalizes_and_bounds_untrusted_fields() { + assert_eq!( + render_without_catalog(&msg("1785070000000-abc123.md", "alice", Some("deploy?"))), + "[DING] ? alice: deploy? [id:abc123]" + ); + assert_eq!( + render_without_catalog(&Message { + from: None, + subject: None, + ..msg("1785070000000-def456.md", "", None) + }), + "[DING] ? unknown: (no subject) [id:def456]" + ); + + let subject = format!("{}\nignored", "s".repeat(SUBJECT_MAX_CHARS + 20)); + let sender = format!("{}\tignored", "f".repeat(SENDER_MAX_CHARS + 20)); + let text = render_without_catalog(&msg("1785070000000-ghi789.md", &sender, Some(&subject))); + assert!(text.contains(&"s".repeat(SUBJECT_MAX_CHARS))); + assert!(!text.contains(&"s".repeat(SUBJECT_MAX_CHARS + 1))); + assert!(text.contains(&"f".repeat(SENDER_MAX_CHARS))); + assert!(!text.contains(&"f".repeat(SENDER_MAX_CHARS + 1))); + assert!(text.ends_with("[id:ghi789]")); + assert!(text.contains("[id:ghi789]")); +} + +#[test] +fn malicious_controls_cannot_escape_the_single_paste_frame() { + let message = msg( + "1785070000000-k0ygwh.md", + "attacker\x1b[201~\r\u{009b}2J", + Some("line one\n\tline two\x1b[201~key:return"), + ); + let text = render_without_catalog(&message); + assert!(!text.chars().any(char::is_control)); + assert!(!text.contains(" ")); + assert!(text.contains("[id:k0ygwh]")); + assert!(text.ends_with("[id:k0ygwh]")); + + let direct = format!("{text}\x1b[201~\nsecond line"); + let args = pty_stage_args("seat", &direct); + let framed = &args[3]; + assert_eq!(framed.matches(BRACKETED_PASTE_START).count(), 1); + assert_eq!(framed.matches(BRACKETED_PASTE_END).count(), 1); + let inner = framed + .strip_prefix(BRACKETED_PASTE_START) + .unwrap() + .strip_suffix(BRACKETED_PASTE_END) + .unwrap(); + assert!(!inner.chars().any(char::is_control)); + assert!(inner.ends_with("[201~ second line")); +} + +#[test] +fn pty_stage_and_submit_are_separate_exact_sequences() { + assert_eq!( + pty_stage_args("my-session", "hello\nworld"), + vec![ + "send", + "my-session", + "--seq", + "\x1b[200~hello world\x1b[201~", + ] + ); + assert_eq!( + pty_submit_args("my-session"), + vec!["send", "my-session", "--seq", "key:return"] + ); + assert!(!pty_stage_args("my-session", "hello").contains(&"key:return".to_string())); +} + +#[test] +fn pty_delivery_uses_face607_delay_order_and_seconds() { + assert_eq!( + pty_delivery_args("s", "hello"), + vec![ + "send", + "s", + "--with-delay", + "0.5", + "--seq", + "\x1b[200~hello\x1b[201~", + "--seq", + "key:return" + ] + ); +} + +fn idle_codex_screen_with_footer(footer: &str) -> String { + format!( + "\x1b[1m›\x1b[1C\x1b[22;2mFind and fix a bug in @filename\r\n\r\n\ + \x1b[2C\x1b[0m{footer}" + ) +} + +fn idle_codex_screen() -> String { + idle_codex_screen_with_footer("gpt-5.6-sol xhigh · /workspace") +} + +fn idle_codex_screen_with_home_relative_cwd() -> String { + idle_codex_screen().replace(" · /workspace", " · ~/Code/st2") +} + +fn staged_codex_screen_with_footer(text: &str, footer: &str) -> String { + let rendered = text.replace(' ', "\x1b[1C"); + format!( + "\x1b[1m›\x1b[1C\x1b[0m{rendered}\r\n\r\n\ + \x1b[2C\x1b[0m{footer}" + ) +} + +fn staged_codex_screen(text: &str) -> String { + staged_codex_screen_with_footer(text, "gpt-5.6-sol xhigh · /workspace") +} + +fn staged_wrapped_codex_screen() -> (&'static str, String) { + let text = "[DING] ↓ supervisor: a deliberately long synthetic notification with enough content to reach another renderer boundary before the final words [id:abc123]"; + let composer = concat!( + "[DING] ↓ supervisor: a deliberately long synthetic notification with enough", + "\x1b[3X\r\n", + " content to reach another renderer boundary before the final words [id:abc123]", + ); + ( + text, + format!( + "\x1b[1m›\x1b[22m {composer}\r\n\r\n\x1b[2C\x1b[0mgpt-5.6-sol low · Context 14% used" + ), + ) +} + +fn human_codex_screen() -> String { + staged_codex_screen("please keep my half-written draft") +} + +fn accepted_codex_screen(text: &str) -> String { + format!( + "{}\r\n\r\n{}", + staged_codex_screen(text), + idle_codex_screen() + ) +} + +fn queued_codex_screen(text: &str) -> String { + format!( + "Messages to be submitted after next tool call:\r\n{text}\r\n\r\n{}", + idle_codex_screen() + ) +} + +fn claude_rule() -> String { + "─".repeat(80) +} + +fn idle_claude_screen() -> String { + let rule = claude_rule(); + format!( + "Claude Code v2.1.220\r\n{rule}\r\n❯\u{00a0}Try \"write a test for validate.rs\"\r\n\ + {rule}\r\n ⏵⏵ bypass permissions on (shift+tab to cycle) · ← for agents" + ) +} + +fn idle_claude_screen_without_hint() -> String { + idle_claude_screen().replace(" (shift+tab to cycle)", "") +} + +fn staged_claude_screen(text: &str) -> String { + assert!(text.is_ascii()); + let rule = claude_rule(); + let composer = if text.len() <= 77 { + format!("❯\u{00a0}{text}") + } else { + let (first, continuation) = text.split_at(77); + format!("❯\u{00a0}{first}\r\n {continuation}") + }; + format!( + "Claude Code v2.1.220\r\n{rule}\r\n{composer}\r\n{rule}\r\n\ + ⏵⏵ bypass permissions on (shift+tab to cycle)" + ) +} + +/// A pane that has been used: the startup banner has scrolled out of the peeked viewport, the +/// composer is empty rather than showing a rotating placeholder, and the footer omits the +/// conditional `(shift+tab to cycle)` hint while keeping the permission-mode indicator. +fn mature_claude_screen(composer: &str) -> String { + let rule = claude_rule(); + let footer = " ⏵⏵ bypass permissions on · PR #42 · 2 shells · ← 1 agent"; + format!(" an earlier turn\r\n{rule}\r\n{composer}\r\n{rule}\r\n{footer}") +} + +fn mature_idle_claude_screen() -> String { + mature_claude_screen("❯\u{00a0}") +} + +fn mature_idle_claude_screen_with_hint() -> String { + mature_idle_claude_screen().replace( + "⏵⏵ bypass permissions on", + "⏵⏵ bypass permissions on (shift+tab to cycle)", + ) +} + +/// The same used pane in accept-edits mode. `permissions on` is specific to the bypass footer, +/// so no accept-edits or auto pane is positively idle to this classifier. +fn mature_idle_accept_edits_claude_screen() -> String { + mature_idle_claude_screen().replace("⏵⏵ bypass permissions on", "⏵⏵ accept edits on") +} + +fn mature_staged_claude_screen(text: &str) -> String { + mature_claude_screen(&format!("❯\u{00a0}{text}")) +} + +/// Selection region captured with `pty peek --plain st2-codex-selection-fixture` from a +/// dedicated 80x24 Codex CLI 0.145.0 pane on 2026-08-04. Codex rendered this built-in trust +/// selection after `codex --no-alt-screen -a untrusted -s read-only` launched in a fresh +/// untrusted directory. The command used no bypass flag. The leading working-directory row is +/// intentionally omitted; every retained row is verbatim from the captured selection. +const CAPTURED_CODEX_TRUST_SELECTION: &str = " + Do you trust the contents of this directory? Working with untrusted contents + comes with higher risk of prompt injection. Trusting the directory allows + project-local config, hooks, and exec policies to load. + +› 1. Yes, continue + 2. No, quit + + Press enter to continue"; + +/// Exact public-safe suffix from default raw `pty peek` while Codex CLI 0.145.0's `/model` +/// picker was open. It begins at the picker heading and preserves every emitted byte through +/// the end of stdout. +const CAPTURED_CODEX_MODEL_PICKER: &str = + include_str!("fixtures/codex-model-picker-default-to-end.bin"); + +const CODEX_LATENCY_RETRY_NOTICE: &str = + "Our systems are thinking a bit more\r\n\r\nRetry with a faster model"; + +/// Form region captured with `pty peek --plain st2-claude-form-fixture-clean` from a dedicated +/// 80x24 Claude Code 2.1.220 pane in safe/manual mode on 2026-08-03. Claude rendered this after +/// an `AskUserQuestion` call. The host-local banner and working directory are intentionally +/// omitted; every retained row is verbatim from the captured form. +const CAPTURED_CLAUDE_QUESTION_FORM: &str = " +──────────────────────────────────────────────────────────────────────────────── +☐ Fixture + +Which fixture should this test use? + +❯ 1. Captured form + Use a captured form fixture + 2. Plain composer + Use a plain composer fixture\u{20} + 3. Type something. +──────────────────────────────────────────────────────────────────────────────── + 4. Chat about this + +Enter to select · ↑/↓ to navigate · Esc to cancel"; + +fn captured_claude_question_form_with_selection(option: usize) -> String { + assert!((1..=4).contains(&option)); + if option == 1 { + return CAPTURED_CLAUDE_QUESTION_FORM.to_string(); + } + CAPTURED_CLAUDE_QUESTION_FORM + .replacen("❯ 1.", " 1.", 1) + .replacen(&format!(" {option}."), &format!("❯ {option}."), 1) +} + +/// The captured form reduced only by deleting its third and fourth option rows. This pins the +/// real two-option shape without adding any screen layout that was not present in the capture. +fn captured_two_option_claude_question_form() -> String { + CAPTURED_CLAUDE_QUESTION_FORM.replace( + &format!( + " 3. Type something.\n{}\n 4. Chat about this\n\n", + claude_rule() + ), + "", + ) +} + +/// The same used pane with an in-flight turn: a spinner status line above the composer. Every +/// frame below was observed on a real 2.1.220 pane; the glyph animates and the elapsed timer is +/// not always rendered, so both variations appear here. +fn mid_turn_claude_screen(status: &str, composer: &str) -> String { + let rule = claude_rule(); + let footer = " ⏵⏵ bypass permissions on · PR #42 · 2 shells · ← 1 agent"; + format!(" an earlier turn\r\n{status}\r\n{rule}\r\n{composer}\r\n{rule}\r\n{footer}") +} + +/// Status lines for an ACTIVE turn — Return must never be sent. +const ACTIVE_TURN_STATUS: [&str; 4] = [ + "✻ Frolicking… (3m 35s · ↓ 6.9k tokens)", + "✽ Schlepping…", + "· Metamorphosing…", + "✶ Schlepping… (9s · ↓ 296 tokens · thinking with high effort)", +]; + +/// Status lines for a FINISHED turn — these sit above every genuinely idle composer, so +/// treating them as blocked would stop delivery entirely. +const FINISHED_TURN_STATUS: [&str; 4] = [ + "✻ Brewed for 5s", + "✻ Crunched for 7s", + "✻ Cogitated for 11s · 1 shell still running", + "✻ Baked for 3s · 1 shell still running", +]; + +/// A live Claude composer with a stale Codex composer above it in scrollback, preceded by +/// escape-heavy output. The escapes inflate the Codex byte offset far past the Claude +/// composer's row, which is what makes the two locators' units observably disagree. +fn live_claude_below_escape_heavy_codex_transcript() -> String { + let padding = + "\x1b[1;32m\x1b[38;5;204mpadding with lots of escapes\x1b[0m\x1b[0m\r\n".repeat(10); + let codex = staged_codex_screen("a stale pasted codex draft"); + let filler = "\x1b[1;32mmore padding\x1b[0m\r\n".repeat(6); + format!( + "{padding}{codex}\r\n{filler}{}", + mature_idle_claude_screen() + ) +} + +/// A Codex pane whose scrollback holds a captured Claude screen — two ruled lines around a `❯` +/// row plus a Claude idle footer — above the live, ANSI-detected Codex composer. Capturing and +/// pasting pane text is routine, so this shape is not exotic. +fn codex_screen_below_claude_transcript(transcript_row: &str, codex: &str) -> String { + let rule = claude_rule(); + format!( + " scrollback: a pasted Claude pane\r\n{rule}\r\n❯\u{00a0}{transcript_row}\r\n{rule}\r\n\ + \u{0020} ⏵⏵ bypass permissions on (shift+tab to cycle)\r\n\r\n{codex}" + ) +} + +/// An in-flight Claude turn ships no interrupt hint, so the composer being empty is not proof +/// that Return is safe — the pane is working. The finished-turn line looks almost identical and +/// sits above every genuinely idle composer, so both directions are pinned here. +#[test] +fn an_in_flight_turn_blocks_return_but_a_finished_one_does_not() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + + for status in ACTIVE_TURN_STATUS { + // Empty composer mid-turn: positively empty, but not safe. + assert_ne!( + classify_composer(&mid_turn_claude_screen(status, "❯\u{00a0}"), expected), + ComposerState::EmptySafe, + "active turn must not be EmptySafe: {status}" + ); + // The notice already staged mid-turn: exact, but Return is still withheld. + assert_eq!( + classify_composer( + &mid_turn_claude_screen(status, &format!("❯\u{00a0}{expected}")), + expected + ), + ComposerState::ExactBlocked, + "active turn must be ExactBlocked: {status}" + ); + } + + // A finished turn is the normal idle screen. Blocking on it would stop delivery forever. + for status in FINISHED_TURN_STATUS { + assert_eq!( + classify_composer(&mid_turn_claude_screen(status, "❯\u{00a0}"), expected), + ComposerState::EmptySafe, + "finished turn must stay deliverable: {status}" + ); + assert_eq!( + classify_composer( + &mid_turn_claude_screen(status, &format!("❯\u{00a0}{expected}")), + expected + ), + ComposerState::ExactSafe, + "finished turn must stay submittable: {status}" + ); + } +} + +#[test] +fn claude_question_form_blocks_while_an_ordinary_idle_composer_stays_deliverable() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + + assert!(composer::looks_like_choice_menu( + CAPTURED_CLAUDE_QUESTION_FORM + )); + assert_eq!( + classify_composer( + &format!( + "{CAPTURED_CLAUDE_QUESTION_FORM}\r\n{}", + mature_staged_claude_screen(expected) + ), + expected + ), + ComposerState::ExactBlocked + ); + + assert_eq!( + classify_composer(&mature_idle_claude_screen(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&mature_staged_claude_screen(expected), expected), + ComposerState::ExactSafe + ); +} + +#[test] +fn codex_trust_selection_blocks_while_an_ordinary_idle_composer_stays_deliverable() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + + assert!(composer::looks_like_choice_menu( + CAPTURED_CODEX_TRUST_SELECTION + )); + assert_eq!( + classify_composer(CAPTURED_CODEX_TRUST_SELECTION, expected), + ComposerState::Ambiguous + ); + assert_eq!( + classify_composer( + &format!( + "{CAPTURED_CODEX_TRUST_SELECTION}\r\n{}", + staged_codex_screen(expected) + ), + expected + ), + ComposerState::ExactBlocked + ); + + assert_eq!( + classify_composer(&idle_codex_screen(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&staged_codex_screen(expected), expected), + ComposerState::ExactSafe + ); +} + +#[test] +fn captured_codex_model_picker_is_recognized_as_a_choice_menu() { + let plain = composer::strip_ansi(CAPTURED_CODEX_MODEL_PICKER); + + assert!(composer::looks_like_choice_menu(&plain)); + assert_eq!( + classify_composer(CAPTURED_CODEX_MODEL_PICKER, "unused notice"), + ComposerState::Ambiguous + ); + + let without_selected_option = plain.replacen("› 1.", " 1.", 1); + assert!(!composer::looks_like_choice_menu(&without_selected_option)); +} + +#[test] +fn fresh_delivery_does_not_return_after_idle_changes_to_model_picker() { + use std::os::unix::fs::PermissionsExt as _; + + let text = "[DING] unread st2 messages remain; check your inbox"; + let temp = tempfile::tempdir().unwrap(); + let bin = temp.path().join("pty"); + let idle = temp.path().join("idle.bin"); + let picker = temp.path().join("picker.bin"); + let changed = temp.path().join("changed"); + let audit = temp.path().join("audit.log"); + std::fs::write(&idle, idle_codex_screen()).unwrap(); + std::fs::write(&picker, CAPTURED_CODEX_MODEL_PICKER).unwrap(); + std::fs::write( + &bin, + format!( + "#!/bin/sh\nprintf '%s\\n' \"$*\" >> '{}'\n\ + if [ \"$1\" = peek ]; then\n\ + if [ -e '{}' ]; then cat '{}'; else cat '{}'; fi\n\ + exit 0\n\ + fi\n\ + if [ \"$1\" = send ]; then : > '{}'; exit 0; fi\n\ + exit 1\n", + audit.display(), + changed.display(), + picker.display(), + idle.display(), + changed.display(), + ), + ) + .unwrap(); + std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let poker = PtyPoker { + bin: bin.to_string_lossy().into_owned(), + session: "fixture".to_string(), + }; + assert_eq!( + poker.poke_with(text, &mut || Ok(())).unwrap(), + PokeOutcome::Staged + ); + + let audit = std::fs::read_to_string(audit).unwrap(); + assert_eq!(audit.lines().next(), Some("peek fixture")); + assert!( + !audit.contains("key:return"), + "Return must be withheld after the staged paste exposes a picker: {audit:?}" + ); +} + +#[test] +fn codex_latency_retry_notice_blocks_without_choice_menu_structure() { + let expected = "[DING] ? cos: latency control [id:abc123]"; + + assert!(!composer::looks_like_choice_menu( + CODEX_LATENCY_RETRY_NOTICE + )); + assert_eq!( + classify_composer( + &format!( + "{CODEX_LATENCY_RETRY_NOTICE}\r\n{}", + staged_codex_screen(expected) + ), + expected, + ), + ComposerState::ExactBlocked + ); +} + +#[test] +fn claude_question_form_blocks_after_selection_moves_to_second_or_last_option() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + + for option in [2, 4] { + let form = captured_claude_question_form_with_selection(option); + assert!( + composer::looks_like_choice_menu(&form), + "selection on option {option} must remain modal" + ); + assert_eq!( + classify_composer( + &format!("{form}\r\n{}", mature_staged_claude_screen(expected)), + expected + ), + ComposerState::ExactBlocked, + "selection on option {option} must block Return" + ); + } +} + +#[test] +fn two_option_claude_question_form_blocks_return() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + let form = captured_two_option_claude_question_form(); + + assert!(!form.contains(" 3.")); + assert!(!form.contains(" 4.")); + assert!(composer::looks_like_choice_menu(&form)); + assert_eq!( + classify_composer( + &format!("{form}\r\n{}", mature_staged_claude_screen(expected)), + expected + ), + ComposerState::ExactBlocked + ); +} + +#[test] +fn legacy_numbered_choice_menu_still_blocks_return() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + let legacy_menu = "› 1. Continue\r\n 2. Cancel"; + + assert!(composer::looks_like_choice_menu(legacy_menu)); + assert_eq!( + classify_composer( + &format!("{legacy_menu}\r\n{}", mature_staged_claude_screen(expected)), + expected + ), + ComposerState::ExactBlocked + ); +} + +/// The two locators do not natively work in the same units. Codex is matched with `rfind` over +/// the raw screen, so it reports a **byte offset** inflated by every escape sequence above it; +/// Claude is matched over stripped lines, so it reports a **row**. Comparing those directly +/// picks Codex almost always, since an offset dwarfs a row — including when the live composer +/// is Claude's and the Codex match is stale scrollback. Both must be normalized to a row. +/// +/// On the screen below, measured: the Codex composer sits at byte offset 560 but row 10, while +/// the live Claude composer is row 20. Comparing row against offset picks Codex, so this would +/// classify from a pasted draft instead of the real composer. +#[test] +fn composer_positions_are_compared_as_rows_not_raw_byte_offsets() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + assert_eq!( + classify_composer(&live_claude_below_escape_heavy_codex_transcript(), expected), + ComposerState::EmptySafe + ); +} + +/// Normalizing the Codex offset means counting newlines in the *stripped* prefix, which is only +/// faithful if stripping preserves them. It does for well-formed input. It does not for an +/// unterminated sequence: the CSI scanner runs until a byte in `0x40..=0x7e` and `\n` is `0x0a`, +/// so it eats newlines, and an unterminated OSC consumes to the end of input. Both are recorded +/// here so a future change to `strip_ansi` cannot silently shift every row. +#[test] +fn stripping_preserves_newlines_for_well_formed_sequences_only() { + let nl = |text: &str| composer::strip_ansi(text).matches('\n').count(); + + assert_eq!( + nl("\x1b[1;32mone\x1b[0m\r\n\x1b[2Ctwo\x1b[0m\r\n\x1b[1mthree\x1b[0m\r\n"), + 3 + ); + // Unterminated CSI: the newline is consumed while hunting for a final byte. + assert_eq!(nl("before\r\n\x1b[999999\r\nafter\r\n"), 2); + // Unterminated OSC: everything to the end of input is consumed. + assert_eq!(nl("before\r\n\x1b]0;no terminator\r\nafter\r\n"), 1); +} + +/// Scrollback that merely looks like a composer must never outrank the live one. The paste and +/// the Return always go to the pane's real bottom composer, so misreading transcript text as +/// "idle" or "already staged" is a wrong positive: it can type into, or submit, a human draft. +#[test] +fn transcript_composers_never_outrank_the_live_bottom_composer() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + + // The live Codex composer holds a human draft in both cases, so both must stay `Changed`. + // An empty transcript row would otherwise read as positively-empty and allow the paste. + assert_eq!( + classify_composer( + &codex_screen_below_claude_transcript("", &human_codex_screen()), + expected + ), + ComposerState::Changed + ); + // A transcript row holding the exact notice is the worse case: it would otherwise satisfy + // the two adjacent exact observations and send a bare Return to the draft. + assert_eq!( + classify_composer( + &codex_screen_below_claude_transcript(expected, &human_codex_screen()), + expected + ), + ComposerState::Changed + ); + + // The rule is positional, not a Codex preference: a genuine Claude pane whose scrollback + // shows a captured Codex composer still classifies from its own live Claude composer. + assert_eq!( + classify_composer( + &format!( + "{}\r\n{}", + staged_codex_screen("a stale pasted codex draft"), + mature_idle_claude_screen() + ), + expected + ), + ComposerState::EmptySafe + ); +} + +#[test] +fn maintained_composer_classifiers_require_exact_idle_state() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + assert_eq!( + classify_composer(&idle_codex_screen(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&idle_codex_screen_with_home_relative_cwd(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&staged_codex_screen(expected), expected), + ComposerState::ExactSafe + ); + assert_eq!( + classify_composer(&human_codex_screen(), expected), + ComposerState::Changed + ); + assert_eq!( + classify_composer( + &format!("Create a plan?\r\n{}", staged_codex_screen(expected)), + expected + ), + ComposerState::ExactBlocked + ); + + assert_eq!( + classify_composer(&idle_claude_screen(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&idle_claude_screen_without_hint(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&staged_claude_screen(expected), expected), + ComposerState::ExactSafe + ); + assert_eq!( + classify_composer(&staged_claude_screen("a changed human composer"), expected), + ComposerState::Changed + ); + assert_eq!( + classify_composer( + &format!("Esc to interrupt\r\n{}", staged_claude_screen(expected)), + expected + ), + ComposerState::ExactBlocked + ); + // A used pane must classify exactly like a fresh one: neither the scrolled-away banner, the + // empty composer, nor the missing cycle hint is evidence that Return is unsafe. + assert_eq!( + classify_composer(&mature_idle_claude_screen(), expected), + ComposerState::EmptySafe + ); + assert_eq!( + classify_composer(&mature_idle_claude_screen_with_hint(), expected), + ComposerState::EmptySafe + ); + // Only the bypass footer carries `permissions on`, so an otherwise identical accept-edits + // pane is never positively idle. It stays unsubmitted rather than being proven safe. + assert_eq!( + classify_composer(&mature_idle_accept_edits_claude_screen(), expected), + ComposerState::Changed + ); + assert_eq!( + classify_composer(&mature_staged_claude_screen(expected), expected), + ComposerState::ExactSafe + ); + // The same pane shape must still fail closed on a human draft and on an active turn. + assert_eq!( + classify_composer( + &mature_staged_claude_screen("a changed human composer"), + expected + ), + ComposerState::Changed + ); + assert_eq!( + classify_composer( + &format!( + "Esc to interrupt\r\n{}", + mature_staged_claude_screen(expected) + ), + expected + ), + ComposerState::ExactBlocked + ); + + assert_eq!( + classify_composer("unknown terminal pixels", expected), + ComposerState::Ambiguous + ); +} + +#[test] +fn maintained_codex_context_footers_are_narrow_and_position_bound() { + let expected = "[DING] ? cos: exact observation [id:abc123]"; + + for footer in [ + "gpt-5.6-sol xhigh · ding-fix · Context 73% left", + "gpt-5.6-sol xhigh · Context 0% used", + "gpt-5.6-sol xhigh · Context 100% left", + ] { + assert_eq!( + classify_composer(&idle_codex_screen_with_footer(footer), expected), + ComposerState::EmptySafe, + "maintained empty composer footer: {footer}" + ); + assert_eq!( + classify_composer(&staged_codex_screen_with_footer(expected, footer), expected), + ComposerState::ExactSafe, + "maintained staged composer footer: {footer}" + ); + } + + for footer in [ + "gpt-5.6-sol xhigh · ding-fix", + "gpt-5.6-sol xhigh · Contextual 73% left", + "gpt-5.6-sol xhigh · Context 73 left", + "gpt-5.6-sol xhigh · Context 101% left", + "gpt-5.6-sol xhigh · Context left 73%", + "gpt-5.6-sol xhigh · Context 73% remaining", + ] { + assert_eq!( + classify_composer(&idle_codex_screen_with_footer(footer), expected), + ComposerState::Ambiguous, + "unsupported empty composer footer: {footer}" + ); + assert_eq!( + classify_composer(&staged_codex_screen_with_footer(expected, footer), expected), + ComposerState::ExactBlocked, + "unsupported staged composer footer: {footer}" + ); + } + + let maintained_footer = "gpt-5.6-sol xhigh · ding-fix · Context 73% left"; + let branch_only_footer = "gpt-5.6-sol xhigh · ding-fix"; + assert_eq!( + classify_composer( + &format!( + "{maintained_footer}\r\n{}", + staged_codex_screen_with_footer(expected, branch_only_footer) + ), + expected + ), + ComposerState::ExactBlocked, + "a valid-looking transcript row above the live composer is not its footer" + ); + assert_eq!( + classify_composer( + &staged_codex_screen_with_footer( + "please keep my half-written draft", + maintained_footer + ), + expected + ), + ComposerState::Changed + ); + for footer in [ + format!("{maintained_footer}\r\n{maintained_footer}"), + format!("{maintained_footer}\r\n{branch_only_footer}"), + format!("{maintained_footer}\r\nunknown trailing chrome"), + ] { + assert_eq!( + classify_composer(&idle_codex_screen_with_footer(&footer), expected), + ComposerState::Ambiguous, + "duplicated or trailing footer chrome must not prove an empty composer idle: {footer}" + ); + assert_eq!( + classify_composer( + &staged_codex_screen_with_footer(expected, &footer), + expected + ), + ComposerState::ExactBlocked, + "duplicated or trailing footer chrome must not prove Return safe: {footer}" + ); + } + for blocking_chrome in [ + "Esc to interrupt", + "Create a plan?", + "› 1. Allow\r\n 2. Deny", + ] { + assert_eq!( + classify_composer( + &format!( + "{blocking_chrome}\r\n{}", + staged_codex_screen_with_footer(expected, maintained_footer) + ), + expected + ), + ComposerState::ExactBlocked, + "blocking chrome: {blocking_chrome}" + ); + } +} + +#[test] +fn codex_renderer_wraps_preserve_possible_inter_word_spaces() { + let (expected, screen) = staged_wrapped_codex_screen(); + let token_split = expected.replace("enough content", "enoughcontent"); + let changed = expected.replace("synthetic", "different"); + + assert_eq!( + classify_composer(&screen, expected), + ComposerState::ExactSafe + ); + assert_eq!( + classify_receipt(&screen, expected), + ReceiptState::RetainedSafe + ); + assert_eq!( + classify_composer(&screen, &token_split), + ComposerState::ExactSafe + ); + assert_eq!(classify_composer(&screen, &changed), ComposerState::Changed); + assert_eq!( + classify_receipt(&screen, &changed), + ReceiptState::NotRetained + ); +} + +#[test] +fn codex_word_wraps_preserve_short_rows_before_continuations() { + let expected = "[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source consistency complete (from dev3.compoundingtech.st2.message-sent.orchestration); check your inbox"; + let screen = "\x1b[1m›\x1b[1C\x1b[0m[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source\r\n consistency complete (from dev3.compoundingtech.st2.message-\r\n sent.orchestration); check your inbox\r\n\r\n\x1b[2C\x1b[0mgpt-5.6-sol xhigh · /workspace"; + + assert_eq!( + classify_composer(screen, expected), + ComposerState::ExactSafe + ); + assert_eq!( + classify_receipt(screen, expected), + ReceiptState::RetainedSafe + ); + + let human_draft = screen.replace("light-work receipt", "human draft"); + assert_eq!( + classify_composer(&human_draft, expected), + ComposerState::Changed + ); + assert_eq!( + classify_receipt(&human_draft, expected), + ReceiptState::NotRetained + ); + + let unfamiliar_shape = screen.replace("\r\n consistency", "\r\nconsistency"); + assert_eq!( + classify_composer(&unfamiliar_shape, expected), + ComposerState::Ambiguous + ); + assert_eq!( + classify_receipt(&unfamiliar_shape, expected), + ReceiptState::Unproven + ); +} + +#[test] +fn codex_hard_newline_shape_is_knowingly_admitted_until_issue_250() { + let expected = "[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source consistency complete (from dev3.compoundingtech.st2.message-sent.orchestration); check your inbox"; + // A buffer containing hard newlines plus two literal spaces renders identically to this + // soft wrap. The screen-only ambiguity is accepted until https://github.com/compoundingtech/st2/issues/250. + let screen = "\x1b[1m›\x1b[1C\x1b[0m[DING] new st2 message: [id:7j8b6n] re: Q97 light-work receipt: VRS/source\r\n consistency complete (from dev3.compoundingtech.st2.message-\r\n sent.orchestration); check your inbox\r\n\r\n\x1b[2C\x1b[0mgpt-5.6-sol xhigh · /workspace"; + + assert_eq!( + classify_composer(screen, expected), + ComposerState::ExactSafe + ); + assert_eq!( + classify_receipt(screen, expected), + ReceiptState::RetainedSafe + ); +} + +#[test] +fn claude_word_wraps_preserve_short_rows_before_continuations() { + let expected = "[DING] ? dev3.dotfiles.st2.claude-composer.worker: inspect the composer word wrapping behavior (from dev3.dotfiles.st2.main.orchestration); check your inbox [id:abc123]"; + let rule = claude_rule(); + let screen = format!( + "Claude Code v2.1.220\r\n{rule}\r\n\ + ❯\u{00a0}[DING] ? dev3.dotfiles.st2.claude-composer.worker: inspect the\r\n \ + composer word wrapping behavior (from\r\n \ + dev3.dotfiles.st2.main.orchestration); check your inbox\r\n \ + [id:abc123]\r\n{rule}\r\n\ + ⏵⏵ bypass permissions on (shift+tab to cycle)" + ); + + assert_eq!( + classify_composer(&screen, expected), + ComposerState::ExactSafe + ); + assert_eq!( + classify_receipt(&screen, expected), + ReceiptState::RetainedSafe + ); + + let human_draft = screen.replace("composer word", "human draft"); + assert_eq!( + classify_composer(&human_draft, expected), + ComposerState::Changed + ); + assert_eq!( + classify_receipt(&human_draft, expected), + ReceiptState::NotRetained + ); + + let unfamiliar_shape = screen.replace("\r\n composer", "\r\ncomposer"); + assert_eq!( + classify_composer(&unfamiliar_shape, expected), + ComposerState::Ambiguous + ); + assert_eq!( + classify_receipt(&unfamiliar_shape, expected), + ReceiptState::Unproven + ); +} + +#[test] +fn preserved_clean_codex_context_footer_is_idle() { + // Composer-to-end slice of clean zero-turn capture bb1eddba2725672287e4a73b0aaf50652250611566d9826977dc3c6639d70360. + let screen = "\x1b[1m›\x1b[22m \x1b[2mImprove documentation in @filename\r\n\x1b[22m \r\n\x1b[0m\x1b[2X\x1b[2C\x1b[38;2;246;226;183mgpt-5.6-sol low\x1b[39;2m · \x1b[38;2;242;181;144;22mContext 0% used\x1b[45X\x1b[2A\x1b[33D\x1b[0m\x1b[?2004h\x1b[?1004h\x1b[?1049l\x1b[?1l\x1b[?7h\x1b[?6l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1004l\x1b[?1006l\x1b[?25h\x1b[?2004l\x1b[4l\x1b[r\x1b[0m\x1b[0 q\x1b>\x1b(B\x1b[<99u\x1b[999;1H\n"; + + assert_eq!( + classify_composer(screen, "generic staged notice"), + ComposerState::EmptySafe + ); +} + +#[test] +fn preserved_post_turn_codex_frame_stays_ambiguous() { + // Composer-to-end slice of post-turn capture f4abd2a65c008db1f52d62fa53f813dedde5c830632133c80a732dfbe12b95fe. + let screen = "\x1b[48;2;30;30;30m \x1b[79X\r\n\x1b[1m›\x1b[22m \x1b[2mFind and fix a bug in @filename\x1b[47X\r\n\x1b[22m \x1b[79X\r\n\x1b[0m \x1b[38;2;246;226;183mgpt-5.6-sol low\x1b[39;2mi· \x1b[38;2;242;181;144;22mContext 3% used\x1b[2A\x1b[33D\x1b[0m\x1b[?2004h\x1b[?1004h\x1b[?1049l\x1b[?1l\x1b[?7h\x1b[?6l\x1b[?1000l\x1b[?1002l\x1b[?1003l\x1b[?1004l\x1b[?1006l\x1b[?25h\x1b[?2004l\x1b[4l\x1b[r\x1b[0m\x1b[0 q\x1b>\x1b(B\x1b[<99u\x1b[999;1H\n"; + + assert_eq!( + classify_composer(screen, "generic staged notice"), + ComposerState::Ambiguous + ); +} + +#[test] +fn strip_ansi_consumes_designate_g0_charset_sequence() { + assert_eq!( + composer::strip_ansi("Context 0% used\x1b>\x1b(B"), + "Context 0% used" + ); +} + +#[test] +fn startup_adopts_only_an_exact_recovery_or_backlog_composer() { + let recovery = RECOVERY_POKE.to_string(); + let backlog = "[DING] ? cos: seeded [id:abc123]".to_string(); + let candidates = vec![recovery.clone(), backlog.clone()]; + assert_eq!( + exact_staged_candidate(&staged_codex_screen(&backlog), &candidates), + Some(backlog.clone()) + ); + assert_eq!( + exact_staged_candidate(&staged_claude_screen(&recovery), &candidates), + Some(recovery) + ); + assert_eq!( + exact_staged_candidate(&human_codex_screen(), &candidates), + None + ); +} + +#[test] +fn paste_then_two_exact_observations_precede_return() { + use std::cell::RefCell; + + let text = "[DING] ? cos: ordered [id:abc123]"; + let screens = RefCell::new(VecDeque::from([ + idle_codex_screen(), + staged_codex_screen(text), + staged_codex_screen(text), + accepted_codex_screen(text), + ])); + let actions = RefCell::new(Vec::new()); + let outcome = observed_poke_with_window( + text, + &mut || { + actions.borrow_mut().push("peek"); + Ok(screens.borrow_mut().pop_front().unwrap()) + }, + &mut || { + actions.borrow_mut().push("paste"); + Ok(()) + }, + &mut || { + actions.borrow_mut().push("return"); + Ok(()) + }, + &mut || actions.borrow_mut().push("poll"), + &mut || { + actions.borrow_mut().push("receipt"); + Ok(()) + }, + Duration::from_millis(10), + ) + .unwrap(); + assert_eq!(outcome, PokeOutcome::Delivered); + assert_eq!( + *actions.borrow(), + ["peek", "paste", "peek", "peek", "receipt", "return", "peek"] + ); +} + +#[test] +fn changed_modal_ambiguous_and_bounded_timeout_never_return() { + use std::cell::RefCell; + + let text = "[DING] ? cos: guarded [id:abc123]"; + for screen in [ + human_codex_screen(), + format!("Create a plan?\r\n{}", staged_codex_screen(text)), + "unrecognized renderer".to_string(), + ] { + let actions = RefCell::new(Vec::new()); + let outcome = observed_poke_with_window( + text, + &mut || { + actions.borrow_mut().push("peek"); + Ok(screen.clone()) + }, + &mut || { + actions.borrow_mut().push("paste"); + Ok(()) + }, + &mut || { + actions.borrow_mut().push("return"); + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(); + assert_ne!(outcome, PokeOutcome::Delivered); + assert!(!actions.borrow().contains(&"return")); + assert!(!actions.borrow().contains(&"paste")); + } + + let screens = RefCell::new(VecDeque::from([idle_claude_screen(), idle_claude_screen()])); + let actions = RefCell::new(Vec::new()); + let outcome = observed_poke_with_window( + text, + &mut || { + actions.borrow_mut().push("peek"); + Ok(screens.borrow_mut().pop_front().unwrap()) + }, + &mut || { + actions.borrow_mut().push("paste"); + Ok(()) + }, + &mut || { + actions.borrow_mut().push("return"); + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(); + assert_eq!(outcome, PokeOutcome::Staged); + assert_eq!(*actions.borrow(), ["peek", "paste", "peek"]); +} + +#[test] +fn final_observation_change_and_staged_retry_are_fail_closed() { + use std::cell::RefCell; + + let text = "[DING] ? cos: final race [id:abc123]"; + let screens = RefCell::new(VecDeque::from([ + idle_codex_screen(), + staged_codex_screen(text), + human_codex_screen(), + ])); + let actions = RefCell::new(Vec::new()); + let outcome = observed_poke_with_window( + text, + &mut || Ok(screens.borrow_mut().pop_front().unwrap()), + &mut || { + actions.borrow_mut().push("paste"); + Ok(()) + }, + &mut || { + actions.borrow_mut().push("return"); + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::from_millis(10), + ) + .unwrap(); + assert_eq!(outcome, PokeOutcome::Staged); + assert_eq!(*actions.borrow(), ["paste"]); + + let retry_screens = RefCell::new(VecDeque::from([ + staged_claude_screen(text), + staged_claude_screen(text), + format!("❯\u{00a0}{text}\r\n{}", idle_claude_screen()), + ])); + let retry_actions = RefCell::new(Vec::new()); + assert_eq!( + retry_staged_with_window( + text, + &mut || Ok(retry_screens.borrow_mut().pop_front().unwrap()), + &mut || { + retry_actions.borrow_mut().push("return"); + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(), + PokeOutcome::Delivered + ); + assert_eq!(*retry_actions.borrow(), ["return"]); +} + +#[test] +fn successful_transport_with_retained_or_unproven_pixels_is_not_delivered() { + use std::cell::RefCell; + + let text = "[DING] ? cos: receipt truth [id:abc123]"; + for screen in [ + staged_codex_screen(text), + idle_codex_screen(), + human_codex_screen(), + "unknown renderer".to_string(), + ] { + let actions = RefCell::new(Vec::new()); + let outcome = transport_and_observe_with_window( + text, + &mut || { + actions.borrow_mut().push("transport"); + Ok(()) + }, + &mut || { + actions.borrow_mut().push("peek"); + Ok(screen.clone()) + }, + &mut || actions.borrow_mut().push("poll"), + &mut || { + actions.borrow_mut().push("before-submit"); + Ok(()) + }, + Duration::ZERO, + ) + .unwrap(); + assert_eq!(outcome, PokeOutcome::Staged); + assert_eq!( + *actions.borrow(), + ["before-submit", "transport", "peek"], + "transport success alone must never become Delivered" + ); + } +} + +#[test] +fn ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership() { + use std::cell::RefCell; + + let text = "[DING] ? cos: error truth [id:abc123]"; + + let actions = RefCell::new(Vec::new()); + assert_eq!( + transport_and_observe_with_window( + text, + &mut || { + actions.borrow_mut().push("transport"); + anyhow::bail!("ambiguous transport") + }, + &mut || { + actions.borrow_mut().push("peek"); + Ok(idle_codex_screen()) + }, + &mut || {}, + &mut || { + actions.borrow_mut().push("before-submit"); + Ok(()) + }, + Duration::ZERO, + ) + .unwrap(), + PokeOutcome::Staged + ); + assert_eq!(*actions.borrow(), ["before-submit", "transport"]); + + assert_eq!( + transport_and_observe_with_window( + text, + &mut || Ok(()), + &mut || anyhow::bail!("unreadable receipt"), + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(), + PokeOutcome::Staged + ); + + let screens = RefCell::new(VecDeque::from([ + staged_codex_screen(text), + staged_codex_screen(text), + ])); + let submits = RefCell::new(0); + assert_eq!( + retry_staged_with_window( + text, + &mut || Ok(screens.borrow_mut().pop_front().unwrap()), + &mut || { + *submits.borrow_mut() += 1; + anyhow::bail!("ambiguous Return") + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(), + PokeOutcome::Staged + ); + assert_eq!(*submits.borrow(), 1); +} + +#[test] +fn adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt() { + let text = "[DING] ? cos: receipt truth [id:abc123]"; + assert_eq!( + classify_receipt(&queued_codex_screen(text), text), + ReceiptState::Accepted + ); + assert_eq!( + classify_receipt(&accepted_codex_screen(text), text), + ReceiptState::Accepted + ); + assert_eq!( + classify_receipt(&staged_codex_screen(text), text), + ReceiptState::RetainedSafe + ); + assert_eq!( + classify_receipt( + &format!("ordinary transcript: {text}\r\n{}", idle_codex_screen()), + text + ), + ReceiptState::NotRetained, + "notice text outside an adapter-recognized accepted pattern is not a receipt" + ); + assert_eq!( + classify_receipt( + &format!("old receipt: {text}\r\n{}", human_codex_screen()), + text + ), + ReceiptState::NotRetained, + "a parsed changed live composer positively excludes the exact owned notice" + ); + assert_eq!( + classify_receipt(&idle_codex_screen(), text), + ReceiptState::NotRetained + ); + assert_eq!( + classify_receipt("unknown renderer", text), + ReceiptState::Unproven, + "an unrecognized screen cannot prove that the owned notice disappeared" + ); + + assert_eq!( + classify_receipt( + &format!("❯\u{00a0}{text}\r\n{}", idle_claude_screen()), + text + ), + ReceiptState::Accepted + ); + assert_eq!( + classify_receipt(&staged_claude_screen(text), text), + ReceiptState::RetainedSafe + ); + assert_eq!( + classify_receipt( + &format!("ordinary transcript: {text}\r\n{}", idle_claude_screen()), + text + ), + ReceiptState::NotRetained + ); + assert_eq!( + classify_receipt(&idle_claude_screen(), text), + ReceiptState::NotRetained + ); +} + +#[test] +fn soft_wrap_proofs_accept_short_known_continuations() { + let text = "[DING] ? cos: receipt truth [id:abc123]"; + let (first, continuation) = text.split_at(32); + let codex = format!( + "\x1b[1m›\x1b[1C\x1b[0m{first}\r\n {continuation}\r\n\r\n\ + \x1b[2C\x1b[0mgpt-5.6-sol xhigh · /workspace" + ); + let rule = claude_rule(); + let claude = format!( + "Claude Code v2.1.220\r\n{rule}\r\n❯\u{00a0}{first}\r\n {continuation}\r\n\ + {rule}\r\n⏵⏵ bypass permissions on (shift+tab to cycle)" + ); + + assert_eq!( + ( + classify_receipt(&codex, text), + classify_receipt(&claude, text), + ), + (ReceiptState::RetainedSafe, ReceiptState::RetainedSafe), + "both maintained composers prove short rows with their two-cell continuation indent" + ); + assert_eq!( + classify_receipt(&human_codex_screen(), text), + ReceiptState::NotRetained + ); + assert_eq!( + classify_receipt(&staged_claude_screen("a changed human composer"), text), + ReceiptState::NotRetained + ); +} + +#[test] +fn staged_retry_submits_only_retained_safe_and_requires_a_receipt() { + use std::cell::RefCell; + + let text = "[DING] ? cos: retry truth [id:abc123]"; + + let retained = RefCell::new(VecDeque::from([ + staged_codex_screen(text), + staged_codex_screen(text), + staged_codex_screen(text), + ])); + let submits = RefCell::new(0); + let outcome = retry_staged_with_window( + text, + &mut || Ok(retained.borrow_mut().pop_front().unwrap()), + &mut || { + *submits.borrow_mut() += 1; + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(); + assert_eq!(outcome, PokeOutcome::Staged); + assert_eq!(*submits.borrow(), 1); + + for (screen, expected) in [ + ( + format!("Create a plan?\r\n{}", staged_codex_screen(text)), + PokeOutcome::Staged, + ), + (idle_codex_screen(), PokeOutcome::NotRetained), + (human_codex_screen(), PokeOutcome::NotRetained), + ("unknown renderer".to_string(), PokeOutcome::Staged), + ] { + let submits = RefCell::new(0); + let outcome = retry_staged_with_window( + text, + &mut || Ok(screen.clone()), + &mut || { + *submits.borrow_mut() += 1; + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(); + assert_eq!(outcome, expected); + assert_eq!( + *submits.borrow(), + 0, + "blocked or unproven receipts must not receive Return" + ); + } + + let accepted = queued_codex_screen(text); + let submits = RefCell::new(0); + assert_eq!( + retry_staged_with_window( + text, + &mut || Ok(accepted.clone()), + &mut || { + *submits.borrow_mut() += 1; + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(), + PokeOutcome::Delivered + ); + assert_eq!(*submits.borrow(), 0); +} + +#[test] +fn staged_retry_keeps_unproven_and_retained_blocked_owned() { + use std::cell::RefCell; + + let text = "[DING] ? cos: retry truth [id:abc123]"; + for screen in [ + "unknown renderer".to_string(), + format!("Create a plan?\r\n{}", staged_codex_screen(text)), + ] { + let submits = RefCell::new(0); + assert_eq!( + retry_staged_with_window( + text, + &mut || Ok(screen.clone()), + &mut || { + *submits.borrow_mut() += 1; + Ok(()) + }, + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(), + PokeOutcome::Staged + ); + assert_eq!(*submits.borrow(), 0); + } +} + +#[test] +fn pty_commands_have_a_real_outer_timeout() { + let started = Instant::now(); + let error = output_with_timeout( + Command::new("sh").args(["-c", "sleep 2"]), + Duration::from_millis(30), + ) + .unwrap_err(); + assert!(error.to_string().contains("timed out")); + assert!(started.elapsed() < Duration::from_secs(1)); +} + +#[test] +fn session_watch_has_startup_grace_debounce_and_live_reset() { + let mut watch = SessionWatch::default(); + for _ in 0..10 { + assert_eq!(watch.step(false), WatchStep::Poll); + } + assert_eq!(watch.step(true), WatchStep::Poll); + assert_eq!(watch.step(false), WatchStep::Poll); + assert_eq!(watch.step(false), WatchStep::Poll); + assert_eq!(watch.step(true), WatchStep::Poll); + assert_eq!(watch.step(false), WatchStep::Poll); + assert_eq!(watch.step(false), WatchStep::Poll); + assert_eq!(watch.step(false), WatchStep::Gone); +} + +#[test] +fn new_arrivals_is_fifo_and_archive_receipts_prevent_reding() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + let archive = archive_dir(agent.path()); + let first = send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); + std::thread::sleep(Duration::from_millis(2)); + let second = send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); + + let mut seen = HashSet::new(); + let initial = new_arrivals(&inbox, &mut seen); + assert_eq!( + initial + .iter() + .map(|message| message.filename.as_str()) + .collect::>(), + [first.as_str(), second.as_str()] + ); + assert!(new_arrivals(&inbox, &mut seen).is_empty()); + + archive_msg(&inbox, &archive, &first).unwrap(); + std::fs::copy(archive.join(&first), inbox.join(&first)).unwrap(); + assert!( + new_arrivals(&inbox, &mut seen).is_empty(), + "an archive receipt must suppress a restored inbox copy" + ); + + std::thread::sleep(Duration::from_millis(2)); + let third = send_to_inbox(&inbox, "carol", Some("third"), None, &[], "three").unwrap(); + assert_eq!( + new_arrivals(&inbox, &mut seen) + .into_iter() + .map(|message| message.filename) + .collect::>(), + [third] + ); +} + +#[test] +fn staged_ownership_survives_archive_and_never_repastes() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + let archive = archive_dir(agent.path()); + let filename = send_to_inbox(&inbox, "cos", Some("owned"), None, &[], "body").unwrap(); + let message = message::list_inbox(&inbox).unwrap().pop().unwrap(); + let expected = render_without_catalog(&message); + let mut pending = VecDeque::from([PendingNotice::message(message)]); + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), + retry_outcomes: Mutex::new(VecDeque::from([ + PokeOutcome::Staged, + PokeOutcome::Delivered, + ])), + }; + + flush_without_catalog(None, &mut pending, &poker); + assert_eq!(pending.len(), 1); + assert_eq!(pending[0].staged_text(), Some(expected.as_str())); + + archive_msg(&inbox, &archive, &filename).unwrap(); + prune_archived_pending(&inbox, &mut pending); + assert_eq!( + pending.len(), + 1, + "an already-started paste remains inspection-owned across archive" + ); + + flush_without_catalog(None, &mut pending, &poker); + assert_eq!(pending.len(), 1); + flush_without_catalog(None, &mut pending, &poker); + assert!(pending.is_empty()); + assert_eq!(poker.pokes.lock().unwrap().as_slice(), [expected.as_str()]); + assert_eq!( + poker.retries.lock().unwrap().as_slice(), + [expected.as_str(), expected.as_str()] + ); +} + +// ----------------------------------------------------------------------------------------- +// SPIKE — producer-side event supersede against staged payload ownership. +// +// Supersede is the one genuinely new semantic in +// `docs/vrs/.experiments/2026-08-20-pipes-event-model-differentiation.md`, and it archives an +// inbox item *from outside DING*, possibly while DING already owns that item's payload in a +// composer. These two tests are the seam: if either fails, the differentiated event model's +// main earned semantic is unsafe and the design collapses back to unification. +// ----------------------------------------------------------------------------------------- + +/// A real catalog with one agent-owned stream. +fn event_catalog() -> (tempfile::TempDir, PathBuf) { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "hetz", "worker", None); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let declaration = catalog.path().join("hetz/worker/agent.kdl"); + let source = std::fs::read_to_string(&declaration).unwrap(); + std::fs::write( + declaration, + source.replacen("\n}\n", "\n stream \"gh-ci\" {}\n}\n", 1), + ) + .unwrap(); + let inbox = inbox_dir(&catalog.path().join("hetz").join("worker")); + (catalog, inbox) +} + +fn emit_ci(root: &Path, event_id: &str, supersede: bool) -> String { + crate::event::emit( + root, + "hetz", + "hetz.worker", + "gh-ci", + event_id, + Some("pr-42"), + Some(&format!("CI {event_id} on PR #42")), + event_id, + supersede, + ) + .unwrap() + .filename +} + +fn flush_in(root: &Path, pending: &mut VecDeque, poker: &dyn Poker) { + flush_pending( + DingContext { + catalog_root: root, + this_host: "hetz", + recipient: "hetz.worker", + }, + None, + pending, + poker, + ); +} + +/// The race: the producer supersedes event N *while DING owns N's staged payload*. The +/// existing ownership rules must carry it — N is pasted exactly once and never again, the +/// archived-and-not-retained head releases FIFO, and N+1 still delivers. Nothing about +/// `flush_pending` or `prune_archived_pending` changes to make this true. +#[test] +fn a_producer_supersede_of_a_staged_event_never_repastes_and_the_successor_delivers() { + let (catalog, inbox) = event_catalog(); + let root = catalog.path(); + + let failure = emit_ci(root, "failure", true); + let mut seen = HashSet::new(); + let mut pending: VecDeque = new_arrivals(&inbox, &mut seen) + .into_iter() + .map(PendingNotice::message) + .collect(); + let failure_text = pending[0].text( + DingContext { + catalog_root: root, + this_host: "hetz", + recipient: "hetz.worker", + }, + &mut None, + ); + assert!( + failure_text.starts_with("[DING] » hetz.worker/gh-ci:"), + "an event announces itself as a world-event: {failure_text}" + ); + + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([ + // the successor, once ownership of the superseded head is released + PokeOutcome::Delivered, + ])), + retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::NotRetained])), + }; + // DING stages the failure notice and owns it. + let stage_only = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), + retry_outcomes: Mutex::new(VecDeque::new()), + }; + flush_in(root, &mut pending, &stage_only); + assert_eq!(pending[0].staged_text(), Some(failure_text.as_str())); + + // The producer now supersedes: `success` is materialized and `failure` is archived under + // DING's feet. + let success = emit_ci(root, "success", true); + assert!(!inbox.join(&failure).exists(), "the head was retired"); + assert!(inbox.join(&success).exists(), "the successor is unread"); + + pending.extend( + new_arrivals(&inbox, &mut seen) + .into_iter() + .map(PendingNotice::message), + ); + prune_archived_pending(&inbox, &mut pending); + assert_eq!( + pending.len(), + 2, + "the staged-but-archived head keeps ownership; the successor queues behind it" + ); + + flush_in(root, &mut pending, &poker); + + assert!(pending.is_empty(), "FIFO drained, nothing stuck"); + let success_text = poker.pokes.lock().unwrap()[0].clone(); + assert!( + success_text.contains("CI success on PR #42"), + "{success_text}" + ); + assert_eq!( + stage_only.pokes.lock().unwrap().as_slice(), + [failure_text.as_str()], + "the superseded notice was pasted exactly once, ever" + ); + assert_eq!( + poker.pokes.lock().unwrap().len(), + 1, + "and the only fresh paste after supersede is the successor" + ); + assert_eq!( + poker.retries.lock().unwrap().as_slice(), + [failure_text.as_str()], + "the superseded head was released by inspection only, never re-pasted" + ); +} + +/// The pessimistic half of the same race: the adapter still sees the superseded notice in the +/// composer. Ownership is retained, FIFO stays blocked behind it, and the successor is *not* +/// pasted on top of a live payload. Supersede therefore cannot leak a second paste into a +/// composer that is still holding the first. +#[test] +fn a_superseded_but_still_retained_staged_event_keeps_ownership_without_repasting() { + let (catalog, inbox) = event_catalog(); + let root = catalog.path(); + + emit_ci(root, "failure", true); + let mut seen = HashSet::new(); + let mut pending: VecDeque = new_arrivals(&inbox, &mut seen) + .into_iter() + .map(PendingNotice::message) + .collect(); + let failure_text = pending[0].text( + DingContext { + catalog_root: root, + this_host: "hetz", + recipient: "hetz.worker", + }, + &mut None, + ); + + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), + retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), + }; + flush_in(root, &mut pending, &poker); + + emit_ci(root, "success", true); + pending.extend( + new_arrivals(&inbox, &mut seen) + .into_iter() + .map(PendingNotice::message), + ); + prune_archived_pending(&inbox, &mut pending); + flush_in(root, &mut pending, &poker); + + assert_eq!(pending.len(), 2, "later FIFO work remains blocked"); + assert_eq!( + poker.pokes.lock().unwrap().as_slice(), + [failure_text.as_str()], + "the successor is never pasted on top of a retained payload" + ); + assert_eq!( + poker.retries.lock().unwrap().as_slice(), + [failure_text.as_str()], + "the retained superseded notice is retried by inspection only" + ); +} + +/// Platforms that cannot hardlink through the open-file descriptor path (macOS fdescfs +/// rejects linkat(AT_SYMLINK_FOLLOW) on /dev/fd/N with EPERM) fall back to a byte-copy +/// archive receipt. The differentiated supersession semantics must hold unchanged on that +/// path, the receipt must carry exactly the validated bytes, and no staging file may leak +/// into the archive. +#[test] +#[cfg(debug_assertions)] +fn archive_copy_fallback_preserves_supersede_ownership_without_staging_leftovers() { + crate::event::TEST_FORCE_ARCHIVE_RECEIPT_COPY.store(true, Ordering::Relaxed); + struct ResetGuard; + impl Drop for ResetGuard { + fn drop(&mut self) { + crate::event::TEST_FORCE_ARCHIVE_RECEIPT_COPY.store(false, Ordering::Relaxed); + } + } + let _guard = ResetGuard; + + let (catalog, inbox) = event_catalog(); + let root = catalog.path(); + let archive = crate::message::archive_dir(&root.join("hetz").join("worker")); + + let failure_filename = emit_ci(root, "failure", true); + let mut seen = HashSet::new(); + let mut pending: VecDeque = new_arrivals(&inbox, &mut seen) + .into_iter() + .map(PendingNotice::message) + .collect(); + assert_eq!(pending.len(), 1); + let failure_bytes = + std::fs::read(inbox.join(&failure_filename)).expect("staged event bytes"); + let failure_text = pending[0].text( + DingContext { + catalog_root: root, + this_host: "hetz", + recipient: "hetz.worker", + }, + &mut None, + ); + + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), + retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Staged])), + }; + flush_in(root, &mut pending, &poker); + + emit_ci(root, "success", true); + pending.extend( + new_arrivals(&inbox, &mut seen) + .into_iter() + .map(PendingNotice::message), + ); + prune_archived_pending(&inbox, &mut pending); + flush_in(root, &mut pending, &poker); + + assert_eq!(pending.len(), 2, "later FIFO work remains blocked"); + assert_eq!( + poker.pokes.lock().unwrap().as_slice(), + [failure_text.as_str()], + "the successor is never pasted on top of a retained payload" + ); + assert!(!inbox.join(&failure_filename).exists(), "head was archived"); + let receipt = std::fs::read(archive.join(&failure_filename)) + .expect("byte-copy archive receipt exists"); + assert_eq!( + receipt, failure_bytes, + "the copy receipt carries exactly the validated bytes" + ); + let staging_leftovers: Vec<_> = std::fs::read_dir(&archive) + .unwrap() + .filter_map(Result::ok) + .map(|entry| entry.file_name().to_string_lossy().into_owned()) + .filter(|name| name.starts_with(".st2-archive-")) + .collect(); + assert!( + staging_leftovers.is_empty(), + "no staging files survive in the archive: {staging_leftovers:?}" + ); +} + +#[test] +fn archived_not_retained_releases_fifo_without_repasting_owned_notice() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + let archive = archive_dir(agent.path()); + let first = send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); + std::thread::sleep(Duration::from_millis(2)); + let second = send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); + let mut pending: VecDeque = message::list_inbox(&inbox) + .unwrap() + .into_iter() + .map(PendingNotice::message) + .collect(); + let first_text = notice_text_without_catalog(&pending[0]); + let second_text = notice_text_without_catalog(&pending[1]); + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([ + PokeOutcome::Staged, + PokeOutcome::Delivered, + ])), + retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::NotRetained])), + }; + + flush_without_catalog(None, &mut pending, &poker); + archive_msg(&inbox, &archive, &first).unwrap(); + prune_archived_pending(&inbox, &mut pending); + flush_without_catalog(None, &mut pending, &poker); + + assert!(pending.is_empty()); + assert_eq!( + poker.pokes.lock().unwrap().as_slice(), + [first_text.as_str(), second_text.as_str()] + ); + assert_eq!(poker.retries.lock().unwrap().as_slice(), [first_text]); + assert!(!inbox.join(first).exists()); + assert!(inbox.join(second).exists()); +} + +#[test] +fn unread_not_retained_keeps_fifo_ownership_without_repasting() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); + std::thread::sleep(Duration::from_millis(2)); + send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); + let mut pending: VecDeque = message::list_inbox(&inbox) + .unwrap() + .into_iter() + .map(PendingNotice::message) + .collect(); + let first_text = notice_text_without_catalog(&pending[0]); + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::from([ + PokeOutcome::Staged, + PokeOutcome::Delivered, + ])), + retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::NotRetained])), + }; + + flush_without_catalog(None, &mut pending, &poker); + flush_without_catalog(None, &mut pending, &poker); + + assert_eq!(pending.len(), 2); + assert_eq!(pending[0].staged_text(), Some(first_text.as_str())); + assert_eq!( + poker.pokes.lock().unwrap().as_slice(), + [first_text.as_str()] + ); + assert_eq!(poker.retries.lock().unwrap().as_slice(), [first_text]); +} + +#[test] +fn pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + let archive = archive_dir(agent.path()); + let status_path = status::status_path(agent.path()); + let first = send_to_inbox(&inbox, "alice", Some("first"), None, &[], "one").unwrap(); + std::thread::sleep(Duration::from_millis(2)); + let second = send_to_inbox(&inbox, "bob", Some("second"), None, &[], "two").unwrap(); + let mut pending: VecDeque = message::list_inbox(&inbox) + .unwrap() + .into_iter() + .map(PendingNotice::message) + .collect(); + let poker = RecordingPoker::live(); + + archive_msg(&inbox, &archive, &first).unwrap(); + prune_archived_pending(&inbox, &mut pending); + assert_eq!( + pending + .iter() + .filter_map(|notice| match notice { + PendingNotice::Message { message, .. } => Some(message.filename.as_str()), + PendingNotice::Recovery { .. } | PendingNotice::Adopted { .. } => None, + }) + .collect::>(), + [second.as_str()] + ); + + *poker.failures.lock().unwrap() = 1; + status::set_state(&status_path, status::State::Busy).unwrap(); + flush_without_catalog(Some(&status_path), &mut pending, &poker); + assert_eq!(pending.len(), 1, "a failed head remains queued"); + flush_without_catalog(Some(&status_path), &mut pending, &poker); + assert!(pending.is_empty()); + + let third = send_to_inbox(&inbox, "carol", Some("third"), None, &[], "three").unwrap(); + pending.extend( + message::list_inbox(&inbox) + .unwrap() + .into_iter() + .filter(|message| message.filename == third) + .map(PendingNotice::message), + ); + status::set_state(&status_path, status::State::Dnd).unwrap(); + flush_without_catalog(Some(&status_path), &mut pending, &poker); + assert_eq!(pending.len(), 1, "fresh dnd suppresses delivery"); + + let stale = std::time::SystemTime::now() - status::STATUS_STALE - Duration::from_secs(1); + let stale_ms = stale + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis(); + std::fs::write(&status_path, format!("dnd\nv1 {stale_ms}\n")).unwrap(); + flush_without_catalog(Some(&status_path), &mut pending, &poker); + assert!( + pending.is_empty(), + "stale dnd reads unknown and no longer suppresses" + ); + + let calls = poker.calls.lock().unwrap(); + assert_eq!(calls.len(), 3); + assert_eq!(calls[0], calls[1], "the failed FIFO head retries first"); + assert!(calls[0].contains("second")); + assert!(calls[2].contains("third")); +} + +#[test] +fn staged_and_recovery_delivery_do_not_traverse_the_catalog() { + let catalog = tempfile::tempdir().unwrap(); + let context = DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }; + + let mut staged = + PendingNotice::message(msg("1785070000000-abc123.md", "h.sender", Some("staged"))); + staged.set_staged_text(Some("immutable staged notice".to_string())); + let mut pending = VecDeque::from([staged]); + let poker = OwnershipPoker { + pokes: Mutex::new(Vec::new()), + retries: Mutex::new(Vec::new()), + poke_outcomes: Mutex::new(VecDeque::new()), + retry_outcomes: Mutex::new(VecDeque::from([PokeOutcome::Delivered])), + }; + crate::discovery::reset_discovery_walk_count_for_test(); + flush_pending(context, None, &mut pending, &poker); + assert_eq!( + crate::discovery::discovery_walk_count_for_test(), + 0, + "immutable staged delivery must not depend on catalog traversal" + ); + + let mut pending = VecDeque::from([PendingNotice::Recovery { + startup: HashSet::from(["1785070000000-def456.md".to_string()]), + in_inbox: true, + staged_text: None, + }]); + let poker = RecordingPoker::live(); + crate::discovery::reset_discovery_walk_count_for_test(); + flush_pending(context, None, &mut pending, &poker); + assert_eq!( + crate::discovery::discovery_walk_count_for_test(), + 0, + "generic recovery delivery must not depend on catalog traversal" + ); + assert_eq!(poker.calls.lock().unwrap().as_slice(), [RECOVERY_POKE]); +} + +#[test] +fn one_pending_batch_traverses_the_catalog_once() { + let catalog = tempfile::tempdir().unwrap(); + declare_agent(catalog.path(), "h", "recipient", None); + let mut pending = (0..10) + .map(|index| { + PendingNotice::message(msg( + &format!("17850700000{index:02}-abc{index:03}.md"), + "h.sender", + Some("batch"), + )) + }) + .collect::>(); + let poker = RecordingPoker::live(); + + crate::discovery::reset_discovery_walk_count_for_test(); + flush_pending( + DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }, + None, + &mut pending, + &poker, + ); + + assert!(pending.is_empty()); + assert_eq!(poker.calls.lock().unwrap().len(), 10); + assert_eq!( + crate::discovery::discovery_walk_count_for_test(), + 1, + "one pending batch must share one catalog observation" + ); +} + +#[test] +fn startup_recovery_notice_retries_in_memory() { + let poker = RecordingPoker::live(); + *poker.failures.lock().unwrap() = 1; + let mut pending = VecDeque::from([PendingNotice::Recovery { + startup: HashSet::from(["1785070000000-abc123.md".to_string()]), + in_inbox: true, + staged_text: None, + }]); + + flush_without_catalog(None, &mut pending, &poker); + assert_eq!(pending.len(), 1); + flush_without_catalog(None, &mut pending, &poker); + assert!(pending.is_empty()); + + let calls = poker.calls.lock().unwrap(); + assert_eq!(calls.len(), 2); + assert_eq!(calls[0], calls[1], "the failed FIFO head retries first"); + assert_eq!(calls[0], RECOVERY_POKE); +} + +#[test] +fn startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + let status_path = status::status_path(agent.path()); + status::set_state(&status_path, status::State::Available).unwrap(); + send_to_inbox(&inbox, "old", Some("seeded"), None, &[], "old").unwrap(); + + let poker = RecordingPoker::live(); + let stop = AtomicBool::new(false); + let config = DingConfig { + poll: Duration::from_millis(5), + status_refresh: Duration::from_secs(60), + }; + + // Two separate barriers, each with its own generous budget. Sharing one 3s deadline + // across both meant the first wait could spend it: the loop then stopped after a single + // poke and the count assertion failed while naming neither barrier. The ceiling bounds a + // loop that never ran at all, so a loaded host cannot turn it into a failure — and `stop` + // is set on every path, because panicking here would leave `run_ding` looping forever. + let mut polled = false; + let mut poked_twice = false; + std::thread::scope(|scope| { + scope.spawn(|| { + polled = await_ding_progress(|| poker.probes.load(Ordering::SeqCst) > 0); + send_to_inbox(&inbox, "new", Some("post-start"), None, &[], "new").unwrap(); + poked_twice = await_ding_progress(|| poker.calls.lock().unwrap().len() >= 2); + stop.store(true, Ordering::SeqCst); + }); + + run_ding( + DingContext { + catalog_root: agent.path(), + this_host: "h", + recipient: "h.recipient", + }, + &inbox, + Some(&status_path), + &poker, + &config, + &stop, + ) + .unwrap(); + }); + + assert!(polled, "the ding loop never reached its first poll"); + assert!(poked_twice, "the ding loop never delivered both notices"); + let calls = poker.calls.lock().unwrap(); + assert_eq!(calls.len(), 2); + assert_eq!(calls[0], RECOVERY_POKE); + assert!(calls[1].contains("post-start")); + assert!(!calls.iter().any(|call| call.contains("seeded"))); +} + +#[test] +fn deferred_delivery_backoff_bounds_short_lived_pty_attempts() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + let status_path = status::status_path(agent.path()); + status::set_state(&status_path, status::State::Available).unwrap(); + send_to_inbox(&inbox, "alice", Some("active"), None, &[], "active").unwrap(); + + let poker = RecordingPoker::live(); + poker.defer.store(true, Ordering::SeqCst); + let stop = AtomicBool::new(false); + let config = DingConfig { + poll: Duration::from_millis(20), + status_refresh: Duration::from_secs(60), + }; + + // Synchronize on the loop's own progress rather than a wall-clock window. `session_alive` + // runs exactly once per iteration, so `probes` counts polls directly: stop only once the + // first poke landed AND the loop polled several more times, which is precisely when a + // missing backoff would poke again. A fixed window instead ends wherever the host's + // scheduler leaves it — on a loaded machine before the first poll, so the test observed zero + // pokes and failed while asserting nothing about backoff. + const POLLS_AFTER_FIRST_POKE: usize = 5; + let barrier = Duration::from_secs(30); + std::thread::scope(|scope| { + scope.spawn(|| { + let deadline = Instant::now() + barrier; + while poker.calls.lock().unwrap().is_empty() && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(2)); + } + let polled = poker.probes.load(Ordering::SeqCst) + POLLS_AFTER_FIRST_POKE; + while poker.probes.load(Ordering::SeqCst) < polled && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(2)); + } + stop.store(true, Ordering::SeqCst); + }); + run_ding( + DingContext { + catalog_root: agent.path(), + this_host: "h", + recipient: "h.recipient", + }, + &inbox, + Some(&status_path), + &poker, + &config, + &stop, + ) + .unwrap(); + }); + + // Without this the counter-test is vacuous: one poke across one poll proves no backoff. + assert!( + poker.probes.load(Ordering::SeqCst) > POLLS_AFTER_FIRST_POKE, + "the loop did not poll again, so no backoff was exercised" + ); + assert_eq!( + poker.calls.lock().unwrap().len(), + 1, + "an unsafe composer must not respawn pty peek/send children every inbox poll" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn idle_ding_does_not_spin_on_its_own_inbox_reads() { + let agent = tempfile::tempdir().unwrap(); + let inbox = inbox_dir(agent.path()); + std::fs::create_dir_all(&inbox).unwrap(); + let status_path = status::status_path(agent.path()); + status::set_state(&status_path, status::State::Available).unwrap(); + + let poker = RecordingPoker::live(); + let stop = AtomicBool::new(false); + let config = DingConfig { + poll: Duration::from_millis(20), + status_refresh: Duration::from_secs(60), + }; + let started = Instant::now(); + + std::thread::scope(|scope| { + scope.spawn(|| { + std::thread::sleep(Duration::from_millis(150)); + stop.store(true, Ordering::SeqCst); + }); + run_ding( + DingContext { + catalog_root: agent.path(), + this_host: "h", + recipient: "h.recipient", + }, + &inbox, + Some(&status_path), + &poker, + &config, + &stop, + ) + .unwrap(); + }); + + assert!(started.elapsed() >= Duration::from_millis(140)); + assert!( + poker.probes.load(Ordering::SeqCst) <= 20, + "idle DING must sleep at its configured cadence" + ); +} + +#[test] +fn session_liveness_probe_reports_only_positive_evidence() { + use crate::harness_state::SessionLiveness; + let tmp = tempfile::tempdir().unwrap(); + + // An unreadable registry proves nothing: missing or garbled pidfiles downgrade nothing. + assert_eq!( + session_liveness_in(tmp.path(), "absent"), + SessionLiveness::Indeterminate + ); + std::fs::write(tmp.path().join("garbled.pid"), "not-a-pid\n").unwrap(); + assert_eq!( + session_liveness_in(tmp.path(), "garbled"), + SessionLiveness::Indeterminate + ); + + // Our own pid is positive evidence of life. + std::fs::write( + tmp.path().join("live.pid"), + format!("{}\n", std::process::id()), + ) + .unwrap(); + assert_eq!( + session_liveness_in(tmp.path(), "live"), + SessionLiveness::Alive + ); + + // A reaped child is positive evidence of death (ESRCH). + let mut child = std::process::Command::new("true").spawn().unwrap(); + let pid = child.id(); + child.wait().unwrap(); + std::fs::write(tmp.path().join("dead.pid"), format!("{pid}\n")).unwrap(); + assert_eq!( + session_liveness_in(tmp.path(), "dead"), + SessionLiveness::Dead + ); +} + +/// A deferral is the one outcome that both delivers nothing and leaves nothing behind, so it +/// has to carry why. The two causes want different responses: a human drafting in a harness we +/// understand clears on its own, a pane no harness can locate never will. +#[test] +fn a_deferral_names_the_cause_that_produced_it() { + let text = "[DING] ? cos: guarded [id:abc123]"; + for (screen, expected) in [ + ( + human_codex_screen(), + DeferralReason::ComposerChanged { harness: "codex" }, + ), + // Located, but an unrecognised footer proves nothing about the screen. This is a wait, + // not a coverage gap, and must not be reported as an unlocatable pane. + ( + idle_codex_screen_with_footer("Esc to interrupt"), + DeferralReason::ComposerUnproven { harness: "codex" }, + ), + ( + "unrecognized renderer".to_string(), + DeferralReason::NoMaintainedComposer, + ), + ] { + let outcome = observed_poke_with_window( + text, + &mut || Ok(screen.clone()), + &mut || Ok(()), + &mut || Ok(()), + &mut || {}, + &mut || Ok(()), + Duration::ZERO, + ) + .unwrap(); + assert_eq!(outcome, PokeOutcome::Deferred(expected)); + } +} + +/// The deferral has to reach the watch loop, which is the only place that knows the recipient +/// and can decide whether it is worth saying. +#[test] +fn flush_reports_a_deferral_outward() { + let catalog = tempfile::tempdir().unwrap(); + let context = DingContext { + catalog_root: catalog.path(), + this_host: "h", + recipient: "h.recipient", + }; + let notice = || { + VecDeque::from([PendingNotice::message(msg( + "1785070000000-abc123.md", + "h.sender", + Some("hello"), + ))]) + }; + + let poker = RecordingPoker::live(); + poker.defer.store(true, Ordering::SeqCst); + assert_eq!( + flush_pending(context, None, &mut notice(), &poker).deferred, + Some(DeferralReason::NoMaintainedComposer) + ); + + // A delivery that lands has nothing to report. + let poker = RecordingPoker::live(); + assert_eq!( + flush_pending(context, None, &mut notice(), &poker).deferred, + None + ); +} + +/// A pane stuck in one verdict costs one line, not one per retry — but a changed verdict, and +/// the first deferral after delivery resumed, are both news. +#[test] +fn only_a_changed_deferral_verdict_is_worth_reporting() { + let changed = DeferralReason::ComposerChanged { harness: "codex" }; + let unknown = DeferralReason::NoMaintainedComposer; + let mut journal = DeferralJournal::default(); + + assert!(journal.observe(Some(changed))); + assert!(!journal.observe(Some(changed))); + assert!(journal.observe(Some(unknown))); + assert!(!journal.observe(Some(unknown))); + assert!(!journal.observe(None)); + assert!(journal.observe(Some(unknown))); +} diff --git a/src/hooks.rs b/src/hooks.rs index 41225d2b..65b0a2aa 100644 --- a/src/hooks.rs +++ b/src/hooks.rs @@ -807,6 +807,7 @@ mod tests { assert!(!argv_invokes_omp(&["/opt/bin/omph".into()], root)); } + #[test] fn pi_launch_classification_is_exact() { let root = Path::new("/catalog"); assert!(command_invokes_pi("exec pi -a 'boot'")); diff --git a/src/main.rs b/src/main.rs index 661ceae2..72632a13 100644 --- a/src/main.rs +++ b/src/main.rs @@ -6,1252 +6,16 @@ use std::path::{Path, PathBuf}; use std::time::Duration; use anyhow::{Context, Result}; -use clap::{Args, CommandFactory, Parser, Subcommand}; +use clap::{CommandFactory, Parser}; use st2::{ HostLock, Runner, SystemRunner, UpReport, detect_host, ding, discover, exec_state_dir, message, up_loop, up_once, }; -#[derive(Parser)] -#[command( - name = "st2", - version = st2::version::display_version(), - about = "Harness-agnostic runner over a unified catalog+inbox folder" -)] -struct Cli { - /// Catalog (or single-file fleet spec) to use. Defaults to $CATALOG, then - /// ${XDG_STATE_HOME:-$HOME/.local/state}/st2/default/catalog. - #[arg(long = "catalog", global = true, value_name = "PATH")] - catalog_path: Option, - #[command(subcommand)] - command: Command, -} - -#[derive(Subcommand)] -enum Command { - /// Discover and print every agent spec under a catalog+inbox folder. - Ls { - /// Legacy positional catalog/spec path. Prefer --catalog; defaults to $CATALOG, then the - /// default st2 catalog. - #[arg(conflicts_with = "catalog_path")] - root: Option, - }, - /// Supervise a catalog+inbox folder: reconcile on a folder-watch + timer, keeping each agent's - /// ptys running. With --once, do a single pass and exit. - Up { - /// Legacy positional catalog/spec path. Prefer --catalog; defaults to $CATALOG, then the - /// default st2 catalog. - #[arg(conflicts_with = "catalog_path")] - root: Option, - /// Host to filter on (which agents this machine runs). Defaults to the local hostname. - #[arg(long)] - host: Option, - /// Do a single reconcile pass and exit, instead of looping. - #[arg(long)] - once: bool, - /// Materialize every local agent's render block and exit without reconciling or spawning. - #[arg(long, conflicts_with = "once")] - materialize_only: bool, - /// Limit materialization to one declared agent identity. - #[arg(long)] - agent: Option, - /// Select one exact local task. Use with --materialize-only to render only its owner, or - /// with --once to render its owner and reconcile only that task. - #[arg(long, conflicts_with = "agent")] - task: Option, - /// Seconds between timer-driven reconcile passes when looping (folder changes reconcile - /// immediately regardless). - #[arg(long, default_value_t = 30)] - interval: u64, - }, - /// Native message bus: send/list/read/archive/reply over agents' `resources/inbox`. - /// The stable wire format is a `-.md` Markdown file. - #[command(subcommand)] - Message(MessageCmd), - /// Declared event streams: durable, bounded, idempotent ingress into an agent inbox. - #[command(subcommand)] - Event(EventCmd), - /// Self-author declared event streams through the serialized catalog path. - #[command(subcommand)] - Stream(StreamCmd), - /// Idempotent JSON request/reply transport for declared non-agent service principals. - #[command(subcommand)] - Request(RequestCmd), - /// An agent's working-state context for lossless restart: read/write/append. - #[command(subcommand)] - Context(ContextCmd), - /// An agent's declared Resource bindings (a named, exact URI a peer can resolve): - /// ls/read/add/remove/rename. - #[command(subcommand)] - Resource(ResourceCmd), - /// Install `st2 up` as a systemd-user service on headless Linux. macOS stays manual (TCC). - /// Subcommands: install / status / uninstall. - #[command(subcommand)] - Service(ServiceCmd), - /// Install and approve the embedded Claude Code channel plugin. - #[command(subcommand)] - ClaudeChannel(ClaudeChannelCmd), - /// Explicit lifecycle-hook management. `up` and materialization only verify; they never install - /// or refresh hooks. - #[command(subcommand)] - Hooks(HooksCmd), - /// Provider-native harness drivers and read-only typed-block expansion. - #[command(subcommand)] - Driver(DriverCmd), - /// The ding sidecar: watch an agent's `resources/inbox` and poke its pty (`[DING] …`) on each new - /// message. Busy does not suppress delivery; only fresh dnd defers FIFO. A startup backlog is - /// coalesced into one recovery notice. Long-running — st2 keeps it alive as a task alongside the - /// agent. Exits when the target pty session is gone. - /// `st2 ping` is an alias (the maintainer is renaming ding → ping, since dinging is the runner's - /// job now); it is the exact same command. - #[command(visible_alias = "ping")] - Ding { - /// The target pty session to poke (a `pty` session ref). Optional — defaults to `--identity` - /// (an agent IS its pty, so the session to poke is the identity), so `st2 ding --identity X` - /// is the common form. - session: Option, - /// Whose inbox to watch — bus id or identity. Defaults to `$ST_AGENT`. Also the default poke - /// target when no positional session is given. - #[arg(long)] - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--identity`. - #[arg(long = "agent-id", conflicts_with = "identity")] - agent_id: Option, - /// Catalog root. Defaults to `$CATALOG`. - #[arg(long, conflicts_with = "catalog_path")] - root: Option, - /// Host used to resolve `.` bus ids. Defaults to the local hostname. - #[arg(long)] - host: Option, - /// Poll/liveness cadence in milliseconds (folder changes poke immediately regardless). - #[arg(long, default_value_t = 1000)] - interval: u64, - }, - /// Internal controlled Codex launch. Generated only for `deliver "app-server"` tasks. - #[command(hide = true)] - CodexAppServer { - /// Exact agent bus identity that owns the controlled thread. - #[arg(long)] - identity: String, - /// Exact reconciled PTY task identity for this runtime. - #[arg(long)] - runtime_id: String, - /// Original structured Codex invocation, including its provider executable. - #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] - codex_argv: Vec, - }, - /// Internal Claude MCP channel server started by Claude from its rendered project declaration. - #[command(hide = true)] - ClaudeMcp { - #[arg(long)] - identity: String, - }, - /// Get or set an agent's presence status. No `--set` prints the status; no identity means yours - /// (`$ST_AGENT`). Settable: offline | available | busy | away | dnd (`unknown` is derived). - Status { - /// Whose status — bus id or identity. Defaults to you (`--as` / `$ST_AGENT`). - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// Set your status to this state instead of printing it. - #[arg(long = "set")] - set: Option, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Set or clear an agent's human-facing name without changing stable identity. - Rename(PresentationArgs), - /// Set or clear an agent's enduring responsibility description. - Describe(PresentationArgs), - /// Transactionally publish one canonical Agent Spec into the live catalog. - #[command(subcommand)] - Agent(AgentCmd), - /// Canonical declaration snapshots and crash-recoverable whole-catalog application. - #[command(subcommand)] - Catalog(CatalogCmd), - /// Explicit teardown: kill every live task of this host's catalog agents. The ONLY thing that ends - /// tasks (stopping/crashing st2 never does). Idempotent. - Down { - /// Optional positional catalog/spec path. Prefer --catalog; defaults to $CATALOG, then the - /// standard st2 catalog. - #[arg(conflicts_with = "catalog_path")] - root: Option, - /// Host to tear down. Defaults to the local hostname. - #[arg(long)] - host: Option, - }, - /// Print shell exports for a catalog's bus — `eval "$(st2 env --catalog )"` sets `CATALOG`/ - /// `ST_ROOT`/`PTY_ROOT` so native bus-aware tools target the catalog. - Env { - /// Optional positional catalog path. Prefer --catalog; defaults to $CATALOG, then the - /// standard st2 catalog. - #[arg(conflicts_with = "catalog_path")] - root: Option, - }, - /// Explicitly pre-trust workspaces in the ambient Claude and Codex configs. This is an operator - /// utility for harnesses that use those ambient configs; `st2 up` never calls it automatically. - /// Account-selecting commands should instead declare trust in the selected harness invocation. - Pretrust { - /// Workspace directories to mark trusted. - #[arg(required = true)] - dirs: Vec, - }, - /// Run an st2-spec eval end to end: copy the fixture, boot the team + judges, deliver the - /// kickoff, wait for the sup's confirmation, run the judges → verdict. `st2 eval ./cells//`. - Eval { - /// The eval folder (or its `.kdl` spec file). - folder: PathBuf, - /// Host. Defaults to the local hostname. - #[arg(long)] - host: Option, - /// Preserve the run's temp catalog instead of deleting it — for inspecting the worker repo - /// (`base..HEAD`), the judge outputs, and the bus after the run (e.g. a gate reproduction). - /// Seats are still torn down (no leaks). Also honored via `ST2_EVAL_KEEP`. - #[arg(long)] - keep: bool, - /// Emit the existing eval report as JSON without changing exit semantics. - #[arg(long)] - json: bool, - }, - /// Run `pty` against this catalog's bus with the env auto-set, so pty subcommands and the - /// interactive UI work without `eval "$(st2 env --catalog )"` first. Catalog selection follows - /// `--catalog`, `$CATALOG`, then the default st2 catalog. `CATALOG`/`ST_ROOT`/`PTY_ROOT` are - /// exported for the child exactly as `st2 env` would. No arguments launches the interactive pty - /// UI. - Pty { - /// Arguments passed through to `pty` verbatim (e.g. `ls`, `peek `). None → the UI. - #[arg(trailing_var_arg = true, allow_hyphen_values = true)] - args: Vec, - }, - /// Drop into `$SHELL` with this catalog's bus env set (`CATALOG`/`ST_ROOT`/`PTY_ROOT`), so `pty`, - /// bus-aware tools target the catalog for the whole session without `eval "$(st2 env …)"`. - /// The general form of `st2 pty`. Catalog selection follows `--catalog`, `$CATALOG`, then the - /// default st2 catalog; extra args go to the shell (e.g. `st2 shell -c "pty ls"`). - Shell { - /// Arguments passed through to `$SHELL` verbatim. None → an interactive shell. - #[arg(trailing_var_arg = true, allow_hyphen_values = true)] - args: Vec, - }, - /// Validate a rendered catalog against the runner contract (spec fields, folder layout, paths) - /// so any renderer can confirm it hit the spec before running. One line per issue with - /// a stable code; exits non-zero on any ERROR (`--strict` also fails on warnings). `--json` emits - /// machine output for a renderer's build gate. - Validate { - /// Legacy positional catalog path. Prefer --catalog; defaults to $CATALOG, then the default - /// st2 catalog. - #[arg(conflicts_with = "catalog_path")] - root: Option, - /// Host whose external workspace/task paths should be checked. Structural checks always - /// cover the whole catalog. Defaults to the local hostname. - #[arg(long)] - host: Option, - /// Validate this one unpublished canonical Agent Spec as an overlay on the live catalog. - #[arg(long, value_name = "FILE")] - candidate: Option, - /// Fail (non-zero exit) on warnings too, not just errors. - #[arg(long)] - strict: bool, - /// Emit the report as JSON instead of human-readable lines. - #[arg(long)] - json: bool, - }, - /// Health check for a catalog: active agents alive, suspended agents not live, and retired - /// agents fully absent. Exits non-zero on problems. - Doctor { - /// Legacy positional catalog path. Prefer --catalog; defaults to $CATALOG, then the default - /// st2 catalog. - #[arg(conflicts_with = "catalog_path")] - root: Option, - /// Host to check. Defaults to the local hostname. - #[arg(long)] - host: Option, - /// Require a live long-running `st2 up` host lock. Omit for manual/--once operation. - #[arg(long)] - require_supervisor: bool, - }, - /// List every agent in the catalog with presence and retirement state. `--json [--enrich]` is - /// the stable machine-readable roster. - Agents { - /// The catalog folder (like `st2 ls`/`up`). Falls back to `--root`/`$CATALOG`. - #[arg(conflicts_with = "catalog_path")] - catalog: Option, - /// Only agents whose effective status matches (offline|available|busy|away|dnd|unknown). - #[arg(long = "status")] - status: Option, - /// Select one exact Agent Spec by its fully qualified `.`. - #[arg(long, value_name = "HOST.IDENTITY")] - identity: Option, - /// Select one exact subject by its immutable agent ID (R24). - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// Machine-readable JSON array, including retirement and declared Resource bindings. - #[arg(long)] - json: bool, - /// With `--json`, add `lastActivity` + `inbox` count per agent. - #[arg(long)] - enrich: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Emit one fail-closed desired-task/runtime diagnostic snapshot. This is - /// read-only observation, not reconciliation or cutover authority. - Tasks { - /// Host whose desired tasks and runtime generations to inspect. Defaults to this host. - #[arg(long)] - host: Option, - /// Emit the versioned machine-readable envelope. Required in v1. - #[arg(long)] - json: bool, - }, - /// Clear one task's park after fixing what crash-looped it. A task parked by its `restart{}` - /// policy (mode=fail) stays parked for the rest of the supervisor run, and this is its per-task - /// exit: the running supervisor relaunches exactly this task on its next pass, leaving every - /// other task on the host untouched. `st2 tasks --json` reports which tasks are parked. - Unpark { - /// The parked task's runtime id, exactly as `st2 tasks --json` reports it. - task: String, - /// Host whose selected-catalog supervisor should grant the request. Defaults to this host. - #[arg(long)] - host: Option, - }, - /// Print a shell completion script for `st2` to stdout (`st2 completions `). - /// Generated from the live command tree, so it never drifts from the actual flags. - Completions { - /// The shell to generate completions for. - shell: clap_complete::Shell, - }, -} - -#[derive(Subcommand)] -enum DriverCmd { - /// Print one typed driver block as plain Agent Spec KDL without running it. - Expand { - /// KDL declaration that contains the typed driver block. - spec: PathBuf, - /// Select one local or fully qualified identity when the file contains multiple agents. - #[arg(long)] - agent: Option, - /// Host fallback when neither the declaration nor its catalog path supplies one. - #[arg(long)] - host: Option, - }, - /// Run the existing controlled Codex app-server path. - Codex { - #[arg(long)] - identity: String, - #[arg(long)] - runtime_id: String, - #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] - argv: Vec, - }, - /// Run the Claude session-owned MCP server over stdio. - ClaudeMcp { - #[arg(long)] - identity: Option, - }, - /// Deprecated name for the Claude MCP server. - // Keep this hidden command until no rendered configuration uses the old name. - #[command(hide = true)] - Claude { - #[arg(long)] - identity: String, - }, - /// Run Claude under the session-owned presence wrapper. - ClaudeSession { - #[arg(long)] - identity: String, - #[arg(long)] - runtime_id: String, - #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] - argv: Vec, - }, - /// Apply one Claude hook event (payload on stdin) to observed harness state. - ClaudeObserve { - #[arg(long)] - identity: String, - /// The wrapper's runtime/task ID; the record's pty session. Defaults to the identity. - #[arg(long)] - runtime_id: Option, - /// The Claude hook event name, e.g. `Stop` or `PermissionRequest`. - #[arg(long)] - event: String, - }, - /// Tee Claude's status-line payload (stdin JSON) into harness context, then chain to the - /// operator's own renderer. - ClaudeStatusline { - #[arg(long)] - identity: String, - }, - /// Run pi under the session-owned presence wrapper. - PiSession { - #[arg(long)] - identity: String, - #[arg(long)] - runtime_id: String, - #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] - argv: Vec, - }, - /// Run the pi native message channel over stdio, owned by the shipped pi extension. - PiChannel { - #[arg(long)] - identity: String, - }, - /// Run omp under the session-owned presence wrapper with a hard version gate. - OmpSession { - #[arg(long)] - identity: String, - #[arg(long)] - runtime_id: String, - #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] - argv: Vec, - }, - /// Run the omp native message channel over stdio, owned by the shipped omp extension. - OmpChannel { - #[arg(long)] - identity: String, - }, - /// Run OpenCode under the session-owned wrapper: presence, observed harness state, and native - /// server delivery over the wrapper-allocated local port. - OpencodeSession { - #[arg(long)] - identity: String, - #[arg(long)] - runtime_id: String, - #[arg(required = true, trailing_var_arg = true, allow_hyphen_values = true)] - argv: Vec, - }, -} - -#[derive(Subcommand)] -enum AgentCmd { - /// Author reversible whole-agent lifecycle intent in one canonical KDL declaration. - DesiredState { - /// Exact bus identity, or a bare stable identity only when unique — or the desired state - /// when `--id` names the subject. Authoring selects the declaration, never the address. - #[arg(value_name = "IDENTITY_OR_STATE")] - first: Option, - /// The desired state, when the first positional is the agent reference. - #[arg(value_name = "STATE")] - second: Option, - /// Exact immutable agent ID (R24). The first positional is then the desired state. - /// - /// Both positionals stay optional so the exact-ID form can shift them; clap refuses a - /// non-required positional ahead of a required one, which is why the state is validated in - /// the handler rather than by a positional `value_parser`. - #[arg(long = "id", conflicts_with = "second")] - agent_id: Option, - /// Required rationale for suspended/retired; forbidden for running. - #[arg(long)] - reason: Option, - /// Assert the ownership marker that owns this declaration, e.g. `nix`. - /// - /// A declaration carrying `meta { managed-by "nix" }` refuses ordinary authoring, - /// because the Nix projection is the writer of those bytes. This is how that - /// projection authors lifecycle on its own declaration — the transition it cannot - /// express in its own source, because the source change being projected is the - /// seat's removal. The assertion is admitted only when it names exactly the one - /// marker the declaration carries. - #[arg(long = "managed-by", value_name = "MARKER")] - managed_by: Option, - /// Host used only to resolve declarations whose host is omitted. - #[arg(long)] - host: Option, - /// Emit a stable JSON authoring receipt. - #[arg(long)] - json: bool, - }, - /// Assign or clear an agent's mutable address — one atomic address-book cutover with no - /// alias, redirect, or rename history. `--clear` restores the positional identity fallback. - Address(PresentationArgs), - /// Compute the authoritative digest bound by `agent publish --input-sha256`. - Digest { - /// A canonical KDL file containing exactly one top-level `agent` node. - #[arg( - long, - value_name = "FILE", - required_unless_present = "bundle", - conflicts_with = "bundle" - )] - spec: Option, - /// A create-only directory whose root contains exactly one canonical `agent.kdl`. - #[arg( - long, - value_name = "DIR", - required_unless_present = "spec", - conflicts_with = "spec" - )] - bundle: Option, - /// Emit the typed source-digest receipt as JSON. - #[arg(long)] - json: bool, - }, - /// Publish exactly one explicit-host, explicit-identity agent under a catalog-wide CAS lock. - Publish { - /// A canonical KDL file containing exactly one top-level `agent` node. - #[arg( - long, - value_name = "FILE", - required_unless_present = "bundle", - conflicts_with = "bundle" - )] - spec: Option, - /// A create-only directory whose root contains exactly one canonical `agent.kdl`. - #[arg( - long, - value_name = "DIR", - required_unless_present = "spec", - conflicts_with = "spec" - )] - bundle: Option, - /// Create only. An identical existing agent.kdl is reported as `unchanged`. - #[arg( - long, - required_unless_present = "expect_sha256", - conflicts_with = "expect_sha256" - )] - expect_absent: bool, - /// Replace only when the current agent.kdl has this lowercase SHA-256. - #[arg( - long, - value_name = "HEX", - required_unless_present = "expect_absent", - conflicts_with = "expect_absent" - )] - expect_sha256: Option, - /// SHA-256 returned by `st2 agent digest` for the exact source capability. - #[arg(long, value_name = "HEX")] - input_sha256: String, - /// Assert the ownership marker that owns the declaration being replaced, e.g. `nix`. - /// - /// A declaration carrying `meta { managed-by "nix" }` refuses an unasserted - /// replacement, because the Nix projection is the writer of those bytes. The - /// assertion is admitted only when it names exactly the one marker the incumbent - /// carries. Create-only publication has no incumbent and needs no assertion. - #[arg(long = "managed-by", value_name = "MARKER")] - managed_by: Option, - /// Emit the typed publication result as JSON. - #[arg(long)] - json: bool, - }, -} - -#[derive(Subcommand)] -enum CatalogCmd { - /// Emit one fail-closed declaration graph plus runtime observation envelope. - Graph { - /// Host used to resolve declarations with no host and host-local runtime facts. - #[arg(long)] - host: Option, - /// Emit the versioned machine-readable envelope. Required in v1. - #[arg(long)] - json: bool, - }, - /// Compute the authoritative digest bound by `catalog apply --input-sha256`. - Digest { - /// Complete prepared declaration directory. Runtime state and control paths are rejected. - #[arg(long, value_name = "DIR")] - prepared: PathBuf, - /// Emit the typed source-digest receipt as JSON. - #[arg(long)] - json: bool, - }, - /// Compare one prepared declaration directory with the coherent live catalog without writing. - Diff { - /// Complete prepared declaration directory. Runtime state and control paths are rejected. - #[arg(long, value_name = "DIR")] - prepared: PathBuf, - /// Expected canonical declaration-root SHA-256 of the live catalog. - #[arg(long, value_name = "HEX")] - expect_sha256: String, - /// Emit the versioned semantic-diff receipt. Required in v1. - #[arg(long)] - json: bool, - }, - /// Publish a complete prepared declaration directory as one absent catalog. - Bootstrap { - /// Complete prepared declaration directory. Runtime state and control paths are rejected. - #[arg(long, value_name = "DIR")] - prepared: PathBuf, - /// Root SHA-256 of the exact prepared projection being published. - #[arg(long, value_name = "HEX")] - input_sha256: String, - /// Emit the typed bootstrap receipt as JSON. - #[arg(long)] - json: bool, - }, - /// Capture the coherent declaration plane into a create-only canonical directory. - Snapshot { - /// Destination directory. It must be outside the live catalog. - #[arg(long, value_name = "DIR")] - output: PathBuf, - /// Hash and capture the declaration plane without parsing it. The captured directory - /// remains unvalidated and is suitable only as an exact-byte CAS preimage. - #[arg(long)] - raw_preimage: bool, - /// Emit the typed snapshot receipt as JSON. - #[arg(long)] - json: bool, - }, - /// Apply a complete canonical declaration directory under declaration-root CAS. - Apply { - /// Complete prepared declaration directory. Runtime state and control paths are rejected. - #[arg( - long, - value_name = "DIR", - required_unless_present = "resume", - conflicts_with = "resume" - )] - prepared: Option, - /// Root SHA-256 of the exact prepared projection being applied. - #[arg( - long, - value_name = "HEX", - required_unless_present = "resume", - conflicts_with = "resume" - )] - input_sha256: Option, - /// Expected canonical declaration-root SHA-256 of the live catalog. - #[arg( - long, - value_name = "HEX", - required_unless_present = "resume", - conflicts_with = "resume" - )] - expect_sha256: Option, - /// Match the current declaration plane without parsing it. The prepared catalog is still - /// fully validated; use this mode only when the current parser cannot admit the preimage. - #[arg(long, conflicts_with = "resume")] - raw_preimage: bool, - /// Resume the durable incomplete marker and internal stage without the original source. - #[arg(long, conflicts_with_all = ["prepared", "input_sha256", "expect_sha256"])] - resume: bool, - /// Emit the typed application receipt as JSON. - #[arg(long)] - json: bool, - }, - /// Move retired, runtime-free identities out of the live catalog into `.st2/archive`, leaving a - /// tombstone row in `st2 catalog graph --json`. An archived spec is not discoverable. - Archive { - /// Identity to archive, repeatable. Every named identity must be eligible or nothing moves. - #[arg( - long, - value_name = "IDENTITY", - required_unless_present = "all_retired", - conflicts_with = "all_retired" - )] - identity: Vec, - /// Archive every eligible retired identity of the selected host. Ineligible ones are - /// reported and skipped. - #[arg(long)] - all_retired: bool, - /// Host whose identities are archived. Defaults to this host; another host's runtime - /// records are not observable from here, so only the local host is eligible. - #[arg(long)] - host: Option, - /// Decide eligibility and print the plan without moving anything. - #[arg(long)] - dry_run: bool, - /// Emit the typed archive receipt as JSON. - #[arg(long)] - json: bool, - }, - /// Move one archived identity back into the live catalog. The exact reverse of `archive`. - Unarchive { - /// Archived identity to restore. - identity: String, - /// Host the identity was archived under. Defaults to this host. - #[arg(long)] - host: Option, - /// Emit the typed restoration receipt as JSON. - #[arg(long)] - json: bool, - }, -} - -/// Shared context for message subcommands: where the catalog is, who "I" am, and the local host. -/// Defaults come from the same env st2 sets on every task it spawns (`$CATALOG`, `$ST_AGENT`), so a -/// running agent needs no flags. -#[derive(Args)] -struct MsgCtx { - /// Legacy catalog/bus root override. Prefer global `--catalog`; defaults to `$CATALOG`, then the - /// default st2 catalog. - #[arg(long, conflicts_with = "catalog_path")] - root: Option, - /// The acting identity — who the message is `from` / whose inbox is "mine". An ordinary - /// address reference, unlike `$ST_AGENT`, which carries the exact agent ID; the two are - /// different strings once a subject declares an explicit `address`. Defaults to `$ST_AGENT`. - #[arg(long = "as")] - as_id: Option, - /// Host used to resolve `.` bus ids. Defaults to the local hostname. - #[arg(long)] - host: Option, -} - -/// `[] ` plus the mutually exclusive exact-ID form. -/// -/// `--id` takes the agent off the positional list, so the first positional is then the value — -/// the same `[identity] ` convention `st2 message read` and `st2 resource read` already -/// use, and the reason clap's exclusion is expressed against the second positional. -#[derive(Args)] -struct PresentationArgs { - /// Exact bus identity, or a bare stable identity only when unique in the selected catalog — - /// or the new value when `--id` names the subject. Authoring selects the declaration, never - /// the address. - #[arg(value_name = "IDENTITY_OR_TEXT")] - first: Option, - /// The new value, when the first positional is the agent reference. - #[arg(value_name = "TEXT")] - second: Option, - /// Exact immutable agent ID (R24). The first positional is then the new value. - #[arg(long = "id", conflicts_with = "second")] - agent_id: Option, - /// Remove the optional field. - #[arg(long)] - clear: bool, - /// Emit a stable JSON receipt or classified refusal. - #[arg(long)] - json: bool, - /// Host used only to resolve declarations whose host is omitted. - #[arg(long)] - host: Option, -} - -impl PresentationArgs { - /// The selected subject and the requested value, where `None` is the cleared representation. - fn selection(self) -> Result<(st2::identity::AgentSelector, Option, bool, Option)> - { - let (selector, value) = match self.agent_id { - Some(id) => (st2::identity::AgentSelector::Id(id), self.first), - None => ( - st2::identity::AgentSelector::Address( - self.first - .context("no agent selected: pass an agent reference or the `--id` form")?, - ), - self.second, - ), - }; - anyhow::ensure!( - !(self.clear && value.is_some()), - "--clear removes the field and takes no value" - ); - anyhow::ensure!( - self.clear || value.is_some(), - "a value is required unless --clear" - ); - Ok(( - selector, - if self.clear { None } else { value }, - self.json, - self.host, - )) - } -} - -#[derive(Subcommand)] -enum ServiceCmd { - /// Write the `st2.service` systemd-user unit, enable it (start on boot), and start it now. - /// Idempotent — safe to re-run. The unit runs `st2 up --catalog `; agents spawn in sibling - /// scopes, so a service restart never cascades to them. - Install { - /// Legacy positional catalog/spec path for `st2 up`. Prefer --catalog; defaults to - /// `$CATALOG`, then the default st2 catalog. It must exist at install time. - #[arg(conflicts_with = "catalog_path")] - catalog: Option, - /// Bake `--host ` into the unit. Omit to let `st2 up` auto-detect the hostname at runtime. - #[arg(long)] - host: Option, - /// Machine-local pty registry to export as PTY_ROOT in the unit. Omit to use - /// `/pty`. Useful when adopting live sessions from a legacy runner. - #[arg(long)] - pty_root: Option, - /// Supervisor memory ceiling (MiB). The agents live in sibling scopes and are NOT bounded. - #[arg(long = "memory-max-mb", default_value_t = st2::service::DEFAULT_MEMORY_MAX_MB)] - memory_max_mb: u64, - }, - /// Show the `st2.service` systemd status. - Status, - /// Stop, disable, and remove the `st2.service` unit. Idempotent. - Uninstall, -} - -#[derive(Subcommand)] -enum ClaudeChannelCmd { - /// Install or update the user plugin and its machine approval policy. - Install { - /// Install only the user plugin. An administrator will manage the machine policy. - #[arg(long)] - no_policy: bool, - }, - /// Verify the embedded files, Claude registration, plugin, and machine policy. - Status, - /// Remove the user plugin, marketplace, embedded files, and machine policy. - Uninstall { - /// Keep the machine approval policy in place. - #[arg(long)] - keep_policy: bool, - }, - /// Write only the machine policy. The main installer runs this through sudo. - #[command(hide = true)] - InstallPolicy, - /// Remove only the st2-owned machine policy fragment. - #[command(hide = true)] - UninstallPolicy, -} - -#[derive(Subcommand)] -enum HooksCmd { - /// Atomically publish this binary's immutable hook set and select it with a receipt. - Install { - /// Select this binary's exact hook set even when it is older or cannot be ordered. - #[arg(long)] - replace: bool, - /// Deprecated compatibility alias for `--replace`. - #[arg(long, hide = true)] - allow_downgrade: bool, - }, - /// Read-only verification of the selected receipt and every embedded hook byte. - Verify, - /// Verify this binary's immutable hook set without requiring it to be selected. - VerifyOwn, -} - -#[derive(Subcommand)] -enum ResourceCmd { - /// List an agent's declared Resource bindings. Defaults to your own. - Ls { - /// Whose declaration to read — bus id or bare identity. Defaults to you (`$ST_AGENT`). - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// Emit the bindings as a JSON array. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Read one declared binding. With a leading identity, from that agent; otherwise your own. - Read { - first: String, - second: Option, - /// Exact immutable agent ID (R24); `first` is then the binding name. - #[arg(long = "id", conflicts_with = "second")] - agent_id: Option, - /// Emit the binding as a JSON object. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Ask the resident profile runtime to observe one binding now and wait for exact evidence. - Refresh { - /// Binding name, or an agent selector when followed by a binding name. - first: String, - /// Binding name when the first positional selects the agent. - second: Option, - /// Exact target agent; defaults to --as / $ST_AGENT. - #[arg(long, conflicts_with = "second")] - agent: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. - #[arg(long = "agent-id", conflicts_with_all = ["second", "agent"])] - agent_id: Option, - /// Client-only wait bound in seconds. Expiry never cancels or retracts queued demand. - #[arg(long, default_value_t = 30)] - wait: u64, - /// Emit the stable receipt (or timeout envelope) as JSON. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Declare a Resource binding, or prove the identical binding already exists. - Add { - /// The agent-local binding name. - name: String, - /// The exact absolute URI this binding names (any `scheme:` — the identity is verbatim). - #[arg(long)] - uri: String, - /// Why this reference belongs in the declaration. - #[arg(long)] - reason: String, - /// Preserve the binding as no longer active for this agent, and say why. - #[arg(long = "inactive-reason", value_name = "TEXT")] - inactive_reason: Option, - /// Profile-specific observation selector as JSON. - #[arg(long = "selector-json", value_name = "JSON")] - selector_json: Option, - /// Exact target agent; defaults to --as / $ST_AGENT. - #[arg(long)] - agent: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. - #[arg(long = "agent-id", conflicts_with = "agent")] - agent_id: Option, - /// Emit a stable JSON receipt. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Remove one declared binding, or prove it is already absent. - Remove { - /// The agent-local binding name. - name: String, - /// Exact target agent; defaults to --as / $ST_AGENT. - #[arg(long)] - agent: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. - #[arg(long = "agent-id", conflicts_with = "agent")] - agent_id: Option, - /// Emit a stable JSON receipt. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Rename one declared binding's agent-local label, keeping its uri and reasons. - Rename { - /// The current binding name. - old: String, - /// The new binding name. - new: String, - /// Exact target agent; defaults to --as / $ST_AGENT. - #[arg(long)] - agent: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. - #[arg(long = "agent-id", conflicts_with = "agent")] - agent_id: Option, - /// Emit a stable JSON receipt. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, -} - -#[derive(Subcommand)] -enum ContextCmd { - /// Print an agent's context. Default = `now.md` (working state); `--decisions` the log; `--full` both. - Read { - /// Whose context — bus id or identity. Defaults to you (`$ST_AGENT`). - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// Print the decision log instead of the working state. - #[arg(long)] - decisions: bool, - /// Print the working state and the decision log. - #[arg(long)] - full: bool, - /// Print `now.md` only when it is newer than this many seconds. - #[arg(long, value_name = "SECONDS")] - fresh_within: Option, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Overwrite an agent's working state (`now.md`) from stdin. - Write { - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Append a single decision (with its reasoning) to the log. - Append { - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// The decision — a single line. - #[arg(long)] - decision: String, - /// Why — a single line. - #[arg(long)] - why: String, - #[command(flatten)] - ctx: MsgCtx, - }, -} - -#[derive(Subcommand)] -enum MessageCmd { - /// Send a new message to a recipient's inbox. - Send { - /// Recipient: a bus address (`.
`) or a bare address in the catalog. - #[arg(required_unless_present = "to_id", conflicts_with = "to_id")] - to: Option, - /// Recipient by exact immutable agent ID (R24). Mutually exclusive with the positional. - #[arg(long = "to-id")] - to_id: Option, - /// The message body. Read from stdin when omitted. - #[arg(short = 'm', long = "message")] - body: Option, - #[arg(long)] - subject: Option, - #[arg(long = "in-reply-to")] - in_reply_to: Option, - /// Comma-separated tags. - #[arg(long, value_delimiter = ',')] - tags: Vec, - /// Reuse one sender-owned operation result across exact retries. - #[arg(long = "idempotency-key")] - idempotency_key: Option, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Reply to a message in your inbox — recipient and threading are derived from it. - Reply { - /// The message filename in your inbox to reply to. - filename: String, - /// The reply body. Read from stdin when omitted. - #[arg(short = 'm', long = "message")] - body: Option, - /// Override the subject (defaults to `re: `). - #[arg(long)] - subject: Option, - /// Reuse one sender-owned operation result across exact retries. - #[arg(long = "idempotency-key")] - idempotency_key: Option, - #[command(flatten)] - ctx: MsgCtx, - }, - /// List an inbox (or `--archive`), sorted by send time. Defaults to your own. - Ls { - /// Whose inbox — bus id or identity. Defaults to you (`--as` / `$ST_AGENT`). - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// List the archive instead of the inbox. - #[arg(long)] - archive: bool, - /// Recovery-only: list the raw flat `/` box without catalog resolution. - #[arg(long)] - orphan: bool, - /// Print only the message count. - #[arg(long)] - count: bool, - /// Include full message bodies in JSON output (opt-in; default shape is unchanged). - #[arg(long)] - include_body: bool, - /// Show only messages from this sender. - #[arg(long = "from")] - from: Option, - /// Show only messages sent after this unix-millisecond timestamp. - #[arg(long)] - since: Option, - /// Machine-readable JSON array. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// List sender-owned durable message history. Defaults to your own index. - Sent { - /// Whose sent index — bus id or identity. Defaults to you (`--as` / `$ST_AGENT`). - identity: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with the positional reference. - #[arg(long = "id", conflicts_with = "identity")] - agent_id: Option, - /// Print only the indexed message count. Refuses unavailable or partial coverage. - #[arg(long)] - count: bool, - /// Include full message bodies in JSON output. - #[arg(long)] - include_body: bool, - /// Show only messages addressed to this canonical recipient. - #[arg(long = "to")] - to: Option, - /// Show only messages sent after this unix-millisecond timestamp. - #[arg(long)] - since: Option, - /// Machine-readable coverage envelope and rows. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Read one message. With a leading identity, read from that agent's box; otherwise your own. - Read { - /// Either the message filename, or an identity followed by a filename. - first: String, - /// The message filename (when `first` is an identity). - second: Option, - /// Exact immutable agent ID (R24) owning the box; `first` is then the filename. - #[arg(long = "id", conflicts_with = "second")] - agent_id: Option, - /// Read from the archive instead of the inbox. - #[arg(long)] - archive: bool, - /// Print the file verbatim (frontmatter + body), not a formatted view. - #[arg(long)] - raw: bool, - /// Machine-readable JSON. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Archive one message (inbox → archive). Defaults to your own inbox. - Archive { - /// Either the message filename, or an identity followed by a filename. - first: String, - /// The message filename (when `first` is an identity). - second: Option, - /// Exact immutable agent ID (R24) owning the box; `first` is then the filename. - #[arg(long = "id", conflicts_with = "second")] - agent_id: Option, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Show a message's thread — the message + everything replying to it (transitively), across the - /// catalog. `--tree` indents by reply depth; otherwise flat chronological. - Thread { - /// Either the message filename, or an identity followed by a filename. - first: String, - /// The message filename (when `first` is an identity). - second: Option, - /// Indented hierarchical output instead of flat chronological. - #[arg(long)] - tree: bool, - #[command(flatten)] - ctx: MsgCtx, - }, -} - -#[derive(Subcommand)] -enum EventCmd { - /// Emit one producer-identified event into a declared agent stream. - Emit { - /// Owning agent: a bus address (`.
`) or a bare local address. - #[arg(required_unless_present = "recipient_id", conflicts_with = "recipient_id")] - recipient: Option, - /// Owning agent by exact immutable agent ID (R24). - #[arg(long = "recipient-id")] - recipient_id: Option, - /// Declared stream name. - #[arg(long)] - stream: String, - /// Stable producer-supplied event identity. - #[arg(long = "event-id")] - event_id: String, - /// Producer grouping key used by --supersede. - #[arg(long)] - key: Option, - /// Archive the unread predecessor for the same key, or the stream-wide head without --key. - #[arg(long)] - supersede: bool, - /// One-line wake-time summary. - #[arg(long)] - subject: Option, - /// Event body. Read from stdin when omitted. - #[arg(short = 'm', long = "message")] - body: Option, - /// Emit the stable machine receipt. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, -} +mod cli; -#[derive(Subcommand)] -enum StreamCmd { - /// Add a stream to your declaration, optionally with a supervised adapter launch. - Add { - name: String, - /// Exact target agent; defaults to --as / $ST_AGENT. - #[arg(long)] - agent: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. - #[arg(long = "agent-id", conflicts_with = "agent")] - agent_id: Option, - /// Adapter command run under `sh -c`; omit both launch forms for external ingress. - #[arg(long, conflicts_with = "adapter_argv")] - command: Option, - /// Direct adapter argv after `--`. Element 0 is the program; values are preserved exactly. - #[arg(trailing_var_arg = true, allow_hyphen_values = true)] - adapter_argv: Vec, - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Remove a stream from your declaration. - Rm { - name: String, - /// Exact target agent; defaults to --as / $ST_AGENT. - #[arg(long)] - agent: Option, - /// Exact immutable agent ID (R24). Mutually exclusive with `--agent`. - #[arg(long = "agent-id", conflicts_with = "agent")] - agent_id: Option, - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, -} - -#[derive(Subcommand)] -enum RequestCmd { - /// Publish one idempotent JSON request from a declared service principal to an agent. - Send { - /// Recipient agent: a bus id (`.`) or a local bare identity. - to: String, - #[arg(long = "idempotency-key")] - idempotency_key: String, - /// Typed request tag as `key=value` (repeatable). - #[arg(long = "tag")] - tags: Vec, - /// JSON body. Read from stdin when omitted. - #[arg(short = 'm', long = "message")] - body: Option, - /// Emit the machine receipt as JSON. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Decode one typed request from an agent's inbox. - Read { - request_filename: String, - /// Emit the request envelope as JSON. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Reply once to a typed request in an agent's inbox. - Reply { - request_filename: String, - /// Typed reply tag as `key=value` (repeatable). - #[arg(long = "tag")] - tags: Vec, - /// JSON body. Read from stdin when omitted. - #[arg(short = 'm', long = "message")] - body: Option, - /// Emit the machine receipt as JSON. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, - /// Observe the typed reply for one previously published request. - Status { - #[arg(long = "idempotency-key")] - idempotency_key: String, - /// Emit the tagged status union as JSON. - #[arg(long)] - json: bool, - #[command(flatten)] - ctx: MsgCtx, - }, -} +use cli::*; fn main() -> Result<()> { let Cli { diff --git a/src/resync.rs b/src/resync.rs index 2be9f9ff..9b1ec90e 100644 --- a/src/resync.rs +++ b/src/resync.rs @@ -10,7 +10,7 @@ //! only a content transition emits. use std::collections::{BTreeMap, BTreeSet}; -use std::path::{Component, Path, PathBuf}; +use std::path::{Path, PathBuf}; use std::sync::Arc; use std::sync::mpsc::{Receiver, RecvTimeoutError, Sender, channel}; use std::thread::JoinHandle; @@ -19,479 +19,16 @@ use std::time::{Duration, Instant}; use notify::Watcher as _; use serde::Serialize; use sha2::{Digest as _, Sha256}; -use agent_spec::profile::{ - ProfileClass, ResourceProfileRefresh, ResourceProfileRegistry, -}; -use agent_spec::spec::{AgentSpec, Resource, decode_percent_path}; +use agent_spec::profile::ResourceProfileRegistry; +use agent_spec::spec::AgentSpec; -use crate::resource_profile::{MAX_FACTS, MAX_FACT_KEY_BYTES, ResourceFact}; +use crate::resource_profile::ResourceFact; /// The reserved stream used only by the supervisor's crate-internal resync publisher. pub const RESYNC_STREAM: &str = "resync"; -/// Provisional coalescing windows (`RESYNC-T02`): tuned by observed notification volume. -const IMMEDIATE_WINDOW: Duration = Duration::from_millis(500); -const COALESCED_WINDOW: Duration = Duration::from_secs(5); - -/// How a carrier notifies (`RESYNC-R04`). Silent carriers never reach the watch set: -/// [`classify`] excludes them, so nothing about them is observed or emitted. -#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] -pub enum CarrierClass { - Immediate, - Coalesced, -} - -impl CarrierClass { - fn window(self) -> Duration { - match self { - CarrierClass::Immediate => IMMEDIATE_WINDOW, - CarrierClass::Coalesced => COALESCED_WINDOW, - } - } -} -/// Catalog-observable resync coverage for one declared Resource binding. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ResyncCoverage { - Immediate, - Coalesced, - Silent, - Unsupported, - Inactive, -} - -impl ResyncCoverage { - pub const fn as_str(self) -> &'static str { - match self { - Self::Immediate => "immediate", - Self::Coalesced => "coalesced", - Self::Silent => "silent", - Self::Unsupported => "unsupported", - Self::Inactive => "inactive", - } - } - - fn carrier_class(self) -> Option { - match self { - Self::Immediate => Some(CarrierClass::Immediate), - Self::Coalesced => Some(CarrierClass::Coalesced), - Self::Silent | Self::Unsupported | Self::Inactive => None, - } - } -} - - -/// One watchable local carrier: binding label, absolute path, notification class, and an optional -/// host root that must confine every read of a resolver-selected path. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct WatchableCarrier { - pub label: String, - pub path: PathBuf, - pub class: CarrierClass, - pub containment_root: Option, -} -#[derive(Debug, Clone, PartialEq, Eq)] -struct DeclarationSummary { - bindings: BTreeMap, - complete: bool, -} - -fn declaration_summary(spec: &AgentSpec) -> DeclarationSummary { - let mut bindings = spec - .resources - .iter() - .map(|resource| { - ( - resource.name().to_owned(), - declaration_resource_digest(resource), - ) - }) - .collect::>(); - bindings.sort_by(|left, right| left.0.cmp(&right.0)); - let complete = bindings.len() <= MAX_FACTS - && bindings - .iter() - .all(|(label, _)| label.len() <= MAX_FACT_KEY_BYTES); - DeclarationSummary { - bindings: bindings.into_iter().take(MAX_FACTS).collect(), - complete, - } -} - -fn declaration_resource_digest(resource: &Resource) -> String { - let mut digest = Sha256::new(); - digest.update(b"st2.resync.resource-declaration.v1\0"); - update_digest_field(&mut digest, resource.uri().as_bytes()); - update_digest_field(&mut digest, resource.reason().as_bytes()); - match resource.inactive_reason() { - Some(reason) => { - digest.update([1]); - update_digest_field(&mut digest, reason.as_bytes()); - } - None => digest.update([0]), - } - let selector = serde_json::to_vec(&resource.selector()) - .expect("a parsed JSON selector always serializes"); - update_digest_field(&mut digest, &selector); - format!("{:x}", digest.finalize()) -} - -fn update_digest_field(digest: &mut Sha256, value: &[u8]) { - let length = u64::try_from(value.len()).expect("declaration fields fit in u64"); - digest.update(length.to_be_bytes()); - digest.update(value); -} - -/// The watchable carriers of one agent, keyed by its declaration path with current routing IDs. -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct AgentWatchSet { - pub declaration_path: PathBuf, - pub bus_id: String, - pub seat_id: Option, - pub carriers: Vec, - declaration_summary: Option, -} - -/// Resolve one spec's watch set: the declaration file plus every active resource binding whose -/// URI denotes a local file (`RESYNC-R01`) — directly (`file://`, catalog-relative) or through a -/// declared resource profile for a scheme URI. Bindings with an inactive reason are skipped; -/// schemes without a local denotation and silent carriers are simply absent. A failing profile -/// resolver is contained: its binding is skipped, the rest of the set survives. -pub fn watch_set_for( - spec: &AgentSpec, - this_host: &str, - profiles: &ResourceProfileRegistry, -) -> AgentWatchSet { - watch_set_for_in_catalog(spec, std::slice::from_ref(spec), this_host, profiles) -} - -/// [`watch_set_for`] with the catalog view a `notify-chain` profile needs to reach the carriers -/// this agent's `supervisor` ancestors declare. Without the other specs, chain carriers cannot be -/// resolved and only the agent's own carriers are produced. -pub fn watch_set_for_in_catalog( - spec: &AgentSpec, - specs: &[AgentSpec], - this_host: &str, - profiles: &ResourceProfileRegistry, -) -> AgentWatchSet { - let refresh = profiles.begin_refresh(); - resolve_watch_set(spec, specs, this_host, &refresh).0 -} - -fn resolve_watch_set( - spec: &AgentSpec, - specs: &[AgentSpec], - this_host: &str, - profiles: &ResourceProfileRefresh<'_>, -) -> (AgentWatchSet, Vec) { - let declaration_path = lexical_clean(&spec.path); - let agent_dir = declaration_path.parent().unwrap_or(Path::new(".")); - let mut carriers = vec![WatchableCarrier { - label: "declaration".to_owned(), - path: declaration_path.clone(), - class: CarrierClass::Immediate, - containment_root: None, - }]; - let mut diagnostics = Vec::new(); - for resource in &spec.resources { - if resource.inactive_reason().is_some() { - continue; - } - // Silent profiles carry no observable transition, so never compile or execute their - // untrusted resolver merely to discard its result. - let registered_profile = resource - .uri() - .split_once(':') - .and_then(|(scheme, _)| profiles.get(scheme)); - if registered_profile.is_some_and(|profile| profile.class() == ProfileClass::Silent) { - continue; - } - // Declared profile schemes resolve through their wasm module; the declared class governs - // notification instead of the local-path defaults. - match profiles.try_resolve(agent_dir, resource.uri()) { - Ok(Some(resolution)) => { - let Some(class) = carrier_class(resolution.class) else { - continue; - }; - carriers.push(WatchableCarrier { - label: resource.name().to_owned(), - path: resolution.path, - class, - containment_root: Some(resolution.containment_root), - }); - continue; - } - Ok(None) => {} - Err(error) => { - diagnostics.push(format!( - "resync profile for {} resource '{}': {error}; binding is unwatchable", - spec.bus_id(this_host), - resource.name() - )); - continue; - } - } - let Some(class) = resource_coverage(agent_dir, resource).carrier_class() else { - continue; - }; - let path = resolve_local_path(agent_dir, resource.uri()) - .expect("watchable coverage must have a local path"); - carriers.push(WatchableCarrier { - label: resource.name().to_owned(), - path, - class, - containment_root: None, - }); - } - append_chain_carriers( - spec, - specs, - this_host, - profiles, - &mut carriers, - &mut diagnostics, - ); - // The supervisor's resolved logical host — not the OS hostname — decides the bus id, so an - // agent supervised under `st2 up --host ` without an explicit declaration host still - // produces a recipient `resolve_stream` can resolve. - ( - AgentWatchSet { - declaration_path, - bus_id: spec.bus_id(this_host), - seat_id: spec.tasks.iter().find(|task| task.name == "agent").map(|task| { - task.id - .clone() - .unwrap_or_else(|| format!("{}.{}", spec.bus_id(this_host), task.name)) - }), - carriers, - declaration_summary: Some(declaration_summary(spec)), - }, - diagnostics, - ) -} - -/// A URI denotes a local file when it is an absolute, authority-free `file://` URI or a -/// scheme-less catalog-relative path resolved against the agent directory. URI scheme syntax is -/// parsed before the `file` scheme name is matched ASCII-case-insensitively, as required by RFC -/// 3986. Unsupported file URI authorities, query/fragment components, -/// malformed escapes, encoded path separators, and encoded parent components have no local -/// denotation. -/// Carriers this agent's `supervisor` ancestors declare through a `notify-chain` profile. -/// -/// A profile whose layers compose along the supervisor edge leaves every descendant's effective -/// view dependent on carriers the descendant does not own. Resync notifies a carrier's owner, so -/// without this the descendant is never told its view changed. -/// -/// The walk deliberately reuses each ancestor's OWN declared URI rather than synthesizing one: -/// st2 does not own any profile's URI grammar, and a resolver is free to ignore the authority -/// component entirely, so a synthesized subject would be a guess. Resolving the ancestor's -/// declaration against the ancestor's directory is the identical call the ancestor's own -/// subscription makes, which is what keeps containment unchanged — the guest is still only ever -/// asked to resolve one agent's URI against that agent's own directory. -/// -/// Matching is by profile scheme, never by binding label: labels are agent-local and replaceable, -/// so keying on them would silently drop a layer whose owner renamed its binding. -fn append_chain_carriers( - spec: &AgentSpec, - specs: &[AgentSpec], - this_host: &str, - profiles: &ResourceProfileRefresh<'_>, - carriers: &mut Vec, - diagnostics: &mut Vec, -) { - let chain_schemes: Vec<&str> = spec - .resources - .iter() - .filter(|resource| resource.inactive_reason().is_none()) - .filter_map(|resource| resource.uri().split_once(':').map(|(scheme, _)| scheme)) - .filter(|scheme| profiles.get(scheme).is_some_and(|p| p.notify_chain())) - .collect(); - if chain_schemes.is_empty() { - return; - } - - let ancestors = match crate::supervisor_chain::ancestors(specs, spec, this_host) { - Ok(ancestors) => ancestors, - Err(error) => { - diagnostics.push(format!( - "resync notify-chain for {}: supervisor chain is unwalkable ({error:?}); \ - ancestor carriers are unwatchable", - spec.bus_id(this_host) - )); - return; - } - }; - - for ancestor in ancestors { - // Skip and continue, never sever: a retired ancestor contributes no layer, but its own - // ancestors still do. `is_retired` normalizes both declaration spellings. - if ancestor.desired_state.is_retired() { - continue; - } - let ancestor_declaration = lexical_clean(&ancestor.path); - let ancestor_dir = ancestor_declaration.parent().unwrap_or(Path::new(".")); - let ancestor_bus_id = ancestor.bus_id(this_host); - for resource in &ancestor.resources { - if resource.inactive_reason().is_some() { - continue; - } - let Some((scheme, _)) = resource.uri().split_once(':') else { - continue; - }; - if !chain_schemes.contains(&scheme) { - continue; - } - match profiles.try_resolve(ancestor_dir, resource.uri()) { - Ok(Some(resolution)) => { - let Some(class) = carrier_class(resolution.class) else { - continue; - }; - carriers.push(WatchableCarrier { - // Qualifying by owner keeps each ancestor's layer on its own supersession - // key, so a burst on one ancestor cannot collapse another's event. - label: format!("{}@{ancestor_bus_id}", resource.name()), - path: resolution.path, - class, - containment_root: Some(resolution.containment_root), - }); - } - Ok(None) => {} - Err(error) => diagnostics.push(format!( - "resync notify-chain for {}: ancestor {ancestor_bus_id} resource '{}': \ - {error}; that ancestor layer is unwatchable", - spec.bus_id(this_host), - resource.name() - )), - } - } - } -} - -fn resolve_local_path(agent_dir: &Path, uri: &str) -> Option { - if let Some((scheme, scheme_specific)) = uri.split_once(':').filter(|(scheme, _)| { - !scheme.is_empty() - && !scheme.contains('/') - && scheme - .chars() - .all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.')) - }) { - if !scheme.eq_ignore_ascii_case("file") { - return None; - } - let encoded_path = scheme_specific.strip_prefix("//")?; - if !encoded_path.starts_with('/') - || encoded_path.starts_with("//") - || encoded_path.contains(['?', '#']) - { - return None; - } - let path = PathBuf::from(decode_percent_path(encoded_path).ok()?); - return path.is_absolute().then(|| lexical_clean(&path)); - } - let path = PathBuf::from(decode_percent_path(uri).ok()?); - Some(lexical_clean(&agent_dir.join(path))) -} -/// Resolve the externally visible resync coverage for one Resource binding. -pub fn resource_coverage(agent_dir: &Path, resource: &agent_spec::spec::Resource) -> ResyncCoverage { - if resource.inactive_reason().is_some() { - return ResyncCoverage::Inactive; - } - let Some(path) = resolve_local_path(agent_dir, resource.uri()) else { - return ResyncCoverage::Unsupported; - }; - match classify(agent_dir, resource.name(), &path) { - Some(CarrierClass::Immediate) => ResyncCoverage::Immediate, - Some(CarrierClass::Coalesced) => ResyncCoverage::Coalesced, - None => ResyncCoverage::Silent, - } -} - - -/// Remove `.` and `..` components lexically. This deliberately does not inspect the filesystem: -/// classification follows the authored path structure without resolving symlinks. -fn lexical_clean(path: &Path) -> PathBuf { - let mut clean = PathBuf::new(); - for component in path.components() { - match component { - Component::Prefix(_) | Component::RootDir | Component::Normal(_) => { - clean.push(component.as_os_str()); - } - Component::CurDir => {} - Component::ParentDir => { - let can_pop = matches!(clean.components().next_back(), Some(Component::Normal(_))); - if can_pop { - clean.pop(); - } else if !clean.has_root() { - clean.push(component.as_os_str()); - } - } - } - } - clean -} - -/// Class defaults for carriers resolved WITHOUT a declared profile (`RESYNC-R04`): goal carriers -/// are immediate; stores the agent itself authors are silent (None); everything else is coalesced. -/// The declaration carrier is immediate by construction in [`watch_set_for`]. Profile-resolved -/// carriers skip this sniffing entirely — their class is what the catalog declares. -fn classify(agent_dir: &Path, binding_name: &str, normalized_path: &Path) -> Option { - let agent_relative = normalized_path.strip_prefix(agent_dir).ok(); - let authored_store = agent_relative.is_some_and(|rel| { - rel.starts_with("resources/context") - || rel.starts_with("resources/decisions") - || rel.starts_with("resources/friction") - }); - if authored_store { - return None; - } - let goal = - binding_name == "goal" || normalized_path.file_name().is_some_and(|n| n == "goal.md"); - Some(if goal { - CarrierClass::Immediate - } else { - CarrierClass::Coalesced - }) -} - -/// A declared profile class maps onto carrier notification: silent profiles are excluded from -/// the watch set exactly like sniffed agent-authored stores. -fn carrier_class(class: ProfileClass) -> Option { - match class { - ProfileClass::Immediate => Some(CarrierClass::Immediate), - ProfileClass::Coalesced => Some(CarrierClass::Coalesced), - ProfileClass::Silent => None, - } -} - -/// Resolve coverage with the catalog's declared profile registry. Silent profiles report their -/// declared class without executing a guest; other registered schemes are watchable only when -/// their resolver succeeds. -pub fn resource_coverage_with_profiles( - agent_dir: &Path, - resource: &agent_spec::spec::Resource, - profiles: &ResourceProfileRefresh<'_>, -) -> ResyncCoverage { - if resource.inactive_reason().is_some() { - return ResyncCoverage::Inactive; - } - let registered = resource - .uri() - .split_once(':') - .and_then(|(scheme, _)| profiles.get(scheme)); - let Some(profile) = registered else { - return resource_coverage(agent_dir, resource); - }; - if profile.class() == ProfileClass::Silent { - return ResyncCoverage::Silent; - } - match profiles.try_resolve(agent_dir, resource.uri()) { - Ok(Some(resolution)) => match resolution.class { - ProfileClass::Immediate => ResyncCoverage::Immediate, - ProfileClass::Coalesced => ResyncCoverage::Coalesced, - ProfileClass::Silent => ResyncCoverage::Silent, - }, - Ok(None) | Err(_) => ResyncCoverage::Unsupported, - } -} +mod watch_set; +pub use watch_set::*; // ---- Supervisor side --------------------------------------------------------------------------- @@ -1895,176 +1432,8 @@ fn emit_resync( -fn read_state(path: &Path, containment_root: Option<&Path>) -> std::io::Result { - match containment_root { - Some(root) => read_confined(path, root), - None => read_regular(path), - } -} - -fn diagnose_read_error(path: &Path, error: &std::io::Error) { - eprintln!( - "st2: resync read for '{}' failed transiently; retrying: {error}", - path.display() - ); -} - -fn hash_reader(mut file: std::fs::File) -> std::io::Result { - let mut digest = Sha256::new(); - let mut buffer = [0_u8; 64 * 1024]; - loop { - let read = std::io::Read::read(&mut file, &mut buffer)?; - if read == 0 { - break; - } - digest.update(&buffer[..read]); - } - Ok(format!("{:x}", digest.finalize())) -} - -#[cfg(unix)] -fn classify_open_error(error: std::io::Error) -> std::io::Result { - match error.raw_os_error() { - Some(libc::ENOENT | libc::ENOTDIR | libc::ELOOP) => Ok(CarrierState::Missing), - _ => Err(error), - } -} - -#[cfg(not(unix))] -fn classify_open_error(error: std::io::Error) -> std::io::Result { - if error.kind() == std::io::ErrorKind::NotFound { - Ok(CarrierState::Missing) - } else { - Err(error) - } -} - -#[cfg(unix)] -fn read_regular(path: &Path) -> std::io::Result { - use std::os::unix::fs::OpenOptionsExt as _; - - let file = match std::fs::OpenOptions::new() - .read(true) - .custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC | libc::O_NONBLOCK) - .open(path) - { - Ok(file) => file, - Err(error) => return classify_open_error(error), - }; - if !file.metadata()?.file_type().is_file() { - return Ok(CarrierState::Missing); - } - hash_reader(file).map(CarrierState::Present) -} - -#[cfg(not(unix))] -fn read_regular(path: &Path) -> std::io::Result { - let file = match std::fs::File::open(path) { - Ok(file) => file, - Err(error) => return classify_open_error(error), - }; - if !file.metadata()?.file_type().is_file() { - return Ok(CarrierState::Missing); - } - hash_reader(file).map(CarrierState::Present) -} - -#[cfg(unix)] -fn read_confined(path: &Path, root: &Path) -> std::io::Result { - use std::ffi::CString; - use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; - use std::os::unix::ffi::OsStrExt as _; - - let invalid_path = || { - std::io::Error::new( - std::io::ErrorKind::InvalidInput, - "carrier path is outside its confinement root or has an unsafe component", - ) - }; - let relative = path.strip_prefix(root).map_err(|_| invalid_path())?; - let mut components = relative.components().peekable(); - if components.peek().is_none() { - return Ok(CarrierState::Missing); - } - - // Open the confinement root component-by-component from the filesystem root. `O_NOFOLLOW` - // on one full pathname protects only its final component; descriptor-relative traversal - // protects every ancestor from symlink replacement as well. - let mut root_components = root.components(); - if root_components.next() != Some(std::path::Component::RootDir) { - return Err(invalid_path()); - } - let slash = CString::new("/").map_err(|_| invalid_path())?; - // SAFETY: `slash` is NUL-terminated and the returned descriptor is checked before ownership. - let filesystem_root = unsafe { - libc::open( - slash.as_ptr(), - libc::O_RDONLY | libc::O_DIRECTORY | libc::O_CLOEXEC | libc::O_NOFOLLOW, - ) - }; - if filesystem_root < 0 { - return classify_open_error(std::io::Error::last_os_error()); - } - // SAFETY: `filesystem_root` is newly owned after the non-negative check. - let mut directory = unsafe { OwnedFd::from_raw_fd(filesystem_root) }; - for component in root_components { - let std::path::Component::Normal(name) = component else { - return Err(invalid_path()); - }; - let name = CString::new(name.as_bytes()).map_err(|_| invalid_path())?; - let flags = libc::O_RDONLY - | libc::O_DIRECTORY - | libc::O_CLOEXEC - | libc::O_NOFOLLOW - | libc::O_NONBLOCK; - // SAFETY: the live directory descriptor and NUL-terminated component are valid. - let opened = unsafe { libc::openat(directory.as_raw_fd(), name.as_ptr(), flags) }; - if opened < 0 { - return classify_open_error(std::io::Error::last_os_error()); - } - // SAFETY: `opened` is newly owned after the non-negative check. - directory = unsafe { OwnedFd::from_raw_fd(opened) }; - } - - while let Some(component) = components.next() { - let std::path::Component::Normal(name) = component else { - return Err(invalid_path()); - }; - let name = CString::new(name.as_bytes()).map_err(|_| invalid_path())?; - let last = components.peek().is_none(); - let flags = libc::O_RDONLY - | libc::O_CLOEXEC - | libc::O_NOFOLLOW - | libc::O_NONBLOCK - | if last { 0 } else { libc::O_DIRECTORY }; - // SAFETY: both the live directory descriptor and NUL-terminated component are valid; - // `O_NOFOLLOW` makes each lookup fail closed if that component is replaced by a symlink. - let opened = unsafe { libc::openat(directory.as_raw_fd(), name.as_ptr(), flags) }; - if opened < 0 { - return classify_open_error(std::io::Error::last_os_error()); - } - // SAFETY: `opened` is a newly-owned descriptor after the non-negative check above. - let opened = unsafe { OwnedFd::from_raw_fd(opened) }; - if last { - let file = std::fs::File::from(opened); - if !file.metadata()?.file_type().is_file() { - return Ok(CarrierState::Missing); - } - return hash_reader(file).map(CarrierState::Present); - } - directory = opened; - } - Ok(CarrierState::Missing) -} - -#[cfg(not(unix))] -fn read_confined(_path: &Path, _root: &Path) -> std::io::Result { - // No std API can atomically enforce no-follow traversal. Fail closed on unsupported hosts. - Err(std::io::Error::new( - std::io::ErrorKind::Unsupported, - "descriptor-relative no-follow reads are unavailable", - )) -} +mod read; +use read::*; #[derive(Serialize)] #[serde(rename_all = "camelCase")] @@ -2090,2270 +1459,4 @@ fn render_body( .expect("validated resync facts always serialize") } #[cfg(test)] -mod tests { - use super::*; - - fn refresh_for(sets: Vec) -> WatchRefresh { - WatchRefresh { - sets, - malformed_declarations: BTreeSet::new(), - live_task_ids: BTreeSet::new(), - } - } - - #[test] - fn resync_subject_uses_the_shared_three_fact_and_96_scalar_renderer() { - let facts = vec![ - ResourceFact::transition("alpha", None::, Some("declared")).unwrap(), - ResourceFact::current("beta", "changed").unwrap(), - ResourceFact::transition("charlie", Some("declared"), None::).unwrap(), - ResourceFact::current("delta", "omitted").unwrap(), - ]; - let subject = crate::resource_profile_supervisor::resource_change_subject( - "declaration", - &facts, - &["declaration".to_owned()], - "content changed", - ); - assert_eq!( - subject, - "declaration · alpha=+declared; beta=changed; charlie=-declared [declaration]" - ); - assert!(subject.chars().count() <= 96); - assert!(!subject.contains("delta")); - } - - fn owner_incarnation(seed: u64) -> crate::event::StreamOwnerIncarnation { - crate::event::StreamOwnerIncarnation::for_test(seed, seed + 1, 42, seed + 2) - } - - fn discover(catalog: &Path) -> AgentSpec { - let found = crate::discover_strict(catalog); - eprintln!("discovery errors: {:?}", found.errors); - found.specs.into_iter().next().unwrap() - } - - fn resync_inbox_event(agent_dir: &Path) -> String { - std::fs::read_dir(agent_dir.join("resources/inbox")) - .unwrap() - .filter_map(Result::ok) - .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) - .find(|event| event.lines().any(|line| line == "stream: resync")) - .expect("current resync inbox event") - } - fn resync_inbox_events(agent_dir: &Path) -> Vec { - std::fs::read_dir(agent_dir.join("resources/inbox")) - .unwrap() - .filter_map(Result::ok) - .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) - .filter(|event| event.lines().any(|line| line == "stream: resync")) - .collect() - } - - fn event_body(event: &str) -> serde_json::Value { - let body = event - .lines() - .rev() - .find(|line| line.starts_with('{')) - .expect("JSON resync body"); - serde_json::from_str(body).expect("valid JSON resync body") - } - - fn event_field(event: &str, field: &str) -> String { - if let Some(value) = event - .lines() - .find_map(|line| line.strip_prefix(&format!("{field}: "))) - { - return value.to_owned(); - } - let body = event_body(event); - if matches!(field, "old" | "new") { - let digest = body["facts"] - .as_array() - .and_then(|facts| facts.iter().find(|fact| fact["key"] == "digest")) - .expect("digest transition fact"); - let value = if field == "old" { - &digest["before"] - } else { - &digest["after"] - }; - return value.as_str().expect("digest fact value").to_owned(); - } - body[field] - .as_str() - .unwrap_or_else(|| panic!("missing {field} in event")) - .to_owned() - } - #[test] - fn declaration_facts_are_ordered_added_removed_and_semantically_changed_labels() { - let root = tempfile::tempdir().unwrap(); - let declaration = root.path().join("agent.kdl"); - std::fs::write( - &declaration, - r#"agent "worker" { - host "host" - command "true" - resource "inactive" uri="file:///inactive" reason="kept" - resource "reason" uri="file:///reason" reason="before" - resource "removed" uri="file:///removed" reason="gone" - resource "uri" uri="file:///before" reason="same" -}"#, - ) - .unwrap(); - let before = declaration_summary(&discover(root.path())); - std::fs::write( - &declaration, - r#"agent "worker" { - host "host" - command "true" - resource "added" uri="file:///added" reason="new" - resource "inactive" uri="file:///inactive" reason="kept" inactive-reason="paused" - resource "reason" uri="file:///reason" reason="after" - resource "uri" uri="file:///after" reason="same" -}"#, - ) - .unwrap(); - let after = declaration_summary(&discover(root.path())); - let facts = declaration_transition_facts( - Some(&before), - Some(&after), - &CarrierState::Present("before-digest".to_owned()), - &CarrierState::Present("after-digest".to_owned()), - ); - assert_eq!( - facts.iter().map(ResourceFact::key).collect::>(), - vec!["added", "inactive", "reason", "removed", "uri"] - ); - assert_eq!(facts[0].before(), Some(None)); - assert_eq!(facts[0].after(), Some(Some("declared"))); - for index in [1, 2, 4] { - assert_eq!(facts[index].before(), None); - assert_eq!(facts[index].after(), Some(Some("changed"))); - } - assert_eq!(facts[3].before(), Some(Some("declared"))); - assert_eq!(facts[3].after(), Some(None)); - } - - #[test] - fn declaration_parse_failure_retains_a_digest_fact_for_later_delivery() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/host/worker"); - std::fs::create_dir_all(&agent_dir).unwrap(); - let declaration = agent_dir.join("agent.kdl"); - let valid = r#"agent "worker" { - host "host" - command "true" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#; - std::fs::write(&declaration, valid).unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let set = watch_set_for(&discover(root.path()), "host", &Default::default()); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.apply_watch_sets(refresh_for(vec![set])); - std::fs::write(&declaration, "not an Agent Spec").unwrap(); - worker.flush_path_publishing(&declaration, None); - let pending = worker.carriers[&declaration][0] - .pending_transition - .as_ref() - .expect("malformed declaration keeps its digest fallback"); - assert_eq!(pending.facts.len(), 1); - assert_eq!(pending.facts[0].key(), "digest"); - assert_eq!(pending.topics, ["declaration"]); - assert_eq!(event_body(&pending.body)["facts"].as_array().unwrap().len(), 1); - std::fs::write(&declaration, valid).unwrap(); - worker.flush_path_publishing(&declaration, None); - let delivered = resync_inbox_event(&agent_dir); - let body = event_body(&delivered); - assert_eq!(body["binding"], "declaration"); - assert_eq!(body["topics"], serde_json::json!(["declaration"])); - assert_eq!(body["facts"][0]["key"], "digest"); - assert!(!delivered.contains("file:///")); - assert!(!delivered.contains("Mission.")); - } - - - #[test] - fn watch_set_covers_declaration_and_local_bindings_only() { - let tmp = tempfile::tempdir().unwrap(); - let dir = tmp.path().join("agents/hetz/worker"); - std::fs::create_dir_all(&dir).unwrap(); - std::fs::write( - dir.join("agent.kdl"), - r#"agent "worker" { - host "hetz" - command "true" - resource "goal" uri="resources/goal.md" reason="Mission." - resource "journal" uri="resources/context/journal.md" reason="Memory." - resource "issue" uri="github-issue://org/repo/41" reason="Task." - resource "old" uri="resources/old.md" reason="History." inactive-reason="No longer used." -}"#, - ) - .unwrap(); - let spec = discover(tmp.path()); - let set = watch_set_for(&spec, "hetz", &Default::default()); - assert_eq!(set.bus_id, "hetz.worker"); - let mut labels: Vec<&str> = set.carriers.iter().map(|c| c.label.as_str()).collect(); - labels.sort(); - assert_eq!(labels, vec!["declaration", "goal"]); - let goal = set.carriers.iter().find(|c| c.label == "goal").unwrap(); - assert_eq!(goal.class, CarrierClass::Immediate); - assert_eq!(goal.path, dir.join("resources/goal.md")); - let coverage = spec - .resources - .iter() - .map(|resource| (resource.name(), resource_coverage(&dir, resource))) - .collect::>(); - assert_eq!(coverage["goal"], ResyncCoverage::Immediate); - assert_eq!(coverage["journal"], ResyncCoverage::Silent); - assert_eq!(coverage["issue"], ResyncCoverage::Unsupported); - assert_eq!(coverage["old"], ResyncCoverage::Inactive); - } - - #[test] - fn bus_id_uses_the_supervisor_host_not_the_os_hostname() { - let tmp = tempfile::tempdir().unwrap(); - // A root-level declaration file supplies neither content nor path host: host stays - // None, so the supervisor's logical alias must decide the recipient. - std::fs::write( - tmp.path().join("worker.kdl"), - r#"agent "worker" { - command "true" -}"#, - ) - .unwrap(); - let spec = discover(tmp.path()); - assert_eq!( - watch_set_for(&spec, "alias", &Default::default()).bus_id, - "alias.worker" - ); - assert_eq!( - watch_set_for(&spec, "other", &Default::default()).bus_id, - "other.worker" - ); - } - - #[test] - fn lexical_paths_drive_store_classification_and_containment() { - let agent_dir = Path::new("/catalog/agents/host/worker"); - - let silent = resolve_local_path( - agent_dir, - "file:///catalog/agents/host/worker/resources/tmp/../context/./journal.md", - ) - .unwrap(); - assert_eq!( - silent, - agent_dir.join("resources/context/journal.md"), - "file URI dot segments are removed before classification" - ); - assert_eq!(classify(agent_dir, "journal", &silent), None); - - let goal = resolve_local_path(agent_dir, "resources/context/.././goal.md").unwrap(); - assert_eq!(goal, agent_dir.join("resources/goal.md")); - assert_eq!( - classify(agent_dir, "notes", &goal), - Some(CarrierClass::Immediate) - ); - - let escaped = - resolve_local_path(agent_dir, "resources/context/../../outside/notes.md").unwrap(); - assert_eq!(escaped, agent_dir.join("outside/notes.md")); - assert_eq!( - classify(agent_dir, "notes", &escaped), - Some(CarrierClass::Coalesced), - "a lexical escape from an authored store is not silent" - ); - assert_eq!( - classify( - agent_dir, - "notes", - Path::new("/catalog/agents/host/worker-copy/resources/context/notes.md"), - ), - Some(CarrierClass::Coalesced), - "path-prefix siblings are not contained by the agent directory" - ); - } - - #[cfg(unix)] - #[test] - fn classification_does_not_follow_symlinks() { - let tmp = tempfile::tempdir().unwrap(); - let agent_dir = tmp.path().join("agent"); - std::fs::create_dir_all(agent_dir.join("resources/context")).unwrap(); - std::os::unix::fs::symlink( - agent_dir.join("resources/context"), - agent_dir.join("resources/linked"), - ) - .unwrap(); - - let linked = resolve_local_path(&agent_dir, "resources/linked/journal.md").unwrap(); - assert_eq!( - classify(&agent_dir, "journal", &linked), - Some(CarrierClass::Coalesced), - "classification is lexical and must not canonicalize through the symlink" - ); - } - - #[test] - fn shared_path_refresh_preserves_every_subscription_state() { - let shared = PathBuf::from("/shared/resource.md"); - let previous = BTreeMap::from([( - shared.clone(), - vec![ - Entry { - bus_id: "host.alpha".to_owned(), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("alpha-before".to_owned())), - declaration_summary: None, - occurrence_sequence: 4, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }, - Entry { - bus_id: "host.beta".to_owned(), - seat_id: None, - label: "spec".to_owned(), - class: CarrierClass::Coalesced, - containment_root: None, - state: Some(CarrierState::Present("beta-before".to_owned())), - declaration_summary: None, - occurrence_sequence: 9, - in_flight: false, - parked: false, - dirty: true, - pending_transition: None, - }, - ], - )]); - let sets = vec![ - AgentWatchSet { - declaration_path: PathBuf::from("/catalog/alpha/agent.kdl"), - bus_id: "host.alpha".to_owned(), - seat_id: None, - carriers: vec![WatchableCarrier { - label: "goal".to_owned(), - path: shared.clone(), - class: CarrierClass::Immediate, - containment_root: None, - }], - declaration_summary: None, - }, - AgentWatchSet { - declaration_path: PathBuf::from("/catalog/beta/agent.kdl"), - bus_id: "host.beta".to_owned(), - seat_id: None, - carriers: vec![WatchableCarrier { - label: "spec".to_owned(), - path: shared.clone(), - class: CarrierClass::Coalesced, - containment_root: None, - }], - declaration_summary: None, - }, - ]; - - let rebuilt = rebuild_carriers(previous, refresh_for(sets), &BTreeMap::new(), &mut BTreeMap::new()); - let entries = rebuilt.get(&shared).expect("shared path remains watched"); - assert_eq!(entries.len(), 2); - for (bus_id, digest) in [ - ("host.alpha", "alpha-before"), - ("host.beta", "beta-before"), - ] { - let entry = entries - .iter() - .find(|entry| entry.bus_id == bus_id) - .expect("subscriber remains present"); - assert_eq!( - entry.state, - Some(CarrierState::Present(digest.to_owned())) - ); - assert!(entry.dirty, "pending mutation remains pending for {bus_id}"); - } - } - - #[test] - fn retained_subscription_uses_current_seat_path_and_class() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/alias/worker"); - std::fs::create_dir_all(agent_dir.join("resources")).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "alias" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let goal = agent_dir.join("resources/goal.md"); - std::fs::write(&goal, "current bytes").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "alias").unwrap(); - - let mut current = watch_set_for( - &discover(root.path()), - "alias", - &ResourceProfileRegistry::empty(), - ); - current.seat_id = Some("current-seat".to_owned()); - let old_path = agent_dir.join("resources/old-goal.md"); - let previous = BTreeMap::from([( - old_path.clone(), - vec![Entry { - bus_id: "alias.worker".to_owned(), - seat_id: Some("stale-seat".to_owned()), - label: "goal".to_owned(), - class: CarrierClass::Coalesced, - containment_root: None, - state: Some(CarrierState::Present("old-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 3, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }], - )]); - - let rebuilt = rebuild_carriers(previous, refresh_for(vec![current]), &BTreeMap::new(), &mut BTreeMap::new()); - assert!(!rebuilt.contains_key(&old_path)); - let entry = rebuilt[&goal] - .iter() - .find(|entry| entry.label == "goal") - .expect("the goal subscription remains pending at its current path"); - assert_eq!(entry.bus_id, "alias.worker"); - assert_eq!(entry.seat_id.as_deref(), Some("current-seat")); - assert_eq!(entry.class, CarrierClass::Immediate); - assert_eq!( - entry.state, - Some(CarrierState::Present("old-digest".to_owned())) - ); - assert!(entry.dirty); - - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "alias".to_owned(), - carriers: rebuilt, - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.flush_path_publishing(&goal, Some(CarrierClass::Immediate)); - - let entry = worker.carriers[&goal] - .iter() - .find(|entry| entry.label == "goal") - .unwrap(); - assert!(!entry.dirty, "the current recipient accepted the transition"); - assert!(entry.pending_transition.is_none()); - let events = std::fs::read_dir(agent_dir.join("resources/inbox")) - .unwrap() - .collect::, _>>() - .unwrap(); - assert_eq!(events.len(), 1, "the event routes to the current bus id"); - } - - #[test] - fn pending_retry_keeps_its_original_snapshot_across_path_rebinding() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/alias/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - let old_path = resources.join("old-goal.md"); - let current_path = resources.join("current-goal.md"); - std::fs::write(&old_path, "pending bytes").unwrap(); - std::fs::write(¤t_path, "current rebound bytes").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "alias").unwrap(); - let declaration = agent_dir.join("agent.kdl"); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "alias".to_owned(), - carriers: BTreeMap::from([( - old_path.clone(), - vec![Entry { - bus_id: "alias.worker".to_owned(), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("old-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - - worker.flush_path_publishing(&old_path, None); - let pending = worker.carriers[&old_path][0] - .pending_transition - .clone() - .expect("failed emit retains an immutable transition"); - std::fs::write( - &declaration, - r#"agent "worker" { - host "alias" - command "agent" - resource "goal" uri="resources/current-goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let current = - watch_set_for(&discover(root.path()), "alias", &ResourceProfileRegistry::empty()); - - worker.apply_watch_sets(refresh_for(vec![current])); - worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); - - let event = std::fs::read_dir(resources.join("inbox")) - .unwrap() - .map(|entry| std::fs::read_to_string(entry.unwrap().path()).unwrap()) - .find(|body| body.contains("stream: resync")) - .expect("the retry routes through the refreshed recipient"); - assert!(event.contains(&format!("event-id: {}", pending.event_id)), "{event}"); - assert!(event.contains(&pending.body), "{event}"); - let pending_body: serde_json::Value = serde_json::from_str(&pending.body).unwrap(); - assert_eq!(pending_body["binding"], "goal"); - assert_eq!(pending_body["facts"][0]["before"], "old-digest"); - assert!( - !pending.body.contains(¤t_path.display().to_string()), - "rebinding must not rewrite bytes reserved under the pending event identity" - ); - let entry = &worker.carriers[¤t_path][0]; - assert_eq!(entry.state.as_ref(), Some(&pending.new_state)); - assert!(entry.pending_transition.is_none()); - assert!( - entry.dirty, - "current rebound bytes are queued only after the pending snapshot completes" - ); - } - - #[test] - fn same_directory_path_rebinding_diffs_the_new_path_without_a_filesystem_event() { - let root = tempfile::tempdir().unwrap(); - let parent = root.path().join("resources"); - std::fs::create_dir_all(&parent).unwrap(); - let old_path = parent.join("old.md"); - let new_path = parent.join("new.md"); - std::fs::write(&old_path, "old bytes").unwrap(); - std::fs::write(&new_path, "new bytes").unwrap(); - let declaration = root.path().join("agent.kdl"); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([( - old_path.clone(), - vec![Entry { - bus_id: "host.worker".to_owned(), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: read_state(&old_path, None).ok(), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: false, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::from([(parent.clone(), dir_identity(&parent))]), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - - worker.apply_watch_sets(refresh_for(vec![AgentWatchSet { - declaration_path: declaration, - bus_id: "host.worker".to_owned(), - seat_id: None, - carriers: vec![WatchableCarrier { - label: "goal".to_owned(), - path: new_path.clone(), - class: CarrierClass::Immediate, - containment_root: None, - }], - declaration_summary: None, - }])); - - assert!(!worker.carriers.contains_key(&old_path)); - assert!(worker.carriers[&new_path][0].dirty); - assert!( - worker.deadlines.contains_key(&CarrierClass::Immediate), - "metadata refresh must enqueue the rebound digest even though its parent stayed watched" - ); - } - - #[test] - fn dirty_entry_deadline_migrates_when_refresh_changes_notification_class() { - let root = tempfile::tempdir().unwrap(); - let carrier = root.path().join("carrier.md"); - std::fs::write(&carrier, "same bytes").unwrap(); - let declaration = root.path().join("agent.kdl"); - let old_deadline = Instant::now(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([( - carrier.clone(), - vec![Entry { - bus_id: "host.worker".to_owned(), - seat_id: None, - label: "spec".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: read_state(&carrier, None).ok(), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::from([(CarrierClass::Immediate, old_deadline)]), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - let refresh = |class| { - refresh_for(vec![AgentWatchSet { - declaration_path: declaration.clone(), - bus_id: "host.worker".to_owned(), - seat_id: None, - carriers: vec![WatchableCarrier { - label: "spec".to_owned(), - path: carrier.clone(), - class, - containment_root: None, - }], - declaration_summary: None, - }]) - }; - - worker.apply_watch_sets(refresh(CarrierClass::Coalesced)); - assert!(!worker.deadlines.contains_key(&CarrierClass::Immediate)); - assert!( - worker.deadlines[&CarrierClass::Coalesced] >= old_deadline + COALESCED_WINDOW, - "immediate-to-coalesced migration receives the new class window" - ); - - let coalesced_deadline = worker.deadlines[&CarrierClass::Coalesced]; - worker.apply_watch_sets(refresh(CarrierClass::Immediate)); - assert!(!worker.deadlines.contains_key(&CarrierClass::Coalesced)); - assert!( - worker.deadlines[&CarrierClass::Immediate] < coalesced_deadline, - "coalesced-to-immediate migration is rescheduled under the shorter window" - ); - assert!(worker.carriers[&carrier][0].dirty); - } - - #[test] - fn malformed_declaration_retains_only_an_observed_live_seat_subscription() { - let declaration = PathBuf::from("/catalog/agents/hetz/worker/agent.kdl"); - let previous = || { - BTreeMap::from([( - declaration.clone(), - vec![Entry { - bus_id: "hetz.worker".to_owned(), - seat_id: Some("custom-worker-seat".to_owned()), - label: "declaration".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("before".to_owned())), - declaration_summary: None, - occurrence_sequence: 1, - pending_transition: Some(PendingTransition::new( - "declaration", - &declaration, - &CarrierState::Present("before".to_owned()), - &CarrierState::Present("corrected".to_owned()), - owner_incarnation(1), - 1, - )), - in_flight: false, - parked: false, - dirty: true, - }], - )]) - }; - let malformed_declarations = BTreeSet::from([declaration.clone()]); - - let retained = rebuild_carriers( - previous(), - WatchRefresh { - sets: Vec::new(), - malformed_declarations: malformed_declarations.clone(), - live_task_ids: BTreeSet::from(["custom-worker-seat".to_owned()]), - }, - &BTreeMap::new(), - &mut BTreeMap::new(), - ); - let entry = &retained[&declaration][0]; - assert_eq!( - entry.state, - Some(CarrierState::Present("before".to_owned())) - ); - assert_eq!( - entry - .pending_transition - .as_ref() - .map(|pending| pending.new_state.clone()), - Some(CarrierState::Present("corrected".to_owned())) - ); - - let dropped = rebuild_carriers( - previous(), - WatchRefresh { - sets: Vec::new(), - malformed_declarations, - live_task_ids: BTreeSet::new(), - }, - &BTreeMap::new(), - &mut BTreeMap::new(), - ); - assert!( - dropped.is_empty(), - "a malformed declaration must not retain a watch after its exact seat is no longer live" - ); - } - - #[test] - fn degraded_poll_replays_a_pending_transition_before_newer_bytes() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/hetz/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "hetz" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let carrier = resources.join("goal.md"); - std::fs::write(&carrier, "newer live bytes").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "hetz").unwrap(); - - let set = watch_set_for(&discover(root.path()), "hetz", &Default::default()); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "hetz".to_owned(), - carriers: BTreeMap::from([( - carrier.clone(), - vec![Entry { - bus_id: "hetz.worker".to_owned(), - seat_id: set.seat_id.clone(), - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("old-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 1, - pending_transition: Some(PendingTransition::new( - "goal", - &carrier, - &CarrierState::Present("old-digest".to_owned()), - &CarrierState::Present("pending-target".to_owned()), - owner_incarnation(1), - 1, - )), - in_flight: false, - parked: false, - dirty: false, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - let now = Instant::now(); - worker.apply_watch_sets(refresh_for(vec![set])); - assert!( - !resources.join("inbox").exists(), - "degraded polling must schedule rather than emit during refresh" - ); - worker.flush_due_publishing(now + IMMEDIATE_WINDOW + Duration::from_secs(1)); - - let event = std::fs::read_dir(resources.join("inbox")) - .unwrap() - .map(|entry| std::fs::read_to_string(entry.unwrap().path()).unwrap()) - .find(|body| body.contains("stream: resync")) - .expect("pending transition is replayed"); - assert_eq!(event_field(&event, "old"), "old-digest"); - assert_eq!(event_field(&event, "new"), "pending-targ"); - let entry = &worker.carriers[&carrier][0]; - assert_eq!( - entry.state, - Some(CarrierState::Present("pending-target".to_owned())) - ); - assert!(entry.pending_transition.is_none()); - assert!( - entry.dirty, - "newer live bytes are scheduled only after the pending transition completes" - ); - } - - #[test] - fn fallback_polling_preserves_the_coalesced_window() { - let root = tempfile::tempdir().unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let carrier = root.path().join("carrier.md"); - std::fs::write(&carrier, "before").unwrap(); - let baseline = read_state(&carrier, None).ok(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([( - carrier.clone(), - vec![Entry { - bus_id: "host.missing".to_owned(), - seat_id: None, - label: "spec".to_owned(), - class: CarrierClass::Coalesced, - containment_root: None, - state: baseline.clone(), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: false, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - std::fs::write(&carrier, "after").unwrap(); - let now = Instant::now(); - worker.apply_watch_sets(refresh_for(vec![AgentWatchSet { - declaration_path: PathBuf::from("/catalog/missing/agent.kdl"), - bus_id: "host.missing".to_owned(), - seat_id: None, - carriers: vec![WatchableCarrier { - label: "spec".to_owned(), - path: carrier.clone(), - class: CarrierClass::Coalesced, - containment_root: None, - }], - declaration_summary: None, - }])); - - worker.flush_due_publishing(now + IMMEDIATE_WINDOW + Duration::from_secs(1)); - let entry = &worker.carriers[&carrier][0]; - assert_eq!(entry.state, baseline); - assert!(entry.pending_transition.is_none(), "coalesced emit ran too early"); - - worker.flush_due_publishing(now + COALESCED_WINDOW + Duration::from_secs(1)); - assert!( - worker.carriers[&carrier][0].pending_transition.is_some(), - "the coalesced transition must be attempted after its full window" - ); - } - - #[test] - fn notify_backend_error_rescans_every_carrier_digest() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/hetz/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "hetz" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let goal = resources.join("goal.md"); - std::fs::write(&goal, "before\n").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "hetz").unwrap(); - - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "hetz".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.apply_watch_sets(refresh_for(vec![watch_set_for( - &discover(root.path()), - "hetz", - &Default::default(), - )])); - std::fs::write(&goal, "after\n").unwrap(); - - let (tx, rx) = std::sync::mpsc::channel(); - forward_watch_result(&tx, Err(notify::Error::generic("backend dropped events"))); - match rx.recv().unwrap() { - Msg::Rescan => worker.rescan_all(), - _ => panic!("a notify backend error must request a full digest rescan"), - } - worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW); - - let inbox = resources.join("inbox"); - let events = std::fs::read_dir(inbox) - .unwrap() - .map(|entry| std::fs::read_to_string(entry.unwrap().path()).unwrap()) - .collect::>(); - assert_eq!(events.len(), 1); - assert!(events[0].contains("stream: resync")); - assert!(events[0].contains(r#""binding":"goal""#)); - } - - #[test] - fn registering_another_directory_rediffs_the_untouched_watch_set() { - // A registration change is not free of the subscriptions it does not name: notify's - // macOS FSEvents backend stops the one shared stream on every `watch`, purges the - // device's pending events, and restarts at "since now", so mutations already queued for - // a directory that stayed in the set are destroyed. The state that leaves behind is a - // registered watch that will never report a change that already happened, and the model - // here is exact — the live parents are recorded as covered but never handed to the - // backend, so no event about them can exist. Only re-diffing the whole set recovers it. - let root = tempfile::tempdir().unwrap(); - let write_agent = |identity: &str| { - let agent_dir = root.path().join("agents/alias").join(identity); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - format!( - r#"agent "{identity}" {{ - host "alias" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}}"# - ), - ) - .unwrap(); - let goal = resources.join("goal.md"); - std::fs::write(&goal, "before\n").unwrap(); - (agent_dir, goal) - }; - let (live_dir, live_goal) = write_agent("live"); - write_agent("joining"); - crate::event::publish_owner_binding_for_test(root.path(), "alias").unwrap(); - - let specs = crate::discover_strict(root.path()).specs; - let set_for = |identity: &str| { - let spec = specs - .iter() - .find(|spec| spec.path.starts_with(root.path().join("agents/alias").join(identity))) - .expect("both declarations are valid"); - watch_set_for(spec, "alias", &ResourceProfileRegistry::empty()) - }; - let live_set = set_for("live"); - let joining_set = set_for("joining"); - - let (tx, _rx) = channel::(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "alias".to_owned(), - carriers: rebuild_carriers( - BTreeMap::new(), - refresh_for(vec![live_set.clone()]), - &BTreeMap::new(), - &mut BTreeMap::new(), - ), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: make_watcher(tx), - emit: Arc::new(EmitQueue::default()), - }; - worker.watched = worker - .carriers - .keys() - .filter_map(|path| path.parent()) - .map(|dir| (dir.to_path_buf(), dir_identity(dir))) - .collect(); - - std::fs::write(&live_goal, "changed with no watch able to report it\n").unwrap(); - - // The joining seat contributes directories the backend has not seen, so this refresh - // changes the registration set without touching the live subscription's own paths. - worker.apply_watch_sets(refresh_for(vec![live_set, joining_set])); - worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); - - let event = resync_inbox_event(&live_dir); - assert_eq!(event_field(&event, "binding"), "goal"); - // The joining seat has no inbox at all: its baseline seeded silently, as a new - // subscription must, so the rescan is not simply emitting for everything it re-reads. - assert!( - !root - .path() - .join("agents/alias/joining/resources/inbox") - .exists(), - "the joining seat seeds its baseline silently" - ); - } - - #[cfg(unix)] - #[test] - fn digesting_a_fifo_fails_without_blocking_the_worker() { - use std::os::unix::ffi::OsStrExt as _; - - let tmp = tempfile::tempdir().unwrap(); - let fifo = tmp.path().join("carrier.fifo"); - let fifo_c = std::ffi::CString::new(fifo.as_os_str().as_bytes()).unwrap(); - // SAFETY: the path is NUL-terminated and points into the live temp directory. - assert_eq!(unsafe { libc::mkfifo(fifo_c.as_ptr(), 0o600) }, 0); - assert_eq!(read_state(&fifo, None).unwrap(), CarrierState::Missing); - assert_eq!( - read_state(&fifo, Some(tmp.path())).unwrap(), - CarrierState::Missing, - "confined carrier reads must reject a FIFO without blocking too" - ); - } - - #[test] - fn due_flush_only_clears_subscribers_of_the_due_class() { - let root = tempfile::tempdir().unwrap(); - let carrier = root.path().join("carrier.md"); - std::fs::write(&carrier, "same bytes").unwrap(); - let state = read_state(&carrier, None).ok(); - let entries = [CarrierClass::Immediate, CarrierClass::Coalesced] - .into_iter() - .map(|class| Entry { - bus_id: "host.worker".to_owned(), - seat_id: None, - label: format!("{class:?}"), - class, - containment_root: None, - state: state.clone(), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }) - .collect(); - let now = Instant::now(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([(carrier, entries)]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::from([(CarrierClass::Immediate, now)]), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - - worker.flush_due_publishing(now); - let entries = worker.carriers.values().next().unwrap(); - assert!(!entries[0].dirty); - assert!(entries[1].dirty, "coalesced subscriber must wait for its own deadline"); - } - - #[cfg(unix)] - #[test] - fn confined_read_refuses_a_symlink_created_after_resolution() { - let agent_dir = tempfile::tempdir().expect("agent directory"); - let resources = agent_dir.path().join("resources"); - std::fs::create_dir(&resources).expect("resources directory"); - let carrier = resources.join("goal.md"); - std::fs::write(&carrier, "inside").expect("inside carrier"); - assert!( - matches!( - read_state(&carrier, Some(agent_dir.path())), - Ok(CarrierState::Present(_)) - ), - "ordinary files beneath the admitted root remain readable" - ); - - std::fs::remove_file(&carrier).expect("remove inside carrier"); - let outside = tempfile::NamedTempFile::new().expect("outside carrier"); - std::fs::write(outside.path(), "external").expect("outside bytes"); - std::os::unix::fs::symlink(outside.path(), &carrier) - .expect("replace absent carrier with an external symlink"); - assert_eq!( - read_state(&carrier, Some(agent_dir.path())).unwrap(), - CarrierState::Missing - ); - - std::fs::remove_file(&carrier).expect("remove final symlink"); - std::fs::remove_dir(&resources).expect("remove resources directory"); - let outside_dir = tempfile::tempdir().expect("outside directory"); - std::fs::write(outside_dir.path().join("goal.md"), "external").expect("outside carrier"); - std::os::unix::fs::symlink(outside_dir.path(), &resources) - .expect("replace absent ancestor with an external symlink"); - assert_eq!( - read_state(&carrier, Some(agent_dir.path())).unwrap(), - CarrierState::Missing - ); - } - - #[cfg(unix)] - #[test] - fn confined_read_refuses_a_symlinked_confinement_root_ancestor() { - let temp = tempfile::tempdir().expect("outer directory"); - let real_root = temp.path().join("real/agent"); - std::fs::create_dir_all(&real_root).expect("real agent directory"); - std::fs::write(real_root.join("goal.md"), "outside admitted ancestry") - .expect("carrier bytes"); - let alias = temp.path().join("alias"); - std::os::unix::fs::symlink(temp.path().join("real"), &alias) - .expect("symlinked root ancestor"); - let admitted_root = alias.join("agent"); - assert_eq!( - read_state(&admitted_root.join("goal.md"), Some(&admitted_root)).unwrap(), - CarrierState::Missing, - "every component of the confinement root must be opened without following symlinks" - ); - } - #[test] - fn deletion_and_same_byte_recreation_are_distinct_carrier_transitions() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/host/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "host" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let carrier = resources.join("goal.md"); - std::fs::write(&carrier, "same bytes").unwrap(); - let CarrierState::Present(original_digest) = read_state(&carrier, None).unwrap() else { - panic!("regular carrier has a digest"); - }; - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let set = watch_set_for(&discover(root.path()), "host", &Default::default()); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.apply_watch_sets(refresh_for(vec![set])); - - std::fs::remove_file(&carrier).unwrap(); - worker.flush_path_publishing(&carrier, None); - let deletion = resync_inbox_events(&agent_dir); - assert_eq!(deletion.len(), 1); - assert_eq!( - event_field(&deletion[0], "old"), - original_digest.chars().take(12).collect::() - ); - assert_eq!(event_field(&deletion[0], "new"), "missing"); - assert!(event_field(&deletion[0], "occurrence").ends_with(":1")); - assert_eq!( - worker.carriers[&carrier][0].state, - Some(CarrierState::Missing) - ); - - worker.flush_path_publishing(&carrier, None); - assert_eq!( - resync_inbox_events(&agent_dir).len(), - 1, - "repeated missing observations are silent" - ); - - std::fs::write(&carrier, "same bytes").unwrap(); - worker.flush_path_publishing(&carrier, None); - let events = resync_inbox_events(&agent_dir); - assert_eq!( - events.len(), - 1, - "creation supersedes the tombstone under the binding key" - ); - let creation = &events[0]; - assert_eq!(event_field(creation, "old"), "missing"); - assert!(event_field(creation, "occurrence").ends_with(":2")); - } - - #[test] - #[cfg(unix)] - fn transient_permission_error_retries_without_emitting_a_tombstone() { - use std::os::unix::fs::PermissionsExt as _; - - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/host/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "host" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let carrier = resources.join("goal.md"); - std::fs::write(&carrier, "before").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let set = watch_set_for(&discover(root.path()), "host", &Default::default()); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.apply_watch_sets(refresh_for(vec![set])); - let baseline = worker.carriers[&carrier][0].state.clone(); - - let original_permissions = std::fs::metadata(&carrier).unwrap().permissions(); - std::fs::set_permissions(&carrier, std::fs::Permissions::from_mode(0)).unwrap(); - worker.flush_path_publishing(&carrier, None); - let entry = &worker.carriers[&carrier][0]; - assert_eq!(entry.state, baseline); - assert!(entry.pending_transition.is_none()); - assert!(entry.dirty); - assert!(!resources.join("inbox").exists()); - - std::fs::set_permissions(&carrier, original_permissions).unwrap(); - std::fs::write(&carrier, "after").unwrap(); - worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); - let event = resync_inbox_event(&agent_dir); - assert_ne!(event_field(&event, "new"), "missing"); - } - - #[test] - #[cfg(unix)] - fn initial_transient_read_failure_schedules_a_baseline_retry() { - use std::os::unix::fs::PermissionsExt as _; - - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/host/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "host" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let carrier = resources.join("goal.md"); - std::fs::write(&carrier, "baseline").unwrap(); - let original_permissions = std::fs::metadata(&carrier).unwrap().permissions(); - std::fs::set_permissions(&carrier, std::fs::Permissions::from_mode(0)).unwrap(); - let set = watch_set_for(&discover(root.path()), "host", &Default::default()); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::from([(resources.clone(), dir_identity(&resources))]), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - - worker.apply_watch_sets(refresh_for(vec![set])); - let entry = worker.carriers[&carrier] - .iter() - .find(|entry| entry.label == "goal") - .unwrap(); - assert_eq!(entry.state, None); - assert!(entry.dirty); - assert!(worker.deadlines.contains_key(&CarrierClass::Immediate)); - - std::fs::set_permissions(&carrier, original_permissions).unwrap(); - worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); - let entry = worker.carriers[&carrier] - .iter() - .find(|entry| entry.label == "goal") - .unwrap(); - assert!(matches!(entry.state, Some(CarrierState::Present(_)))); - assert!(!entry.dirty); - assert!(!resources.join("inbox").exists()); - } - - #[test] - fn reinstalled_subscription_keeps_occurrence_identity_without_an_active_watch() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/host/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "host" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let carrier = resources.join("goal.md"); - std::fs::write(&carrier, "A").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let set = watch_set_for(&discover(root.path()), "host", &Default::default()); - let seen_subscription_count = set.carriers.len(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.apply_watch_sets(refresh_for(vec![set.clone()])); - - std::fs::write(&carrier, "B").unwrap(); - worker.flush_path_publishing(&carrier, None); - let before_suspend = resync_inbox_event(&agent_dir); - - // Suspension removes every carrier and watch, while retaining one scalar sequence floor - // for each declaration/binding identity seen during this supervisor incarnation. - worker - .watched - .insert(resources.clone(), dir_identity(&resources)); - worker.apply_watch_sets(refresh_for(Vec::new())); - assert!(worker.carriers.is_empty()); - assert!(worker.watched.is_empty()); - assert_eq!( - worker.subscription_sequences.len(), - seen_subscription_count - ); - - std::fs::write(&carrier, "A").unwrap(); - worker.apply_watch_sets(refresh_for(vec![set])); - let resumed = worker.carriers[&carrier] - .iter() - .find(|entry| entry.label == "goal") - .unwrap(); - assert_eq!(resumed.occurrence_sequence, 1); - std::fs::write(&carrier, "B").unwrap(); - worker.flush_path_publishing(&carrier, None); - let after_resume = resync_inbox_event(&agent_dir); - - assert_eq!( - event_field(&before_suspend, "old"), - event_field(&after_resume, "old") - ); - assert_eq!( - event_field(&before_suspend, "new"), - event_field(&after_resume, "new") - ); - assert_ne!( - event_field(&before_suspend, "event-id"), - event_field(&after_resume, "event-id"), - "the post-resume A→B occurrence must not deduplicate against the pre-suspend one" - ); - assert!(event_field(&before_suspend, "occurrence").ends_with(":1")); - assert!(event_field(&after_resume, "occurrence").ends_with(":2")); - } - - #[test] - fn relocated_subscription_keeps_occurrence_sequence_in_the_recipient_namespace() { - let root = tempfile::tempdir().unwrap(); - let agent_dir = root.path().join("agents/host/worker"); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - r#"agent "worker" { - host "host" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let original_carrier = resources.join("goal.md"); - let relocated_carrier = resources.join("relocated-goal.md"); - std::fs::write(&original_carrier, "A").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let set = - watch_set_for(&discover(root.path()), "host", &ResourceProfileRegistry::empty()); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::new(), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - worker.apply_watch_sets(refresh_for(vec![set.clone()])); - - std::fs::write(&original_carrier, "B").unwrap(); - worker.flush_path_publishing(&original_carrier, None); - let first_a_to_b = resync_inbox_event(&agent_dir); - - std::fs::write(&relocated_carrier, "A").unwrap(); - let mut relocated = set; - relocated.declaration_path = agent_dir.join("relocated/agent.kdl"); - for carrier in &mut relocated.carriers { - if carrier.label == "declaration" { - carrier.path = relocated.declaration_path.clone(); - } else if carrier.label == "goal" { - carrier.path = relocated_carrier.clone(); - } - } - worker.apply_watch_sets(refresh_for(vec![relocated.clone()])); - let rebound = worker.carriers[&relocated_carrier] - .iter() - .find(|entry| entry.label == "goal") - .unwrap(); - assert_eq!(rebound.occurrence_sequence, 1); - assert_eq!( - rebound.state, - read_state(&original_carrier, None).ok() - ); - - worker.flush_path_publishing(&relocated_carrier, None); - let back_to_a = resync_inbox_event(&agent_dir); - assert_eq!(event_field(&back_to_a, "old"), event_field(&first_a_to_b, "new")); - assert_eq!(event_field(&back_to_a, "new"), event_field(&first_a_to_b, "old")); - assert!(event_field(&back_to_a, "occurrence").ends_with(":2")); - - std::fs::write(&relocated_carrier, "B").unwrap(); - worker.flush_path_publishing(&relocated_carrier, None); - let second_a_to_b = resync_inbox_event(&agent_dir); - assert_eq!( - event_field(&first_a_to_b, "old"), - event_field(&second_a_to_b, "old") - ); - assert_eq!( - event_field(&first_a_to_b, "new"), - event_field(&second_a_to_b, "new") - ); - assert_ne!( - event_field(&first_a_to_b, "event-id"), - event_field(&second_a_to_b, "event-id") - ); - assert!(event_field(&first_a_to_b, "occurrence").ends_with(":1")); - assert!(event_field(&second_a_to_b, "occurrence").ends_with(":3")); - - relocated.bus_id = "host.replacement".to_owned(); - worker.apply_watch_sets(refresh_for(vec![relocated])); - assert_eq!( - worker.carriers[&relocated_carrier] - .iter() - .find(|entry| entry.label == "goal") - .unwrap() - .occurrence_sequence, - 0, - "a different recipient starts a distinct deduplication namespace" - ); - } - - #[test] - fn subscribers_advance_occurrence_sequences_independently() { - let root = tempfile::tempdir().unwrap(); - let carrier = root.path().join("shared.md"); - std::fs::write(&carrier, "new bytes").unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let entries = ["host.alpha", "host.beta"] - .into_iter() - .map(|bus_id| Entry { - bus_id: bus_id.to_owned(), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("old-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }) - .collect(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([(carrier.clone(), entries)]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - - worker.flush_path_publishing(&carrier, None); - - let entries = &worker.carriers[&carrier]; - assert_eq!(entries[0].occurrence_sequence, 1); - assert_eq!(entries[1].occurrence_sequence, 1); - assert_eq!( - event_field(&entries[0].pending_transition.as_ref().unwrap().body, "occurrence"), - event_field(&entries[1].pending_transition.as_ref().unwrap().body, "occurrence"), - "one subscriber must not consume sequence numbers from another" - ); - } - - #[test] - fn supervisor_restart_incarnation_changes_the_occurrence_namespace() { - let first = PendingTransition::new( - "goal", - Path::new("/agent/goal.md"), - &CarrierState::Present("old".to_owned()), - &CarrierState::Present("new".to_owned()), - owner_incarnation(1), - 1, - ); - let restarted = PendingTransition::new( - "goal", - Path::new("/agent/goal.md"), - &CarrierState::Present("old".to_owned()), - &CarrierState::Present("new".to_owned()), - owner_incarnation(2), - 1, - ); - - assert_ne!(first.body, restarted.body); - assert_ne!(first.event_id, restarted.event_id); - } - - #[test] - fn failed_tombstone_emit_retains_present_state_and_immutable_retry_snapshot() { - let root = tempfile::tempdir().unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let carrier = root.path().join("carrier.md"); - std::fs::write(&carrier, "old bytes").unwrap(); - let mut worker = Worker { - root: root.path().to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([( - carrier.clone(), - vec![Entry { - bus_id: "host.missing".to_owned(), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("old-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - }; - std::fs::remove_file(&carrier).unwrap(); - - worker.flush_path_publishing(&carrier, None); - let pending_transition = worker.carriers[&carrier][0] - .pending_transition - .clone() - .expect("failed transition snapshot is retained"); - assert_eq!(pending_transition.new_state, CarrierState::Missing); - assert_eq!(event_field(&pending_transition.body, "old"), "old-digest"); - assert_eq!(event_field(&pending_transition.body, "new"), "missing"); - assert_eq!(worker.carriers[&carrier][0].occurrence_sequence, 1); - std::fs::write(&carrier, "old bytes").unwrap(); - worker.flush_path_publishing(&carrier, None); - let entry = &worker.carriers[&carrier][0]; - assert_eq!(entry.occurrence_sequence, 1); - assert_eq!( - entry.state, - Some(CarrierState::Present("old-digest".to_owned())) - ); - assert_eq!( - entry.pending_transition.as_ref(), - Some(&pending_transition), - "a retry must replay the tombstone snapshot even after the carrier is recreated" - ); - assert!(entry.dirty); - assert!(worker.deadlines.contains_key(&CarrierClass::Immediate)); - } - - fn handoff_worker(root: &Path, carrier: &Path, recipients: &[&str]) -> Worker { - crate::event::publish_owner_binding_for_test(root, "host").unwrap(); - std::fs::write(carrier, "current bytes").unwrap(); - Worker { - root: root.to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([( - carrier.to_path_buf(), - recipients - .iter() - .map(|bus_id| Entry { - bus_id: (*bus_id).to_owned(), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("stale-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }) - .collect(), - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - } - } - - /// One agent declaration plus its goal carrier, and a subscription to that carrier whose - /// baseline is stale, so the next flush captures exactly one transition for it. - /// - /// A subscription exists only for a canonical seat a pass proved alive, so a subscription - /// addressed to a declaration that says `suspended` IS the suspended-and-running state — the - /// one that produced ~2010 refusals per seat in #431. A seat whose task had already exited - /// produces no subscription at all, which is why a fixture built on one proves nothing. - fn declared_recipient_worker(root: &Path, identity: &str, declaration_extra: &str) -> Worker { - crate::event::publish_owner_binding_for_test(root, "host").unwrap(); - let agent_dir = root.join("agents/host").join(identity); - std::fs::create_dir_all(agent_dir.join("resources")).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - format!( - "agent \"{identity}\" {{\n host \"host\"\n command \"agent\"\n{declaration_extra} resource \"goal\" uri=\"resources/goal.md\" reason=\"Mission.\"\n}}" - ), - ) - .unwrap(); - let goal = agent_dir.join("resources/goal.md"); - std::fs::write(&goal, "current bytes").unwrap(); - Worker { - root: root.to_path_buf(), - this_host: "host".to_owned(), - carriers: BTreeMap::from([( - goal, - vec![Entry { - bus_id: format!("host.{identity}"), - seat_id: None, - label: "goal".to_owned(), - class: CarrierClass::Immediate, - containment_root: None, - state: Some(CarrierState::Present("stale-digest".to_owned())), - declaration_summary: None, - occurrence_sequence: 0, - pending_transition: None, - in_flight: false, - parked: false, - dirty: true, - }], - )]), - subscription_sequences: BTreeMap::new(), - parked_transitions: BTreeMap::new(), - deadlines: BTreeMap::new(), - watched: BTreeMap::new(), - watcher: None, - emit: Arc::new(EmitQueue::default()), - } - } - - /// A recipient that refuses because it is not running keeps its reservation and is attempted - /// exactly once. Retrying it cannot make it admissible: eligibility was resolved under the - /// catalog-authoring lock, and each attempt re-resolves the whole catalog to reach the same - /// answer. That retry is the CPU burn and the shared-lock coverage of #431. - #[test] - fn a_not_running_recipient_parks_its_reservation_and_is_attempted_once() { - let root = tempfile::tempdir().unwrap(); - let mut worker = declared_recipient_worker( - root.path(), - "worker", - " desired-state \"suspended\" reason=\"Waiting for capacity\"\n", - ); - let goal = worker.carriers.keys().next().unwrap().clone(); - - worker.flush_path_publishing(&goal, None); - - let entry = &worker.carriers[&goal][0]; - assert!(entry.parked, "the subscription must be parked"); - assert!( - entry.pending_transition.is_none(), - "the reservation moves out of the subscription, which a refresh will drop" - ); - assert!( - worker - .parked_transitions - .contains_key(&("host.worker".to_owned(), "goal".to_owned())), - "the reservation must be retained, not dropped: {:?}", - worker.parked_transitions.keys().collect::>() - ); - assert!( - worker.deadlines.is_empty(), - "a not-running recipient must schedule no retry deadline: {:?}", - worker.deadlines - ); - - // Each source that would ordinarily re-arm this carrier, checked on its own: a parked - // subscription must schedule nothing, not merely publish nothing. - std::fs::write(&goal, "changed while the recipient is suspended\n").unwrap(); - worker.mark_mutated(vec![goal.clone()]); - assert!( - worker.deadlines.is_empty() && !worker.carriers[&goal][0].dirty, - "a mutation wakeup must not schedule a parked subscription: {:?}", - worker.deadlines - ); - worker.rescan_all(); - assert!( - worker.deadlines.is_empty() && !worker.carriers[&goal][0].dirty, - "a rescan must not schedule a parked subscription: {:?}", - worker.deadlines - ); - worker.flush_path_publishing(&goal, None); - worker.flush_due_publishing(Instant::now() + COALESCED_WINDOW + Duration::from_secs(1)); - assert!(worker.deadlines.is_empty(), "{:?}", worker.deadlines); - assert_eq!( - worker.emit.handed_off(), - 1, - "a parked reservation must not be attempted again" - ); - } - - /// The reservation re-arms when its recipient is running again, and replays the exact bytes - /// it reserved. Dropping it at the refusal would lose a resync the agent should see on - /// resume; a refresh drops the suspended subscription itself, so the reservation has to - /// outlive it. - #[test] - fn a_parked_reservation_re_arms_and_replays_when_its_recipient_runs_again() { - let root = tempfile::tempdir().unwrap(); - let mut worker = declared_recipient_worker( - root.path(), - "worker", - " desired-state \"suspended\" reason=\"Waiting for capacity\"\n", - ); - let goal = worker.carriers.keys().next().unwrap().clone(); - let agent_dir = root.path().join("agents/host/worker"); - - worker.flush_path_publishing(&goal, None); - let reserved = worker - .parked_transitions - .values() - .next() - .expect("the refusal retains its reservation") - .clone(); - - std::fs::write( - agent_dir.join("agent.kdl"), - "agent \"worker\" {\n host \"host\"\n command \"agent\"\n resource \"goal\" uri=\"resources/goal.md\" reason=\"Mission.\"\n}", - ) - .unwrap(); - let resumed = watch_set_for( - &discover(root.path()), - "host", - &ResourceProfileRegistry::empty(), - ); - worker.apply_watch_sets(refresh_for(vec![resumed])); - - let entry = &worker.carriers[&goal][0]; - assert!(!entry.parked, "a carried recipient is running"); - assert_eq!( - entry.pending_transition.as_ref().map(|held| &held.event_id), - Some(&reserved.event_id), - "the restored reservation must be the reserved one" - ); - assert!( - worker.deadlines.contains_key(&CarrierClass::Immediate), - "re-arming must schedule the carrier's class: {:?}", - worker.deadlines - ); - assert!(worker.parked_transitions.is_empty()); - - worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); - let event = resync_inbox_event(&agent_dir); - assert!( - event.contains(&format!("event-id: {}", reserved.event_id)), - "{event}" - ); - assert!(event.contains(&reserved.body), "{event}"); - } - - /// A refusal no retry can admit drops its reservation and advances the baseline, so the same - /// carrier transition is not captured again on the next observation. Without advancing it, - /// "drop" would only mean "re-capture and refuse again". - #[test] - fn a_permanently_refused_reservation_is_dropped_and_not_recaptured() { - let root = tempfile::tempdir().unwrap(); - let mut worker = declared_recipient_worker(root.path(), "worker", ""); - // A second declaration of the same identity makes the recipient ambiguous. No retry can - // resolve that from the publisher's side. - let twin = root.path().join("agents/host/worker-copy"); - std::fs::create_dir_all(&twin).unwrap(); - std::fs::write( - twin.join("agent.kdl"), - "agent \"worker\" {\n host \"host\"\n command \"agent\"\n}", - ) - .unwrap(); - let goal = worker.carriers.keys().next().unwrap().clone(); - - worker.flush_path_publishing(&goal, None); - - let entry = &worker.carriers[&goal][0]; - assert!( - entry.pending_transition.is_none(), - "a permanent refusal drops its reservation" - ); - assert!(!entry.parked, "a permanent refusal is not parked"); - assert!(worker.parked_transitions.is_empty()); - assert!( - worker.deadlines.is_empty(), - "a permanent refusal must schedule no retry: {:?}", - worker.deadlines - ); - - worker.rescan_all(); - worker.flush_due_publishing(Instant::now() + COALESCED_WINDOW + Duration::from_secs(1)); - assert_eq!( - worker.emit.handed_off(), - 1, - "the dropped transition must not be recaptured" - ); - } - - /// Every refusal the catalog can return, classified. `RecipientNotRunning` is the only one a - /// resume can admit; the rest never become admissible. An unknown recipient and a catalog - /// mid-edit stay retryable on purpose: a declaration being replaced by rename is briefly - /// absent, and dropping the reservation there would lose a resync nothing was wrong with. - #[test] - fn refusals_are_classified_by_what_could_admit_them_later() { - use crate::event::{RefusalKind, refusal_kind}; - - let root = tempfile::tempdir().unwrap(); - crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); - let write = |identity: &str, body: &str| { - let dir = root.path().join("agents/host").join(identity); - std::fs::create_dir_all(&dir).unwrap(); - std::fs::write(dir.join("agent.kdl"), body).unwrap(); - }; - let emit = |recipient: &str| { - crate::event::emit_builtin_resync( - root.path(), - "host", - recipient, - "eventid", - Some("goal"), - Some("subject"), - "{}", - true, - ) - .expect_err("every recipient in this fixture refuses") - }; - - write( - "suspended", - "agent \"suspended\" {\n host \"host\"\n command \"agent\"\n desired-state \"suspended\" reason=\"Waiting\"\n}", - ); - write( - "foreign", - "agent \"foreign\" {\n host \"elsewhere\"\n command \"agent\"\n}", - ); - write("plain", "agent \"plain\" {\n host \"host\"\n command \"agent\"\n}"); - write( - "twin", - "agent \"plain\" {\n host \"host\"\n command \"agent\"\n}", - ); - write("solo", "agent \"solo\" {\n host \"host\"\n command \"agent\"\n}"); - - let suspended = emit("host.suspended"); - assert!( - suspended.to_string().contains("eyes are closed"), - "{suspended:#}" - ); - assert_eq!( - refusal_kind(&suspended), - Some(RefusalKind::RecipientNotRunning) - ); - - // A foreign spec is only reachable under its own host-qualified bus id; by local identity - // it is simply not in this host's namespace. - let foreign = emit("elsewhere.foreign"); - assert!( - foreign - .to_string() - .contains("event publication must run on that host"), - "{foreign:#}" - ); - assert_eq!(refusal_kind(&foreign), Some(RefusalKind::Permanent)); - - let ambiguous = emit("host.plain"); - assert!(ambiguous.to_string().contains("is ambiguous"), "{ambiguous:#}"); - assert_eq!(refusal_kind(&ambiguous), Some(RefusalKind::Permanent)); - - let undeclared = crate::event::emit( - root.path(), - "host", - "host.solo", - "other", - "eventid", - None, - None, - "{}", - true, - ) - .expect_err("an undeclared stream is refused"); - assert_eq!(refusal_kind(&undeclared), Some(RefusalKind::Permanent)); - - let unknown = emit("host.absent"); - assert!(unknown.to_string().contains("no agent"), "{unknown:#}"); - assert_eq!( - refusal_kind(&unknown), - None, - "an absent declaration may be a rename in flight, so it stays retryable" - ); - } - - /// One outstanding publication per subscription. Flushing again while a publication is still - /// in flight must not hand off the same subscription twice: whether its reserved event - /// identity is spent is decided by an outcome that has not returned yet. - #[test] - fn a_flush_never_hands_off_a_subscription_whose_publication_is_outstanding() { - let root = tempfile::tempdir().unwrap(); - let carrier = root.path().join("carrier.md"); - let mut worker = handoff_worker(root.path(), &carrier, &["host.missing"]); - - worker.flush_path(&carrier, None); - assert_eq!(worker.emit.queued_recipients(), ["host.missing"]); - assert!(worker.carriers[&carrier][0].in_flight); - - std::fs::write(&carrier, "newer bytes while the publication is outstanding").unwrap(); - worker.mark_mutated(vec![carrier.clone()]); - worker.flush_path(&carrier, None); - assert_eq!( - worker.emit.queued_recipients(), - ["host.missing"], - "an outstanding publication must not be handed off a second time" - ); - } - - /// A deactivation acknowledgement is the reconcile pass's guarantee that nothing starts - /// publishing to that recipient afterwards. A publication still queued when the seat is - /// deactivated is dropped, and only that recipient's. - #[test] - fn deactivation_drops_only_that_recipients_queued_publication() { - let root = tempfile::tempdir().unwrap(); - let carrier = root.path().join("carrier.md"); - let mut worker = handoff_worker(root.path(), &carrier, &["host.leaving", "host.staying"]); - - worker.flush_path(&carrier, None); - assert_eq!( - worker.emit.queued_recipients(), - ["host.leaving", "host.staying"] - ); - - worker.deactivate_watch_set("host.leaving"); - assert_eq!(worker.emit.queued_recipients(), ["host.staying"]); - } - - /// A refresh that drops a subscription — the suspended recipient of #431 — must drop its - /// queued publication too. The pass has already decided that seat receives no events, and a - /// queued refusal published afterwards is work no outcome can apply. - #[test] - fn a_refresh_drops_a_queued_publication_for_a_subscription_it_removed() { - let root = tempfile::tempdir().unwrap(); - let carrier = root.path().join("carrier.md"); - let mut worker = handoff_worker(root.path(), &carrier, &["host.missing"]); - - worker.flush_path(&carrier, None); - assert_eq!(worker.emit.queued_recipients(), ["host.missing"]); - - worker.apply_watch_sets(refresh_for(Vec::new())); - assert!(worker.carriers.is_empty()); - assert!( - worker.emit.queued_recipients().is_empty(), - "a dropped subscription's queued publication must not survive the refresh" - ); - } - - #[test] - fn transition_identity_covers_every_rendered_transition_dimension() { - let topics = vec!["content".to_owned()]; - let facts = - vec![ResourceFact::transition("digest", Some("old"), Some("new")).unwrap()]; - let baseline = render_body("goal", &topics, &facts, "v1:1:2:42:3:1"); - assert_eq!( - transition_identity(&baseline), - transition_identity(&baseline), - "replaying one canonical body must reproduce its identity" - ); - - let changed_facts = - vec![ResourceFact::transition("digest", Some("old"), Some("other")).unwrap()]; - for (dimension, changed) in [ - ( - "binding", - render_body("spec", &topics, &facts, "v1:1:2:42:3:1"), - ), - ( - "topic", - render_body("goal", &["other".to_owned()], &facts, "v1:1:2:42:3:1"), - ), - ( - "fact", - render_body("goal", &topics, &changed_facts, "v1:1:2:42:3:1"), - ), - ( - "occurrence", - render_body("goal", &topics, &facts, "v1:1:2:42:3:2"), - ), - ] { - assert_ne!( - transition_identity(&baseline), - transition_identity(&changed), - "changing {dimension} must change the event identity" - ); - } - } - - #[test] - fn local_path_resolution_parses_supported_file_uris_without_uri_metadata_bytes() { - let agent_dir = Path::new("/cat/agents/hetz/w"); - assert_eq!( - resolve_local_path(agent_dir, "file:///etc/demo.kdl"), - Some(PathBuf::from("/etc/demo.kdl")) - ); - for scheme in ["file", "FILE", "FiLe"] { - assert_eq!( - resolve_local_path(agent_dir, &format!("{scheme}:///etc/demo.kdl")), - Some(PathBuf::from("/etc/demo.kdl")) - ); - } - assert_eq!( - resolve_local_path(agent_dir, "file:///tmp/with%20space/%E2%82%AC.md"), - Some(PathBuf::from("/tmp/with space/€.md")) - ); - assert_eq!( - resolve_local_path(agent_dir, "file:///tmp/literal%3Fmark"), - Some(PathBuf::from("/tmp/literal?mark")) - ); - assert_eq!( - resolve_local_path(agent_dir, "file:///"), - Some(PathBuf::from("/")) - ); - assert_eq!( - resolve_local_path(agent_dir, "resources/journal.md"), - Some(agent_dir.join("resources/journal.md")) - ); - assert_eq!( - resolve_local_path( - agent_dir, - "resources/with%20space/%E2%82%AC-journal.md" - ), - Some(agent_dir.join("resources/with space/€-journal.md")) - ); - - for unsupported in [ - "file://authority/etc/demo.kdl", - "file:////authority/etc/demo.kdl", - "file:///etc/demo.kdl?revision=2", - "file:///etc/demo.kdl#section", - "file:///tmp/encoded%2Fseparator", - "file:///tmp/encoded%2fseparator", - "file:///tmp/encoded%5Cseparator", - "file:///tmp/bad%escape", - "file:///tmp/%2E%2E/escape", - "file:///tmp/a%00b", - "file:///tmp/%FF.md", - "resources/encoded%2Fseparator", - "resources/encoded%5Cseparator", - "resources/%2E%2E/outside.md", - "resources/bad%escape", - "resources/a%00b", - "resources/%FF.md", - "http://x/y", - "worktree://repo/main", - "GitHub-Issue://org/repo/41", - "ProFiLe:opaque", - ] { - assert_eq!( - resolve_local_path(agent_dir, unsupported), - None, - "{unsupported} must not become filesystem bytes" - ); - } - } - - #[test] - fn classification_is_goal_immediate_stores_silent_other_coalesced() { - let agent_dir = Path::new("/cat/agents/hetz/w"); - assert_eq!( - classify(agent_dir, "mission", &agent_dir.join("resources/goal.md")), - Some(CarrierClass::Immediate), - "basename goal.md is immediate regardless of binding name" - ); - assert_eq!( - classify( - agent_dir, - "journal", - &agent_dir.join("resources/context/journal.md") - ), - None, - "agent-authored stores are silent and excluded from the watch set" - ); - assert_eq!( - classify( - agent_dir, - "decision-log", - &agent_dir.join("resources/decisions/x.md") - ), - None - ); - assert_eq!( - classify(agent_dir, "spec", Path::new("/etc/demo/spec.md")), - Some(CarrierClass::Coalesced) - ); - } - - #[test] - fn profile_classes_map_onto_carrier_notification() { - assert_eq!(carrier_class(ProfileClass::Immediate), Some(CarrierClass::Immediate)); - assert_eq!(carrier_class(ProfileClass::Coalesced), Some(CarrierClass::Coalesced)); - assert_eq!( - carrier_class(ProfileClass::Silent), - None, - "silent profiles are excluded from the watch set like sniffed authored stores" - ); - } - - #[test] - fn registered_profile_failures_are_reported_while_other_bindings_survive() { - let tmp = tempfile::tempdir().unwrap(); - let dir = tmp.path().join("agents/hetz/worker"); - std::fs::create_dir_all(&dir).unwrap(); - std::fs::write( - dir.join("agent.kdl"), - r#"agent "worker" { - host "hetz" - command "true" - resource "goal" uri="dev.schickling.agent-goal://hetz/worker" reason="Mission." - resource "issue" uri="worktree://repo/main" reason="Opaque scheme." -}"#, - ) - .unwrap(); - - let broken = tmp.path().join("broken.wasm"); - std::fs::write(&broken, b"not a module").unwrap(); - let profiles = ResourceProfileRegistry::empty().with_profile( - agent_spec::ResourceProfile::wasm( - "dev.schickling.agent-goal", - &broken, - ProfileClass::Coalesced, - ), - ); - let refresh = profiles.begin_refresh(); - let spec = discover(tmp.path()); - let (set, diagnostics) = - resolve_watch_set(&spec, std::slice::from_ref(&spec), "hetz", &refresh); - assert!(!set.carriers.iter().any(|c| c.label == "goal")); - assert!(set.carriers.iter().any(|c| c.label == "declaration")); - assert!(!set.carriers.iter().any(|c| c.label == "issue")); - assert_eq!(diagnostics.len(), 1); - assert!(diagnostics[0].contains("resource 'goal'")); - assert!(diagnostics[0].contains("unwatchable")); - assert_eq!( - resource_coverage_with_profiles(&dir, &spec.resources[0], &refresh), - ResyncCoverage::Unsupported - ); - } - - #[test] - fn silent_profile_skips_its_resolver_entirely() { - let tmp = tempfile::tempdir().unwrap(); - let dir = tmp.path().join("agents/hetz/worker"); - std::fs::create_dir_all(&dir).unwrap(); - std::fs::write( - dir.join("agent.kdl"), - r#"agent "worker" { - host "hetz" - command "true" - resource "goal" uri="dev.schickling.agent-goal://hetz/worker" reason="Mission." -}"#, - ) - .unwrap(); - let missing = tmp.path().join("must-not-load.wasm"); - let profiles = ResourceProfileRegistry::empty().with_profile( - agent_spec::ResourceProfile::wasm( - "dev.schickling.agent-goal", - missing, - ProfileClass::Silent, - ), - ); - let refresh = profiles.begin_refresh(); - let spec = discover(tmp.path()); - let (set, diagnostics) = - resolve_watch_set(&spec, std::slice::from_ref(&spec), "hetz", &refresh); - assert!(!set.carriers.iter().any(|carrier| carrier.label == "goal")); - assert!( - diagnostics.is_empty(), - "a silent profile must not execute its missing resolver: {diagnostics:?}" - ); - assert_eq!( - resource_coverage_with_profiles(&dir, &spec.resources[0], &refresh), - ResyncCoverage::Silent - ); - } -} +mod tests; diff --git a/src/resync/read.rs b/src/resync/read.rs new file mode 100644 index 00000000..e1145e1f --- /dev/null +++ b/src/resync/read.rs @@ -0,0 +1,181 @@ +//! Bounded, symlink-refusing reads of a watched carrier. +//! +//! Moved verbatim out of the parent module: the confined open, the hash, and the classification +//! of an open error into carrier state. + +use std::path::Path; + +use sha2::{Digest as _, Sha256}; + +use super::*; + +pub(super) fn read_state(path: &Path, containment_root: Option<&Path>) -> std::io::Result { + match containment_root { + Some(root) => read_confined(path, root), + None => read_regular(path), + } +} + +pub(super) fn diagnose_read_error(path: &Path, error: &std::io::Error) { + eprintln!( + "st2: resync read for '{}' failed transiently; retrying: {error}", + path.display() + ); +} + +fn hash_reader(mut file: std::fs::File) -> std::io::Result { + let mut digest = Sha256::new(); + let mut buffer = [0_u8; 64 * 1024]; + loop { + let read = std::io::Read::read(&mut file, &mut buffer)?; + if read == 0 { + break; + } + digest.update(&buffer[..read]); + } + Ok(format!("{:x}", digest.finalize())) +} + +#[cfg(unix)] +fn classify_open_error(error: std::io::Error) -> std::io::Result { + match error.raw_os_error() { + Some(libc::ENOENT | libc::ENOTDIR | libc::ELOOP) => Ok(CarrierState::Missing), + _ => Err(error), + } +} + +#[cfg(not(unix))] +fn classify_open_error(error: std::io::Error) -> std::io::Result { + if error.kind() == std::io::ErrorKind::NotFound { + Ok(CarrierState::Missing) + } else { + Err(error) + } +} + +#[cfg(unix)] +fn read_regular(path: &Path) -> std::io::Result { + use std::os::unix::fs::OpenOptionsExt as _; + + let file = match std::fs::OpenOptions::new() + .read(true) + .custom_flags(libc::O_NOFOLLOW | libc::O_CLOEXEC | libc::O_NONBLOCK) + .open(path) + { + Ok(file) => file, + Err(error) => return classify_open_error(error), + }; + if !file.metadata()?.file_type().is_file() { + return Ok(CarrierState::Missing); + } + hash_reader(file).map(CarrierState::Present) +} + +#[cfg(not(unix))] +fn read_regular(path: &Path) -> std::io::Result { + let file = match std::fs::File::open(path) { + Ok(file) => file, + Err(error) => return classify_open_error(error), + }; + if !file.metadata()?.file_type().is_file() { + return Ok(CarrierState::Missing); + } + hash_reader(file).map(CarrierState::Present) +} + +#[cfg(unix)] +fn read_confined(path: &Path, root: &Path) -> std::io::Result { + use std::ffi::CString; + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + use std::os::unix::ffi::OsStrExt as _; + + let invalid_path = || { + std::io::Error::new( + std::io::ErrorKind::InvalidInput, + "carrier path is outside its confinement root or has an unsafe component", + ) + }; + let relative = path.strip_prefix(root).map_err(|_| invalid_path())?; + let mut components = relative.components().peekable(); + if components.peek().is_none() { + return Ok(CarrierState::Missing); + } + + // Open the confinement root component-by-component from the filesystem root. `O_NOFOLLOW` + // on one full pathname protects only its final component; descriptor-relative traversal + // protects every ancestor from symlink replacement as well. + let mut root_components = root.components(); + if root_components.next() != Some(std::path::Component::RootDir) { + return Err(invalid_path()); + } + let slash = CString::new("/").map_err(|_| invalid_path())?; + // SAFETY: `slash` is NUL-terminated and the returned descriptor is checked before ownership. + let filesystem_root = unsafe { + libc::open( + slash.as_ptr(), + libc::O_RDONLY | libc::O_DIRECTORY | libc::O_CLOEXEC | libc::O_NOFOLLOW, + ) + }; + if filesystem_root < 0 { + return classify_open_error(std::io::Error::last_os_error()); + } + // SAFETY: `filesystem_root` is newly owned after the non-negative check. + let mut directory = unsafe { OwnedFd::from_raw_fd(filesystem_root) }; + for component in root_components { + let std::path::Component::Normal(name) = component else { + return Err(invalid_path()); + }; + let name = CString::new(name.as_bytes()).map_err(|_| invalid_path())?; + let flags = libc::O_RDONLY + | libc::O_DIRECTORY + | libc::O_CLOEXEC + | libc::O_NOFOLLOW + | libc::O_NONBLOCK; + // SAFETY: the live directory descriptor and NUL-terminated component are valid. + let opened = unsafe { libc::openat(directory.as_raw_fd(), name.as_ptr(), flags) }; + if opened < 0 { + return classify_open_error(std::io::Error::last_os_error()); + } + // SAFETY: `opened` is newly owned after the non-negative check. + directory = unsafe { OwnedFd::from_raw_fd(opened) }; + } + + while let Some(component) = components.next() { + let std::path::Component::Normal(name) = component else { + return Err(invalid_path()); + }; + let name = CString::new(name.as_bytes()).map_err(|_| invalid_path())?; + let last = components.peek().is_none(); + let flags = libc::O_RDONLY + | libc::O_CLOEXEC + | libc::O_NOFOLLOW + | libc::O_NONBLOCK + | if last { 0 } else { libc::O_DIRECTORY }; + // SAFETY: both the live directory descriptor and NUL-terminated component are valid; + // `O_NOFOLLOW` makes each lookup fail closed if that component is replaced by a symlink. + let opened = unsafe { libc::openat(directory.as_raw_fd(), name.as_ptr(), flags) }; + if opened < 0 { + return classify_open_error(std::io::Error::last_os_error()); + } + // SAFETY: `opened` is a newly-owned descriptor after the non-negative check above. + let opened = unsafe { OwnedFd::from_raw_fd(opened) }; + if last { + let file = std::fs::File::from(opened); + if !file.metadata()?.file_type().is_file() { + return Ok(CarrierState::Missing); + } + return hash_reader(file).map(CarrierState::Present); + } + directory = opened; + } + Ok(CarrierState::Missing) +} + +#[cfg(not(unix))] +fn read_confined(_path: &Path, _root: &Path) -> std::io::Result { + // No std API can atomically enforce no-follow traversal. Fail closed on unsupported hosts. + Err(std::io::Error::new( + std::io::ErrorKind::Unsupported, + "descriptor-relative no-follow reads are unavailable", + )) +} diff --git a/src/resync/tests.rs b/src/resync/tests.rs new file mode 100644 index 00000000..86b5b80e --- /dev/null +++ b/src/resync/tests.rs @@ -0,0 +1,2265 @@ +use super::*; + +fn refresh_for(sets: Vec) -> WatchRefresh { + WatchRefresh { + sets, + malformed_declarations: BTreeSet::new(), + live_task_ids: BTreeSet::new(), + } +} + +#[test] +fn resync_subject_uses_the_shared_three_fact_and_96_scalar_renderer() { + let facts = vec![ + ResourceFact::transition("alpha", None::, Some("declared")).unwrap(), + ResourceFact::current("beta", "changed").unwrap(), + ResourceFact::transition("charlie", Some("declared"), None::).unwrap(), + ResourceFact::current("delta", "omitted").unwrap(), + ]; + let subject = crate::resource_profile_supervisor::resource_change_subject( + "declaration", + &facts, + &["declaration".to_owned()], + "content changed", + ); + assert_eq!( + subject, + "declaration · alpha=+declared; beta=changed; charlie=-declared [declaration]" + ); + assert!(subject.chars().count() <= 96); + assert!(!subject.contains("delta")); +} + +fn owner_incarnation(seed: u64) -> crate::event::StreamOwnerIncarnation { + crate::event::StreamOwnerIncarnation::for_test(seed, seed + 1, 42, seed + 2) +} + +fn discover(catalog: &Path) -> AgentSpec { + let found = crate::discover_strict(catalog); + eprintln!("discovery errors: {:?}", found.errors); + found.specs.into_iter().next().unwrap() +} + +fn resync_inbox_event(agent_dir: &Path) -> String { + std::fs::read_dir(agent_dir.join("resources/inbox")) + .unwrap() + .filter_map(Result::ok) + .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) + .find(|event| event.lines().any(|line| line == "stream: resync")) + .expect("current resync inbox event") +} +fn resync_inbox_events(agent_dir: &Path) -> Vec { + std::fs::read_dir(agent_dir.join("resources/inbox")) + .unwrap() + .filter_map(Result::ok) + .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) + .filter(|event| event.lines().any(|line| line == "stream: resync")) + .collect() +} + +fn event_body(event: &str) -> serde_json::Value { + let body = event + .lines() + .rev() + .find(|line| line.starts_with('{')) + .expect("JSON resync body"); + serde_json::from_str(body).expect("valid JSON resync body") +} + +fn event_field(event: &str, field: &str) -> String { + if let Some(value) = event + .lines() + .find_map(|line| line.strip_prefix(&format!("{field}: "))) + { + return value.to_owned(); + } + let body = event_body(event); + if matches!(field, "old" | "new") { + let digest = body["facts"] + .as_array() + .and_then(|facts| facts.iter().find(|fact| fact["key"] == "digest")) + .expect("digest transition fact"); + let value = if field == "old" { + &digest["before"] + } else { + &digest["after"] + }; + return value.as_str().expect("digest fact value").to_owned(); + } + body[field] + .as_str() + .unwrap_or_else(|| panic!("missing {field} in event")) + .to_owned() +} +#[test] +fn declaration_facts_are_ordered_added_removed_and_semantically_changed_labels() { + let root = tempfile::tempdir().unwrap(); + let declaration = root.path().join("agent.kdl"); + std::fs::write( + &declaration, + r#"agent "worker" { + host "host" + command "true" + resource "inactive" uri="file:///inactive" reason="kept" + resource "reason" uri="file:///reason" reason="before" + resource "removed" uri="file:///removed" reason="gone" + resource "uri" uri="file:///before" reason="same" +}"#, + ) + .unwrap(); + let before = declaration_summary(&discover(root.path())); + std::fs::write( + &declaration, + r#"agent "worker" { + host "host" + command "true" + resource "added" uri="file:///added" reason="new" + resource "inactive" uri="file:///inactive" reason="kept" inactive-reason="paused" + resource "reason" uri="file:///reason" reason="after" + resource "uri" uri="file:///after" reason="same" +}"#, + ) + .unwrap(); + let after = declaration_summary(&discover(root.path())); + let facts = declaration_transition_facts( + Some(&before), + Some(&after), + &CarrierState::Present("before-digest".to_owned()), + &CarrierState::Present("after-digest".to_owned()), + ); + assert_eq!( + facts.iter().map(ResourceFact::key).collect::>(), + vec!["added", "inactive", "reason", "removed", "uri"] + ); + assert_eq!(facts[0].before(), Some(None)); + assert_eq!(facts[0].after(), Some(Some("declared"))); + for index in [1, 2, 4] { + assert_eq!(facts[index].before(), None); + assert_eq!(facts[index].after(), Some(Some("changed"))); + } + assert_eq!(facts[3].before(), Some(Some("declared"))); + assert_eq!(facts[3].after(), Some(None)); +} + +#[test] +fn declaration_parse_failure_retains_a_digest_fact_for_later_delivery() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/host/worker"); + std::fs::create_dir_all(&agent_dir).unwrap(); + let declaration = agent_dir.join("agent.kdl"); + let valid = r#"agent "worker" { + host "host" + command "true" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#; + std::fs::write(&declaration, valid).unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let set = watch_set_for(&discover(root.path()), "host", &Default::default()); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.apply_watch_sets(refresh_for(vec![set])); + std::fs::write(&declaration, "not an Agent Spec").unwrap(); + worker.flush_path_publishing(&declaration, None); + let pending = worker.carriers[&declaration][0] + .pending_transition + .as_ref() + .expect("malformed declaration keeps its digest fallback"); + assert_eq!(pending.facts.len(), 1); + assert_eq!(pending.facts[0].key(), "digest"); + assert_eq!(pending.topics, ["declaration"]); + assert_eq!(event_body(&pending.body)["facts"].as_array().unwrap().len(), 1); + std::fs::write(&declaration, valid).unwrap(); + worker.flush_path_publishing(&declaration, None); + let delivered = resync_inbox_event(&agent_dir); + let body = event_body(&delivered); + assert_eq!(body["binding"], "declaration"); + assert_eq!(body["topics"], serde_json::json!(["declaration"])); + assert_eq!(body["facts"][0]["key"], "digest"); + assert!(!delivered.contains("file:///")); + assert!(!delivered.contains("Mission.")); +} + + +#[test] +fn watch_set_covers_declaration_and_local_bindings_only() { + let tmp = tempfile::tempdir().unwrap(); + let dir = tmp.path().join("agents/hetz/worker"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("agent.kdl"), + r#"agent "worker" { + host "hetz" + command "true" + resource "goal" uri="resources/goal.md" reason="Mission." + resource "journal" uri="resources/context/journal.md" reason="Memory." + resource "issue" uri="github-issue://org/repo/41" reason="Task." + resource "old" uri="resources/old.md" reason="History." inactive-reason="No longer used." +}"#, + ) + .unwrap(); + let spec = discover(tmp.path()); + let set = watch_set_for(&spec, "hetz", &Default::default()); + assert_eq!(set.bus_id, "hetz.worker"); + let mut labels: Vec<&str> = set.carriers.iter().map(|c| c.label.as_str()).collect(); + labels.sort(); + assert_eq!(labels, vec!["declaration", "goal"]); + let goal = set.carriers.iter().find(|c| c.label == "goal").unwrap(); + assert_eq!(goal.class, CarrierClass::Immediate); + assert_eq!(goal.path, dir.join("resources/goal.md")); + let coverage = spec + .resources + .iter() + .map(|resource| (resource.name(), resource_coverage(&dir, resource))) + .collect::>(); + assert_eq!(coverage["goal"], ResyncCoverage::Immediate); + assert_eq!(coverage["journal"], ResyncCoverage::Silent); + assert_eq!(coverage["issue"], ResyncCoverage::Unsupported); + assert_eq!(coverage["old"], ResyncCoverage::Inactive); +} + +#[test] +fn bus_id_uses_the_supervisor_host_not_the_os_hostname() { + let tmp = tempfile::tempdir().unwrap(); + // A root-level declaration file supplies neither content nor path host: host stays + // None, so the supervisor's logical alias must decide the recipient. + std::fs::write( + tmp.path().join("worker.kdl"), + r#"agent "worker" { + command "true" +}"#, + ) + .unwrap(); + let spec = discover(tmp.path()); + assert_eq!( + watch_set_for(&spec, "alias", &Default::default()).bus_id, + "alias.worker" + ); + assert_eq!( + watch_set_for(&spec, "other", &Default::default()).bus_id, + "other.worker" + ); +} + +#[test] +fn lexical_paths_drive_store_classification_and_containment() { + let agent_dir = Path::new("/catalog/agents/host/worker"); + + let silent = resolve_local_path( + agent_dir, + "file:///catalog/agents/host/worker/resources/tmp/../context/./journal.md", + ) + .unwrap(); + assert_eq!( + silent, + agent_dir.join("resources/context/journal.md"), + "file URI dot segments are removed before classification" + ); + assert_eq!(classify(agent_dir, "journal", &silent), None); + + let goal = resolve_local_path(agent_dir, "resources/context/.././goal.md").unwrap(); + assert_eq!(goal, agent_dir.join("resources/goal.md")); + assert_eq!( + classify(agent_dir, "notes", &goal), + Some(CarrierClass::Immediate) + ); + + let escaped = + resolve_local_path(agent_dir, "resources/context/../../outside/notes.md").unwrap(); + assert_eq!(escaped, agent_dir.join("outside/notes.md")); + assert_eq!( + classify(agent_dir, "notes", &escaped), + Some(CarrierClass::Coalesced), + "a lexical escape from an authored store is not silent" + ); + assert_eq!( + classify( + agent_dir, + "notes", + Path::new("/catalog/agents/host/worker-copy/resources/context/notes.md"), + ), + Some(CarrierClass::Coalesced), + "path-prefix siblings are not contained by the agent directory" + ); +} + +#[cfg(unix)] +#[test] +fn classification_does_not_follow_symlinks() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path().join("agent"); + std::fs::create_dir_all(agent_dir.join("resources/context")).unwrap(); + std::os::unix::fs::symlink( + agent_dir.join("resources/context"), + agent_dir.join("resources/linked"), + ) + .unwrap(); + + let linked = resolve_local_path(&agent_dir, "resources/linked/journal.md").unwrap(); + assert_eq!( + classify(&agent_dir, "journal", &linked), + Some(CarrierClass::Coalesced), + "classification is lexical and must not canonicalize through the symlink" + ); +} + +#[test] +fn shared_path_refresh_preserves_every_subscription_state() { + let shared = PathBuf::from("/shared/resource.md"); + let previous = BTreeMap::from([( + shared.clone(), + vec![ + Entry { + bus_id: "host.alpha".to_owned(), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("alpha-before".to_owned())), + declaration_summary: None, + occurrence_sequence: 4, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }, + Entry { + bus_id: "host.beta".to_owned(), + seat_id: None, + label: "spec".to_owned(), + class: CarrierClass::Coalesced, + containment_root: None, + state: Some(CarrierState::Present("beta-before".to_owned())), + declaration_summary: None, + occurrence_sequence: 9, + in_flight: false, + parked: false, + dirty: true, + pending_transition: None, + }, + ], + )]); + let sets = vec![ + AgentWatchSet { + declaration_path: PathBuf::from("/catalog/alpha/agent.kdl"), + bus_id: "host.alpha".to_owned(), + seat_id: None, + carriers: vec![WatchableCarrier { + label: "goal".to_owned(), + path: shared.clone(), + class: CarrierClass::Immediate, + containment_root: None, + }], + declaration_summary: None, + }, + AgentWatchSet { + declaration_path: PathBuf::from("/catalog/beta/agent.kdl"), + bus_id: "host.beta".to_owned(), + seat_id: None, + carriers: vec![WatchableCarrier { + label: "spec".to_owned(), + path: shared.clone(), + class: CarrierClass::Coalesced, + containment_root: None, + }], + declaration_summary: None, + }, + ]; + + let rebuilt = rebuild_carriers(previous, refresh_for(sets), &BTreeMap::new(), &mut BTreeMap::new()); + let entries = rebuilt.get(&shared).expect("shared path remains watched"); + assert_eq!(entries.len(), 2); + for (bus_id, digest) in [ + ("host.alpha", "alpha-before"), + ("host.beta", "beta-before"), + ] { + let entry = entries + .iter() + .find(|entry| entry.bus_id == bus_id) + .expect("subscriber remains present"); + assert_eq!( + entry.state, + Some(CarrierState::Present(digest.to_owned())) + ); + assert!(entry.dirty, "pending mutation remains pending for {bus_id}"); + } +} + +#[test] +fn retained_subscription_uses_current_seat_path_and_class() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/alias/worker"); + std::fs::create_dir_all(agent_dir.join("resources")).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "alias" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let goal = agent_dir.join("resources/goal.md"); + std::fs::write(&goal, "current bytes").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "alias").unwrap(); + + let mut current = watch_set_for( + &discover(root.path()), + "alias", + &ResourceProfileRegistry::empty(), + ); + current.seat_id = Some("current-seat".to_owned()); + let old_path = agent_dir.join("resources/old-goal.md"); + let previous = BTreeMap::from([( + old_path.clone(), + vec![Entry { + bus_id: "alias.worker".to_owned(), + seat_id: Some("stale-seat".to_owned()), + label: "goal".to_owned(), + class: CarrierClass::Coalesced, + containment_root: None, + state: Some(CarrierState::Present("old-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 3, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }], + )]); + + let rebuilt = rebuild_carriers(previous, refresh_for(vec![current]), &BTreeMap::new(), &mut BTreeMap::new()); + assert!(!rebuilt.contains_key(&old_path)); + let entry = rebuilt[&goal] + .iter() + .find(|entry| entry.label == "goal") + .expect("the goal subscription remains pending at its current path"); + assert_eq!(entry.bus_id, "alias.worker"); + assert_eq!(entry.seat_id.as_deref(), Some("current-seat")); + assert_eq!(entry.class, CarrierClass::Immediate); + assert_eq!( + entry.state, + Some(CarrierState::Present("old-digest".to_owned())) + ); + assert!(entry.dirty); + + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "alias".to_owned(), + carriers: rebuilt, + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.flush_path_publishing(&goal, Some(CarrierClass::Immediate)); + + let entry = worker.carriers[&goal] + .iter() + .find(|entry| entry.label == "goal") + .unwrap(); + assert!(!entry.dirty, "the current recipient accepted the transition"); + assert!(entry.pending_transition.is_none()); + let events = std::fs::read_dir(agent_dir.join("resources/inbox")) + .unwrap() + .collect::, _>>() + .unwrap(); + assert_eq!(events.len(), 1, "the event routes to the current bus id"); +} + +#[test] +fn pending_retry_keeps_its_original_snapshot_across_path_rebinding() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/alias/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + let old_path = resources.join("old-goal.md"); + let current_path = resources.join("current-goal.md"); + std::fs::write(&old_path, "pending bytes").unwrap(); + std::fs::write(¤t_path, "current rebound bytes").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "alias").unwrap(); + let declaration = agent_dir.join("agent.kdl"); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "alias".to_owned(), + carriers: BTreeMap::from([( + old_path.clone(), + vec![Entry { + bus_id: "alias.worker".to_owned(), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("old-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + + worker.flush_path_publishing(&old_path, None); + let pending = worker.carriers[&old_path][0] + .pending_transition + .clone() + .expect("failed emit retains an immutable transition"); + std::fs::write( + &declaration, + r#"agent "worker" { + host "alias" + command "agent" + resource "goal" uri="resources/current-goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let current = + watch_set_for(&discover(root.path()), "alias", &ResourceProfileRegistry::empty()); + + worker.apply_watch_sets(refresh_for(vec![current])); + worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); + + let event = std::fs::read_dir(resources.join("inbox")) + .unwrap() + .map(|entry| std::fs::read_to_string(entry.unwrap().path()).unwrap()) + .find(|body| body.contains("stream: resync")) + .expect("the retry routes through the refreshed recipient"); + assert!(event.contains(&format!("event-id: {}", pending.event_id)), "{event}"); + assert!(event.contains(&pending.body), "{event}"); + let pending_body: serde_json::Value = serde_json::from_str(&pending.body).unwrap(); + assert_eq!(pending_body["binding"], "goal"); + assert_eq!(pending_body["facts"][0]["before"], "old-digest"); + assert!( + !pending.body.contains(¤t_path.display().to_string()), + "rebinding must not rewrite bytes reserved under the pending event identity" + ); + let entry = &worker.carriers[¤t_path][0]; + assert_eq!(entry.state.as_ref(), Some(&pending.new_state)); + assert!(entry.pending_transition.is_none()); + assert!( + entry.dirty, + "current rebound bytes are queued only after the pending snapshot completes" + ); +} + +#[test] +fn same_directory_path_rebinding_diffs_the_new_path_without_a_filesystem_event() { + let root = tempfile::tempdir().unwrap(); + let parent = root.path().join("resources"); + std::fs::create_dir_all(&parent).unwrap(); + let old_path = parent.join("old.md"); + let new_path = parent.join("new.md"); + std::fs::write(&old_path, "old bytes").unwrap(); + std::fs::write(&new_path, "new bytes").unwrap(); + let declaration = root.path().join("agent.kdl"); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([( + old_path.clone(), + vec![Entry { + bus_id: "host.worker".to_owned(), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: read_state(&old_path, None).ok(), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: false, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::from([(parent.clone(), dir_identity(&parent))]), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + + worker.apply_watch_sets(refresh_for(vec![AgentWatchSet { + declaration_path: declaration, + bus_id: "host.worker".to_owned(), + seat_id: None, + carriers: vec![WatchableCarrier { + label: "goal".to_owned(), + path: new_path.clone(), + class: CarrierClass::Immediate, + containment_root: None, + }], + declaration_summary: None, + }])); + + assert!(!worker.carriers.contains_key(&old_path)); + assert!(worker.carriers[&new_path][0].dirty); + assert!( + worker.deadlines.contains_key(&CarrierClass::Immediate), + "metadata refresh must enqueue the rebound digest even though its parent stayed watched" + ); +} + +#[test] +fn dirty_entry_deadline_migrates_when_refresh_changes_notification_class() { + let root = tempfile::tempdir().unwrap(); + let carrier = root.path().join("carrier.md"); + std::fs::write(&carrier, "same bytes").unwrap(); + let declaration = root.path().join("agent.kdl"); + let old_deadline = Instant::now(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([( + carrier.clone(), + vec![Entry { + bus_id: "host.worker".to_owned(), + seat_id: None, + label: "spec".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: read_state(&carrier, None).ok(), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::from([(CarrierClass::Immediate, old_deadline)]), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + let refresh = |class| { + refresh_for(vec![AgentWatchSet { + declaration_path: declaration.clone(), + bus_id: "host.worker".to_owned(), + seat_id: None, + carriers: vec![WatchableCarrier { + label: "spec".to_owned(), + path: carrier.clone(), + class, + containment_root: None, + }], + declaration_summary: None, + }]) + }; + + worker.apply_watch_sets(refresh(CarrierClass::Coalesced)); + assert!(!worker.deadlines.contains_key(&CarrierClass::Immediate)); + assert!( + worker.deadlines[&CarrierClass::Coalesced] >= old_deadline + COALESCED_WINDOW, + "immediate-to-coalesced migration receives the new class window" + ); + + let coalesced_deadline = worker.deadlines[&CarrierClass::Coalesced]; + worker.apply_watch_sets(refresh(CarrierClass::Immediate)); + assert!(!worker.deadlines.contains_key(&CarrierClass::Coalesced)); + assert!( + worker.deadlines[&CarrierClass::Immediate] < coalesced_deadline, + "coalesced-to-immediate migration is rescheduled under the shorter window" + ); + assert!(worker.carriers[&carrier][0].dirty); +} + +#[test] +fn malformed_declaration_retains_only_an_observed_live_seat_subscription() { + let declaration = PathBuf::from("/catalog/agents/hetz/worker/agent.kdl"); + let previous = || { + BTreeMap::from([( + declaration.clone(), + vec![Entry { + bus_id: "hetz.worker".to_owned(), + seat_id: Some("custom-worker-seat".to_owned()), + label: "declaration".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("before".to_owned())), + declaration_summary: None, + occurrence_sequence: 1, + pending_transition: Some(PendingTransition::new( + "declaration", + &declaration, + &CarrierState::Present("before".to_owned()), + &CarrierState::Present("corrected".to_owned()), + owner_incarnation(1), + 1, + )), + in_flight: false, + parked: false, + dirty: true, + }], + )]) + }; + let malformed_declarations = BTreeSet::from([declaration.clone()]); + + let retained = rebuild_carriers( + previous(), + WatchRefresh { + sets: Vec::new(), + malformed_declarations: malformed_declarations.clone(), + live_task_ids: BTreeSet::from(["custom-worker-seat".to_owned()]), + }, + &BTreeMap::new(), + &mut BTreeMap::new(), + ); + let entry = &retained[&declaration][0]; + assert_eq!( + entry.state, + Some(CarrierState::Present("before".to_owned())) + ); + assert_eq!( + entry + .pending_transition + .as_ref() + .map(|pending| pending.new_state.clone()), + Some(CarrierState::Present("corrected".to_owned())) + ); + + let dropped = rebuild_carriers( + previous(), + WatchRefresh { + sets: Vec::new(), + malformed_declarations, + live_task_ids: BTreeSet::new(), + }, + &BTreeMap::new(), + &mut BTreeMap::new(), + ); + assert!( + dropped.is_empty(), + "a malformed declaration must not retain a watch after its exact seat is no longer live" + ); +} + +#[test] +fn degraded_poll_replays_a_pending_transition_before_newer_bytes() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/hetz/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "hetz" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let carrier = resources.join("goal.md"); + std::fs::write(&carrier, "newer live bytes").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "hetz").unwrap(); + + let set = watch_set_for(&discover(root.path()), "hetz", &Default::default()); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "hetz".to_owned(), + carriers: BTreeMap::from([( + carrier.clone(), + vec![Entry { + bus_id: "hetz.worker".to_owned(), + seat_id: set.seat_id.clone(), + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("old-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 1, + pending_transition: Some(PendingTransition::new( + "goal", + &carrier, + &CarrierState::Present("old-digest".to_owned()), + &CarrierState::Present("pending-target".to_owned()), + owner_incarnation(1), + 1, + )), + in_flight: false, + parked: false, + dirty: false, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + let now = Instant::now(); + worker.apply_watch_sets(refresh_for(vec![set])); + assert!( + !resources.join("inbox").exists(), + "degraded polling must schedule rather than emit during refresh" + ); + worker.flush_due_publishing(now + IMMEDIATE_WINDOW + Duration::from_secs(1)); + + let event = std::fs::read_dir(resources.join("inbox")) + .unwrap() + .map(|entry| std::fs::read_to_string(entry.unwrap().path()).unwrap()) + .find(|body| body.contains("stream: resync")) + .expect("pending transition is replayed"); + assert_eq!(event_field(&event, "old"), "old-digest"); + assert_eq!(event_field(&event, "new"), "pending-targ"); + let entry = &worker.carriers[&carrier][0]; + assert_eq!( + entry.state, + Some(CarrierState::Present("pending-target".to_owned())) + ); + assert!(entry.pending_transition.is_none()); + assert!( + entry.dirty, + "newer live bytes are scheduled only after the pending transition completes" + ); +} + +#[test] +fn fallback_polling_preserves_the_coalesced_window() { + let root = tempfile::tempdir().unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let carrier = root.path().join("carrier.md"); + std::fs::write(&carrier, "before").unwrap(); + let baseline = read_state(&carrier, None).ok(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([( + carrier.clone(), + vec![Entry { + bus_id: "host.missing".to_owned(), + seat_id: None, + label: "spec".to_owned(), + class: CarrierClass::Coalesced, + containment_root: None, + state: baseline.clone(), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: false, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + std::fs::write(&carrier, "after").unwrap(); + let now = Instant::now(); + worker.apply_watch_sets(refresh_for(vec![AgentWatchSet { + declaration_path: PathBuf::from("/catalog/missing/agent.kdl"), + bus_id: "host.missing".to_owned(), + seat_id: None, + carriers: vec![WatchableCarrier { + label: "spec".to_owned(), + path: carrier.clone(), + class: CarrierClass::Coalesced, + containment_root: None, + }], + declaration_summary: None, + }])); + + worker.flush_due_publishing(now + IMMEDIATE_WINDOW + Duration::from_secs(1)); + let entry = &worker.carriers[&carrier][0]; + assert_eq!(entry.state, baseline); + assert!(entry.pending_transition.is_none(), "coalesced emit ran too early"); + + worker.flush_due_publishing(now + COALESCED_WINDOW + Duration::from_secs(1)); + assert!( + worker.carriers[&carrier][0].pending_transition.is_some(), + "the coalesced transition must be attempted after its full window" + ); +} + +#[test] +fn notify_backend_error_rescans_every_carrier_digest() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/hetz/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "hetz" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let goal = resources.join("goal.md"); + std::fs::write(&goal, "before\n").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "hetz").unwrap(); + + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "hetz".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.apply_watch_sets(refresh_for(vec![watch_set_for( + &discover(root.path()), + "hetz", + &Default::default(), + )])); + std::fs::write(&goal, "after\n").unwrap(); + + let (tx, rx) = std::sync::mpsc::channel(); + forward_watch_result(&tx, Err(notify::Error::generic("backend dropped events"))); + match rx.recv().unwrap() { + Msg::Rescan => worker.rescan_all(), + _ => panic!("a notify backend error must request a full digest rescan"), + } + worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW); + + let inbox = resources.join("inbox"); + let events = std::fs::read_dir(inbox) + .unwrap() + .map(|entry| std::fs::read_to_string(entry.unwrap().path()).unwrap()) + .collect::>(); + assert_eq!(events.len(), 1); + assert!(events[0].contains("stream: resync")); + assert!(events[0].contains(r#""binding":"goal""#)); +} + +#[test] +fn registering_another_directory_rediffs_the_untouched_watch_set() { + // A registration change is not free of the subscriptions it does not name: notify's + // macOS FSEvents backend stops the one shared stream on every `watch`, purges the + // device's pending events, and restarts at "since now", so mutations already queued for + // a directory that stayed in the set are destroyed. The state that leaves behind is a + // registered watch that will never report a change that already happened, and the model + // here is exact — the live parents are recorded as covered but never handed to the + // backend, so no event about them can exist. Only re-diffing the whole set recovers it. + let root = tempfile::tempdir().unwrap(); + let write_agent = |identity: &str| { + let agent_dir = root.path().join("agents/alias").join(identity); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + format!( + r#"agent "{identity}" {{ + host "alias" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}}"# + ), + ) + .unwrap(); + let goal = resources.join("goal.md"); + std::fs::write(&goal, "before\n").unwrap(); + (agent_dir, goal) + }; + let (live_dir, live_goal) = write_agent("live"); + write_agent("joining"); + crate::event::publish_owner_binding_for_test(root.path(), "alias").unwrap(); + + let specs = crate::discover_strict(root.path()).specs; + let set_for = |identity: &str| { + let spec = specs + .iter() + .find(|spec| spec.path.starts_with(root.path().join("agents/alias").join(identity))) + .expect("both declarations are valid"); + watch_set_for(spec, "alias", &ResourceProfileRegistry::empty()) + }; + let live_set = set_for("live"); + let joining_set = set_for("joining"); + + let (tx, _rx) = channel::(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "alias".to_owned(), + carriers: rebuild_carriers( + BTreeMap::new(), + refresh_for(vec![live_set.clone()]), + &BTreeMap::new(), + &mut BTreeMap::new(), + ), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: make_watcher(tx), + emit: Arc::new(EmitQueue::default()), + }; + worker.watched = worker + .carriers + .keys() + .filter_map(|path| path.parent()) + .map(|dir| (dir.to_path_buf(), dir_identity(dir))) + .collect(); + + std::fs::write(&live_goal, "changed with no watch able to report it\n").unwrap(); + + // The joining seat contributes directories the backend has not seen, so this refresh + // changes the registration set without touching the live subscription's own paths. + worker.apply_watch_sets(refresh_for(vec![live_set, joining_set])); + worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); + + let event = resync_inbox_event(&live_dir); + assert_eq!(event_field(&event, "binding"), "goal"); + // The joining seat has no inbox at all: its baseline seeded silently, as a new + // subscription must, so the rescan is not simply emitting for everything it re-reads. + assert!( + !root + .path() + .join("agents/alias/joining/resources/inbox") + .exists(), + "the joining seat seeds its baseline silently" + ); +} + +#[cfg(unix)] +#[test] +fn digesting_a_fifo_fails_without_blocking_the_worker() { + use std::os::unix::ffi::OsStrExt as _; + + let tmp = tempfile::tempdir().unwrap(); + let fifo = tmp.path().join("carrier.fifo"); + let fifo_c = std::ffi::CString::new(fifo.as_os_str().as_bytes()).unwrap(); + // SAFETY: the path is NUL-terminated and points into the live temp directory. + assert_eq!(unsafe { libc::mkfifo(fifo_c.as_ptr(), 0o600) }, 0); + assert_eq!(read_state(&fifo, None).unwrap(), CarrierState::Missing); + assert_eq!( + read_state(&fifo, Some(tmp.path())).unwrap(), + CarrierState::Missing, + "confined carrier reads must reject a FIFO without blocking too" + ); +} + +#[test] +fn due_flush_only_clears_subscribers_of_the_due_class() { + let root = tempfile::tempdir().unwrap(); + let carrier = root.path().join("carrier.md"); + std::fs::write(&carrier, "same bytes").unwrap(); + let state = read_state(&carrier, None).ok(); + let entries = [CarrierClass::Immediate, CarrierClass::Coalesced] + .into_iter() + .map(|class| Entry { + bus_id: "host.worker".to_owned(), + seat_id: None, + label: format!("{class:?}"), + class, + containment_root: None, + state: state.clone(), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }) + .collect(); + let now = Instant::now(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([(carrier, entries)]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::from([(CarrierClass::Immediate, now)]), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + + worker.flush_due_publishing(now); + let entries = worker.carriers.values().next().unwrap(); + assert!(!entries[0].dirty); + assert!(entries[1].dirty, "coalesced subscriber must wait for its own deadline"); +} + +#[cfg(unix)] +#[test] +fn confined_read_refuses_a_symlink_created_after_resolution() { + let agent_dir = tempfile::tempdir().expect("agent directory"); + let resources = agent_dir.path().join("resources"); + std::fs::create_dir(&resources).expect("resources directory"); + let carrier = resources.join("goal.md"); + std::fs::write(&carrier, "inside").expect("inside carrier"); + assert!( + matches!( + read_state(&carrier, Some(agent_dir.path())), + Ok(CarrierState::Present(_)) + ), + "ordinary files beneath the admitted root remain readable" + ); + + std::fs::remove_file(&carrier).expect("remove inside carrier"); + let outside = tempfile::NamedTempFile::new().expect("outside carrier"); + std::fs::write(outside.path(), "external").expect("outside bytes"); + std::os::unix::fs::symlink(outside.path(), &carrier) + .expect("replace absent carrier with an external symlink"); + assert_eq!( + read_state(&carrier, Some(agent_dir.path())).unwrap(), + CarrierState::Missing + ); + + std::fs::remove_file(&carrier).expect("remove final symlink"); + std::fs::remove_dir(&resources).expect("remove resources directory"); + let outside_dir = tempfile::tempdir().expect("outside directory"); + std::fs::write(outside_dir.path().join("goal.md"), "external").expect("outside carrier"); + std::os::unix::fs::symlink(outside_dir.path(), &resources) + .expect("replace absent ancestor with an external symlink"); + assert_eq!( + read_state(&carrier, Some(agent_dir.path())).unwrap(), + CarrierState::Missing + ); +} + +#[cfg(unix)] +#[test] +fn confined_read_refuses_a_symlinked_confinement_root_ancestor() { + let temp = tempfile::tempdir().expect("outer directory"); + let real_root = temp.path().join("real/agent"); + std::fs::create_dir_all(&real_root).expect("real agent directory"); + std::fs::write(real_root.join("goal.md"), "outside admitted ancestry") + .expect("carrier bytes"); + let alias = temp.path().join("alias"); + std::os::unix::fs::symlink(temp.path().join("real"), &alias) + .expect("symlinked root ancestor"); + let admitted_root = alias.join("agent"); + assert_eq!( + read_state(&admitted_root.join("goal.md"), Some(&admitted_root)).unwrap(), + CarrierState::Missing, + "every component of the confinement root must be opened without following symlinks" + ); +} +#[test] +fn deletion_and_same_byte_recreation_are_distinct_carrier_transitions() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/host/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "host" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let carrier = resources.join("goal.md"); + std::fs::write(&carrier, "same bytes").unwrap(); + let CarrierState::Present(original_digest) = read_state(&carrier, None).unwrap() else { + panic!("regular carrier has a digest"); + }; + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let set = watch_set_for(&discover(root.path()), "host", &Default::default()); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.apply_watch_sets(refresh_for(vec![set])); + + std::fs::remove_file(&carrier).unwrap(); + worker.flush_path_publishing(&carrier, None); + let deletion = resync_inbox_events(&agent_dir); + assert_eq!(deletion.len(), 1); + assert_eq!( + event_field(&deletion[0], "old"), + original_digest.chars().take(12).collect::() + ); + assert_eq!(event_field(&deletion[0], "new"), "missing"); + assert!(event_field(&deletion[0], "occurrence").ends_with(":1")); + assert_eq!( + worker.carriers[&carrier][0].state, + Some(CarrierState::Missing) + ); + + worker.flush_path_publishing(&carrier, None); + assert_eq!( + resync_inbox_events(&agent_dir).len(), + 1, + "repeated missing observations are silent" + ); + + std::fs::write(&carrier, "same bytes").unwrap(); + worker.flush_path_publishing(&carrier, None); + let events = resync_inbox_events(&agent_dir); + assert_eq!( + events.len(), + 1, + "creation supersedes the tombstone under the binding key" + ); + let creation = &events[0]; + assert_eq!(event_field(creation, "old"), "missing"); + assert!(event_field(creation, "occurrence").ends_with(":2")); +} + +#[test] +#[cfg(unix)] +fn transient_permission_error_retries_without_emitting_a_tombstone() { + use std::os::unix::fs::PermissionsExt as _; + + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/host/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "host" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let carrier = resources.join("goal.md"); + std::fs::write(&carrier, "before").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let set = watch_set_for(&discover(root.path()), "host", &Default::default()); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.apply_watch_sets(refresh_for(vec![set])); + let baseline = worker.carriers[&carrier][0].state.clone(); + + let original_permissions = std::fs::metadata(&carrier).unwrap().permissions(); + std::fs::set_permissions(&carrier, std::fs::Permissions::from_mode(0)).unwrap(); + worker.flush_path_publishing(&carrier, None); + let entry = &worker.carriers[&carrier][0]; + assert_eq!(entry.state, baseline); + assert!(entry.pending_transition.is_none()); + assert!(entry.dirty); + assert!(!resources.join("inbox").exists()); + + std::fs::set_permissions(&carrier, original_permissions).unwrap(); + std::fs::write(&carrier, "after").unwrap(); + worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); + let event = resync_inbox_event(&agent_dir); + assert_ne!(event_field(&event, "new"), "missing"); +} + +#[test] +#[cfg(unix)] +fn initial_transient_read_failure_schedules_a_baseline_retry() { + use std::os::unix::fs::PermissionsExt as _; + + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/host/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "host" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let carrier = resources.join("goal.md"); + std::fs::write(&carrier, "baseline").unwrap(); + let original_permissions = std::fs::metadata(&carrier).unwrap().permissions(); + std::fs::set_permissions(&carrier, std::fs::Permissions::from_mode(0)).unwrap(); + let set = watch_set_for(&discover(root.path()), "host", &Default::default()); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::from([(resources.clone(), dir_identity(&resources))]), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + + worker.apply_watch_sets(refresh_for(vec![set])); + let entry = worker.carriers[&carrier] + .iter() + .find(|entry| entry.label == "goal") + .unwrap(); + assert_eq!(entry.state, None); + assert!(entry.dirty); + assert!(worker.deadlines.contains_key(&CarrierClass::Immediate)); + + std::fs::set_permissions(&carrier, original_permissions).unwrap(); + worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); + let entry = worker.carriers[&carrier] + .iter() + .find(|entry| entry.label == "goal") + .unwrap(); + assert!(matches!(entry.state, Some(CarrierState::Present(_)))); + assert!(!entry.dirty); + assert!(!resources.join("inbox").exists()); +} + +#[test] +fn reinstalled_subscription_keeps_occurrence_identity_without_an_active_watch() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/host/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "host" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let carrier = resources.join("goal.md"); + std::fs::write(&carrier, "A").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let set = watch_set_for(&discover(root.path()), "host", &Default::default()); + let seen_subscription_count = set.carriers.len(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.apply_watch_sets(refresh_for(vec![set.clone()])); + + std::fs::write(&carrier, "B").unwrap(); + worker.flush_path_publishing(&carrier, None); + let before_suspend = resync_inbox_event(&agent_dir); + + // Suspension removes every carrier and watch, while retaining one scalar sequence floor + // for each declaration/binding identity seen during this supervisor incarnation. + worker + .watched + .insert(resources.clone(), dir_identity(&resources)); + worker.apply_watch_sets(refresh_for(Vec::new())); + assert!(worker.carriers.is_empty()); + assert!(worker.watched.is_empty()); + assert_eq!( + worker.subscription_sequences.len(), + seen_subscription_count + ); + + std::fs::write(&carrier, "A").unwrap(); + worker.apply_watch_sets(refresh_for(vec![set])); + let resumed = worker.carriers[&carrier] + .iter() + .find(|entry| entry.label == "goal") + .unwrap(); + assert_eq!(resumed.occurrence_sequence, 1); + std::fs::write(&carrier, "B").unwrap(); + worker.flush_path_publishing(&carrier, None); + let after_resume = resync_inbox_event(&agent_dir); + + assert_eq!( + event_field(&before_suspend, "old"), + event_field(&after_resume, "old") + ); + assert_eq!( + event_field(&before_suspend, "new"), + event_field(&after_resume, "new") + ); + assert_ne!( + event_field(&before_suspend, "event-id"), + event_field(&after_resume, "event-id"), + "the post-resume A→B occurrence must not deduplicate against the pre-suspend one" + ); + assert!(event_field(&before_suspend, "occurrence").ends_with(":1")); + assert!(event_field(&after_resume, "occurrence").ends_with(":2")); +} + +#[test] +fn relocated_subscription_keeps_occurrence_sequence_in_the_recipient_namespace() { + let root = tempfile::tempdir().unwrap(); + let agent_dir = root.path().join("agents/host/worker"); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + r#"agent "worker" { + host "host" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let original_carrier = resources.join("goal.md"); + let relocated_carrier = resources.join("relocated-goal.md"); + std::fs::write(&original_carrier, "A").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let set = + watch_set_for(&discover(root.path()), "host", &ResourceProfileRegistry::empty()); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::new(), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + worker.apply_watch_sets(refresh_for(vec![set.clone()])); + + std::fs::write(&original_carrier, "B").unwrap(); + worker.flush_path_publishing(&original_carrier, None); + let first_a_to_b = resync_inbox_event(&agent_dir); + + std::fs::write(&relocated_carrier, "A").unwrap(); + let mut relocated = set; + relocated.declaration_path = agent_dir.join("relocated/agent.kdl"); + for carrier in &mut relocated.carriers { + if carrier.label == "declaration" { + carrier.path = relocated.declaration_path.clone(); + } else if carrier.label == "goal" { + carrier.path = relocated_carrier.clone(); + } + } + worker.apply_watch_sets(refresh_for(vec![relocated.clone()])); + let rebound = worker.carriers[&relocated_carrier] + .iter() + .find(|entry| entry.label == "goal") + .unwrap(); + assert_eq!(rebound.occurrence_sequence, 1); + assert_eq!( + rebound.state, + read_state(&original_carrier, None).ok() + ); + + worker.flush_path_publishing(&relocated_carrier, None); + let back_to_a = resync_inbox_event(&agent_dir); + assert_eq!(event_field(&back_to_a, "old"), event_field(&first_a_to_b, "new")); + assert_eq!(event_field(&back_to_a, "new"), event_field(&first_a_to_b, "old")); + assert!(event_field(&back_to_a, "occurrence").ends_with(":2")); + + std::fs::write(&relocated_carrier, "B").unwrap(); + worker.flush_path_publishing(&relocated_carrier, None); + let second_a_to_b = resync_inbox_event(&agent_dir); + assert_eq!( + event_field(&first_a_to_b, "old"), + event_field(&second_a_to_b, "old") + ); + assert_eq!( + event_field(&first_a_to_b, "new"), + event_field(&second_a_to_b, "new") + ); + assert_ne!( + event_field(&first_a_to_b, "event-id"), + event_field(&second_a_to_b, "event-id") + ); + assert!(event_field(&first_a_to_b, "occurrence").ends_with(":1")); + assert!(event_field(&second_a_to_b, "occurrence").ends_with(":3")); + + relocated.bus_id = "host.replacement".to_owned(); + worker.apply_watch_sets(refresh_for(vec![relocated])); + assert_eq!( + worker.carriers[&relocated_carrier] + .iter() + .find(|entry| entry.label == "goal") + .unwrap() + .occurrence_sequence, + 0, + "a different recipient starts a distinct deduplication namespace" + ); +} + +#[test] +fn subscribers_advance_occurrence_sequences_independently() { + let root = tempfile::tempdir().unwrap(); + let carrier = root.path().join("shared.md"); + std::fs::write(&carrier, "new bytes").unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let entries = ["host.alpha", "host.beta"] + .into_iter() + .map(|bus_id| Entry { + bus_id: bus_id.to_owned(), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("old-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }) + .collect(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([(carrier.clone(), entries)]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + + worker.flush_path_publishing(&carrier, None); + + let entries = &worker.carriers[&carrier]; + assert_eq!(entries[0].occurrence_sequence, 1); + assert_eq!(entries[1].occurrence_sequence, 1); + assert_eq!( + event_field(&entries[0].pending_transition.as_ref().unwrap().body, "occurrence"), + event_field(&entries[1].pending_transition.as_ref().unwrap().body, "occurrence"), + "one subscriber must not consume sequence numbers from another" + ); +} + +#[test] +fn supervisor_restart_incarnation_changes_the_occurrence_namespace() { + let first = PendingTransition::new( + "goal", + Path::new("/agent/goal.md"), + &CarrierState::Present("old".to_owned()), + &CarrierState::Present("new".to_owned()), + owner_incarnation(1), + 1, + ); + let restarted = PendingTransition::new( + "goal", + Path::new("/agent/goal.md"), + &CarrierState::Present("old".to_owned()), + &CarrierState::Present("new".to_owned()), + owner_incarnation(2), + 1, + ); + + assert_ne!(first.body, restarted.body); + assert_ne!(first.event_id, restarted.event_id); +} + +#[test] +fn failed_tombstone_emit_retains_present_state_and_immutable_retry_snapshot() { + let root = tempfile::tempdir().unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let carrier = root.path().join("carrier.md"); + std::fs::write(&carrier, "old bytes").unwrap(); + let mut worker = Worker { + root: root.path().to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([( + carrier.clone(), + vec![Entry { + bus_id: "host.missing".to_owned(), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("old-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + }; + std::fs::remove_file(&carrier).unwrap(); + + worker.flush_path_publishing(&carrier, None); + let pending_transition = worker.carriers[&carrier][0] + .pending_transition + .clone() + .expect("failed transition snapshot is retained"); + assert_eq!(pending_transition.new_state, CarrierState::Missing); + assert_eq!(event_field(&pending_transition.body, "old"), "old-digest"); + assert_eq!(event_field(&pending_transition.body, "new"), "missing"); + assert_eq!(worker.carriers[&carrier][0].occurrence_sequence, 1); + std::fs::write(&carrier, "old bytes").unwrap(); + worker.flush_path_publishing(&carrier, None); + let entry = &worker.carriers[&carrier][0]; + assert_eq!(entry.occurrence_sequence, 1); + assert_eq!( + entry.state, + Some(CarrierState::Present("old-digest".to_owned())) + ); + assert_eq!( + entry.pending_transition.as_ref(), + Some(&pending_transition), + "a retry must replay the tombstone snapshot even after the carrier is recreated" + ); + assert!(entry.dirty); + assert!(worker.deadlines.contains_key(&CarrierClass::Immediate)); +} + +fn handoff_worker(root: &Path, carrier: &Path, recipients: &[&str]) -> Worker { + crate::event::publish_owner_binding_for_test(root, "host").unwrap(); + std::fs::write(carrier, "current bytes").unwrap(); + Worker { + root: root.to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([( + carrier.to_path_buf(), + recipients + .iter() + .map(|bus_id| Entry { + bus_id: (*bus_id).to_owned(), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("stale-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }) + .collect(), + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + } +} + +/// One agent declaration plus its goal carrier, and a subscription to that carrier whose +/// baseline is stale, so the next flush captures exactly one transition for it. +/// +/// A subscription exists only for a canonical seat a pass proved alive, so a subscription +/// addressed to a declaration that says `suspended` IS the suspended-and-running state — the +/// one that produced ~2010 refusals per seat in #431. A seat whose task had already exited +/// produces no subscription at all, which is why a fixture built on one proves nothing. +fn declared_recipient_worker(root: &Path, identity: &str, declaration_extra: &str) -> Worker { + crate::event::publish_owner_binding_for_test(root, "host").unwrap(); + let agent_dir = root.join("agents/host").join(identity); + std::fs::create_dir_all(agent_dir.join("resources")).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + format!( + "agent \"{identity}\" {{\n host \"host\"\n command \"agent\"\n{declaration_extra} resource \"goal\" uri=\"resources/goal.md\" reason=\"Mission.\"\n}}" + ), + ) + .unwrap(); + let goal = agent_dir.join("resources/goal.md"); + std::fs::write(&goal, "current bytes").unwrap(); + Worker { + root: root.to_path_buf(), + this_host: "host".to_owned(), + carriers: BTreeMap::from([( + goal, + vec![Entry { + bus_id: format!("host.{identity}"), + seat_id: None, + label: "goal".to_owned(), + class: CarrierClass::Immediate, + containment_root: None, + state: Some(CarrierState::Present("stale-digest".to_owned())), + declaration_summary: None, + occurrence_sequence: 0, + pending_transition: None, + in_flight: false, + parked: false, + dirty: true, + }], + )]), + subscription_sequences: BTreeMap::new(), + parked_transitions: BTreeMap::new(), + deadlines: BTreeMap::new(), + watched: BTreeMap::new(), + watcher: None, + emit: Arc::new(EmitQueue::default()), + } +} + +/// A recipient that refuses because it is not running keeps its reservation and is attempted +/// exactly once. Retrying it cannot make it admissible: eligibility was resolved under the +/// catalog-authoring lock, and each attempt re-resolves the whole catalog to reach the same +/// answer. That retry is the CPU burn and the shared-lock coverage of #431. +#[test] +fn a_not_running_recipient_parks_its_reservation_and_is_attempted_once() { + let root = tempfile::tempdir().unwrap(); + let mut worker = declared_recipient_worker( + root.path(), + "worker", + " desired-state \"suspended\" reason=\"Waiting for capacity\"\n", + ); + let goal = worker.carriers.keys().next().unwrap().clone(); + + worker.flush_path_publishing(&goal, None); + + let entry = &worker.carriers[&goal][0]; + assert!(entry.parked, "the subscription must be parked"); + assert!( + entry.pending_transition.is_none(), + "the reservation moves out of the subscription, which a refresh will drop" + ); + assert!( + worker + .parked_transitions + .contains_key(&("host.worker".to_owned(), "goal".to_owned())), + "the reservation must be retained, not dropped: {:?}", + worker.parked_transitions.keys().collect::>() + ); + assert!( + worker.deadlines.is_empty(), + "a not-running recipient must schedule no retry deadline: {:?}", + worker.deadlines + ); + + // Each source that would ordinarily re-arm this carrier, checked on its own: a parked + // subscription must schedule nothing, not merely publish nothing. + std::fs::write(&goal, "changed while the recipient is suspended\n").unwrap(); + worker.mark_mutated(vec![goal.clone()]); + assert!( + worker.deadlines.is_empty() && !worker.carriers[&goal][0].dirty, + "a mutation wakeup must not schedule a parked subscription: {:?}", + worker.deadlines + ); + worker.rescan_all(); + assert!( + worker.deadlines.is_empty() && !worker.carriers[&goal][0].dirty, + "a rescan must not schedule a parked subscription: {:?}", + worker.deadlines + ); + worker.flush_path_publishing(&goal, None); + worker.flush_due_publishing(Instant::now() + COALESCED_WINDOW + Duration::from_secs(1)); + assert!(worker.deadlines.is_empty(), "{:?}", worker.deadlines); + assert_eq!( + worker.emit.handed_off(), + 1, + "a parked reservation must not be attempted again" + ); +} + +/// The reservation re-arms when its recipient is running again, and replays the exact bytes +/// it reserved. Dropping it at the refusal would lose a resync the agent should see on +/// resume; a refresh drops the suspended subscription itself, so the reservation has to +/// outlive it. +#[test] +fn a_parked_reservation_re_arms_and_replays_when_its_recipient_runs_again() { + let root = tempfile::tempdir().unwrap(); + let mut worker = declared_recipient_worker( + root.path(), + "worker", + " desired-state \"suspended\" reason=\"Waiting for capacity\"\n", + ); + let goal = worker.carriers.keys().next().unwrap().clone(); + let agent_dir = root.path().join("agents/host/worker"); + + worker.flush_path_publishing(&goal, None); + let reserved = worker + .parked_transitions + .values() + .next() + .expect("the refusal retains its reservation") + .clone(); + + std::fs::write( + agent_dir.join("agent.kdl"), + "agent \"worker\" {\n host \"host\"\n command \"agent\"\n resource \"goal\" uri=\"resources/goal.md\" reason=\"Mission.\"\n}", + ) + .unwrap(); + let resumed = watch_set_for( + &discover(root.path()), + "host", + &ResourceProfileRegistry::empty(), + ); + worker.apply_watch_sets(refresh_for(vec![resumed])); + + let entry = &worker.carriers[&goal][0]; + assert!(!entry.parked, "a carried recipient is running"); + assert_eq!( + entry.pending_transition.as_ref().map(|held| &held.event_id), + Some(&reserved.event_id), + "the restored reservation must be the reserved one" + ); + assert!( + worker.deadlines.contains_key(&CarrierClass::Immediate), + "re-arming must schedule the carrier's class: {:?}", + worker.deadlines + ); + assert!(worker.parked_transitions.is_empty()); + + worker.flush_due_publishing(Instant::now() + IMMEDIATE_WINDOW + Duration::from_secs(1)); + let event = resync_inbox_event(&agent_dir); + assert!( + event.contains(&format!("event-id: {}", reserved.event_id)), + "{event}" + ); + assert!(event.contains(&reserved.body), "{event}"); +} + +/// A refusal no retry can admit drops its reservation and advances the baseline, so the same +/// carrier transition is not captured again on the next observation. Without advancing it, +/// "drop" would only mean "re-capture and refuse again". +#[test] +fn a_permanently_refused_reservation_is_dropped_and_not_recaptured() { + let root = tempfile::tempdir().unwrap(); + let mut worker = declared_recipient_worker(root.path(), "worker", ""); + // A second declaration of the same identity makes the recipient ambiguous. No retry can + // resolve that from the publisher's side. + let twin = root.path().join("agents/host/worker-copy"); + std::fs::create_dir_all(&twin).unwrap(); + std::fs::write( + twin.join("agent.kdl"), + "agent \"worker\" {\n host \"host\"\n command \"agent\"\n}", + ) + .unwrap(); + let goal = worker.carriers.keys().next().unwrap().clone(); + + worker.flush_path_publishing(&goal, None); + + let entry = &worker.carriers[&goal][0]; + assert!( + entry.pending_transition.is_none(), + "a permanent refusal drops its reservation" + ); + assert!(!entry.parked, "a permanent refusal is not parked"); + assert!(worker.parked_transitions.is_empty()); + assert!( + worker.deadlines.is_empty(), + "a permanent refusal must schedule no retry: {:?}", + worker.deadlines + ); + + worker.rescan_all(); + worker.flush_due_publishing(Instant::now() + COALESCED_WINDOW + Duration::from_secs(1)); + assert_eq!( + worker.emit.handed_off(), + 1, + "the dropped transition must not be recaptured" + ); +} + +/// Every refusal the catalog can return, classified. `RecipientNotRunning` is the only one a +/// resume can admit; the rest never become admissible. An unknown recipient and a catalog +/// mid-edit stay retryable on purpose: a declaration being replaced by rename is briefly +/// absent, and dropping the reservation there would lose a resync nothing was wrong with. +#[test] +fn refusals_are_classified_by_what_could_admit_them_later() { + use crate::event::{RefusalKind, refusal_kind}; + + let root = tempfile::tempdir().unwrap(); + crate::event::publish_owner_binding_for_test(root.path(), "host").unwrap(); + let write = |identity: &str, body: &str| { + let dir = root.path().join("agents/host").join(identity); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write(dir.join("agent.kdl"), body).unwrap(); + }; + let emit = |recipient: &str| { + crate::event::emit_builtin_resync( + root.path(), + "host", + recipient, + "eventid", + Some("goal"), + Some("subject"), + "{}", + true, + ) + .expect_err("every recipient in this fixture refuses") + }; + + write( + "suspended", + "agent \"suspended\" {\n host \"host\"\n command \"agent\"\n desired-state \"suspended\" reason=\"Waiting\"\n}", + ); + write( + "foreign", + "agent \"foreign\" {\n host \"elsewhere\"\n command \"agent\"\n}", + ); + write("plain", "agent \"plain\" {\n host \"host\"\n command \"agent\"\n}"); + write( + "twin", + "agent \"plain\" {\n host \"host\"\n command \"agent\"\n}", + ); + write("solo", "agent \"solo\" {\n host \"host\"\n command \"agent\"\n}"); + + let suspended = emit("host.suspended"); + assert!( + suspended.to_string().contains("eyes are closed"), + "{suspended:#}" + ); + assert_eq!( + refusal_kind(&suspended), + Some(RefusalKind::RecipientNotRunning) + ); + + // A foreign spec is only reachable under its own host-qualified bus id; by local identity + // it is simply not in this host's namespace. + let foreign = emit("elsewhere.foreign"); + assert!( + foreign + .to_string() + .contains("event publication must run on that host"), + "{foreign:#}" + ); + assert_eq!(refusal_kind(&foreign), Some(RefusalKind::Permanent)); + + let ambiguous = emit("host.plain"); + assert!(ambiguous.to_string().contains("is ambiguous"), "{ambiguous:#}"); + assert_eq!(refusal_kind(&ambiguous), Some(RefusalKind::Permanent)); + + let undeclared = crate::event::emit( + root.path(), + "host", + "host.solo", + "other", + "eventid", + None, + None, + "{}", + true, + ) + .expect_err("an undeclared stream is refused"); + assert_eq!(refusal_kind(&undeclared), Some(RefusalKind::Permanent)); + + let unknown = emit("host.absent"); + assert!(unknown.to_string().contains("no agent"), "{unknown:#}"); + assert_eq!( + refusal_kind(&unknown), + None, + "an absent declaration may be a rename in flight, so it stays retryable" + ); +} + +/// One outstanding publication per subscription. Flushing again while a publication is still +/// in flight must not hand off the same subscription twice: whether its reserved event +/// identity is spent is decided by an outcome that has not returned yet. +#[test] +fn a_flush_never_hands_off_a_subscription_whose_publication_is_outstanding() { + let root = tempfile::tempdir().unwrap(); + let carrier = root.path().join("carrier.md"); + let mut worker = handoff_worker(root.path(), &carrier, &["host.missing"]); + + worker.flush_path(&carrier, None); + assert_eq!(worker.emit.queued_recipients(), ["host.missing"]); + assert!(worker.carriers[&carrier][0].in_flight); + + std::fs::write(&carrier, "newer bytes while the publication is outstanding").unwrap(); + worker.mark_mutated(vec![carrier.clone()]); + worker.flush_path(&carrier, None); + assert_eq!( + worker.emit.queued_recipients(), + ["host.missing"], + "an outstanding publication must not be handed off a second time" + ); +} + +/// A deactivation acknowledgement is the reconcile pass's guarantee that nothing starts +/// publishing to that recipient afterwards. A publication still queued when the seat is +/// deactivated is dropped, and only that recipient's. +#[test] +fn deactivation_drops_only_that_recipients_queued_publication() { + let root = tempfile::tempdir().unwrap(); + let carrier = root.path().join("carrier.md"); + let mut worker = handoff_worker(root.path(), &carrier, &["host.leaving", "host.staying"]); + + worker.flush_path(&carrier, None); + assert_eq!( + worker.emit.queued_recipients(), + ["host.leaving", "host.staying"] + ); + + worker.deactivate_watch_set("host.leaving"); + assert_eq!(worker.emit.queued_recipients(), ["host.staying"]); +} + +/// A refresh that drops a subscription — the suspended recipient of #431 — must drop its +/// queued publication too. The pass has already decided that seat receives no events, and a +/// queued refusal published afterwards is work no outcome can apply. +#[test] +fn a_refresh_drops_a_queued_publication_for_a_subscription_it_removed() { + let root = tempfile::tempdir().unwrap(); + let carrier = root.path().join("carrier.md"); + let mut worker = handoff_worker(root.path(), &carrier, &["host.missing"]); + + worker.flush_path(&carrier, None); + assert_eq!(worker.emit.queued_recipients(), ["host.missing"]); + + worker.apply_watch_sets(refresh_for(Vec::new())); + assert!(worker.carriers.is_empty()); + assert!( + worker.emit.queued_recipients().is_empty(), + "a dropped subscription's queued publication must not survive the refresh" + ); +} + +#[test] +fn transition_identity_covers_every_rendered_transition_dimension() { + let topics = vec!["content".to_owned()]; + let facts = + vec![ResourceFact::transition("digest", Some("old"), Some("new")).unwrap()]; + let baseline = render_body("goal", &topics, &facts, "v1:1:2:42:3:1"); + assert_eq!( + transition_identity(&baseline), + transition_identity(&baseline), + "replaying one canonical body must reproduce its identity" + ); + + let changed_facts = + vec![ResourceFact::transition("digest", Some("old"), Some("other")).unwrap()]; + for (dimension, changed) in [ + ( + "binding", + render_body("spec", &topics, &facts, "v1:1:2:42:3:1"), + ), + ( + "topic", + render_body("goal", &["other".to_owned()], &facts, "v1:1:2:42:3:1"), + ), + ( + "fact", + render_body("goal", &topics, &changed_facts, "v1:1:2:42:3:1"), + ), + ( + "occurrence", + render_body("goal", &topics, &facts, "v1:1:2:42:3:2"), + ), + ] { + assert_ne!( + transition_identity(&baseline), + transition_identity(&changed), + "changing {dimension} must change the event identity" + ); + } +} + +#[test] +fn local_path_resolution_parses_supported_file_uris_without_uri_metadata_bytes() { + let agent_dir = Path::new("/cat/agents/hetz/w"); + assert_eq!( + resolve_local_path(agent_dir, "file:///etc/demo.kdl"), + Some(PathBuf::from("/etc/demo.kdl")) + ); + for scheme in ["file", "FILE", "FiLe"] { + assert_eq!( + resolve_local_path(agent_dir, &format!("{scheme}:///etc/demo.kdl")), + Some(PathBuf::from("/etc/demo.kdl")) + ); + } + assert_eq!( + resolve_local_path(agent_dir, "file:///tmp/with%20space/%E2%82%AC.md"), + Some(PathBuf::from("/tmp/with space/€.md")) + ); + assert_eq!( + resolve_local_path(agent_dir, "file:///tmp/literal%3Fmark"), + Some(PathBuf::from("/tmp/literal?mark")) + ); + assert_eq!( + resolve_local_path(agent_dir, "file:///"), + Some(PathBuf::from("/")) + ); + assert_eq!( + resolve_local_path(agent_dir, "resources/journal.md"), + Some(agent_dir.join("resources/journal.md")) + ); + assert_eq!( + resolve_local_path( + agent_dir, + "resources/with%20space/%E2%82%AC-journal.md" + ), + Some(agent_dir.join("resources/with space/€-journal.md")) + ); + + for unsupported in [ + "file://authority/etc/demo.kdl", + "file:////authority/etc/demo.kdl", + "file:///etc/demo.kdl?revision=2", + "file:///etc/demo.kdl#section", + "file:///tmp/encoded%2Fseparator", + "file:///tmp/encoded%2fseparator", + "file:///tmp/encoded%5Cseparator", + "file:///tmp/bad%escape", + "file:///tmp/%2E%2E/escape", + "file:///tmp/a%00b", + "file:///tmp/%FF.md", + "resources/encoded%2Fseparator", + "resources/encoded%5Cseparator", + "resources/%2E%2E/outside.md", + "resources/bad%escape", + "resources/a%00b", + "resources/%FF.md", + "http://x/y", + "worktree://repo/main", + "GitHub-Issue://org/repo/41", + "ProFiLe:opaque", + ] { + assert_eq!( + resolve_local_path(agent_dir, unsupported), + None, + "{unsupported} must not become filesystem bytes" + ); + } +} + +#[test] +fn classification_is_goal_immediate_stores_silent_other_coalesced() { + let agent_dir = Path::new("/cat/agents/hetz/w"); + assert_eq!( + classify(agent_dir, "mission", &agent_dir.join("resources/goal.md")), + Some(CarrierClass::Immediate), + "basename goal.md is immediate regardless of binding name" + ); + assert_eq!( + classify( + agent_dir, + "journal", + &agent_dir.join("resources/context/journal.md") + ), + None, + "agent-authored stores are silent and excluded from the watch set" + ); + assert_eq!( + classify( + agent_dir, + "decision-log", + &agent_dir.join("resources/decisions/x.md") + ), + None + ); + assert_eq!( + classify(agent_dir, "spec", Path::new("/etc/demo/spec.md")), + Some(CarrierClass::Coalesced) + ); +} + +#[test] +fn profile_classes_map_onto_carrier_notification() { + assert_eq!(carrier_class(ProfileClass::Immediate), Some(CarrierClass::Immediate)); + assert_eq!(carrier_class(ProfileClass::Coalesced), Some(CarrierClass::Coalesced)); + assert_eq!( + carrier_class(ProfileClass::Silent), + None, + "silent profiles are excluded from the watch set like sniffed authored stores" + ); +} + +#[test] +fn registered_profile_failures_are_reported_while_other_bindings_survive() { + let tmp = tempfile::tempdir().unwrap(); + let dir = tmp.path().join("agents/hetz/worker"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("agent.kdl"), + r#"agent "worker" { + host "hetz" + command "true" + resource "goal" uri="dev.schickling.agent-goal://hetz/worker" reason="Mission." + resource "issue" uri="worktree://repo/main" reason="Opaque scheme." +}"#, + ) + .unwrap(); + + let broken = tmp.path().join("broken.wasm"); + std::fs::write(&broken, b"not a module").unwrap(); + let profiles = ResourceProfileRegistry::empty().with_profile( + agent_spec::ResourceProfile::wasm( + "dev.schickling.agent-goal", + &broken, + ProfileClass::Coalesced, + ), + ); + let refresh = profiles.begin_refresh(); + let spec = discover(tmp.path()); + let (set, diagnostics) = + resolve_watch_set(&spec, std::slice::from_ref(&spec), "hetz", &refresh); + assert!(!set.carriers.iter().any(|c| c.label == "goal")); + assert!(set.carriers.iter().any(|c| c.label == "declaration")); + assert!(!set.carriers.iter().any(|c| c.label == "issue")); + assert_eq!(diagnostics.len(), 1); + assert!(diagnostics[0].contains("resource 'goal'")); + assert!(diagnostics[0].contains("unwatchable")); + assert_eq!( + resource_coverage_with_profiles(&dir, &spec.resources[0], &refresh), + ResyncCoverage::Unsupported + ); +} + +#[test] +fn silent_profile_skips_its_resolver_entirely() { + let tmp = tempfile::tempdir().unwrap(); + let dir = tmp.path().join("agents/hetz/worker"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("agent.kdl"), + r#"agent "worker" { + host "hetz" + command "true" + resource "goal" uri="dev.schickling.agent-goal://hetz/worker" reason="Mission." +}"#, + ) + .unwrap(); + let missing = tmp.path().join("must-not-load.wasm"); + let profiles = ResourceProfileRegistry::empty().with_profile( + agent_spec::ResourceProfile::wasm( + "dev.schickling.agent-goal", + missing, + ProfileClass::Silent, + ), + ); + let refresh = profiles.begin_refresh(); + let spec = discover(tmp.path()); + let (set, diagnostics) = + resolve_watch_set(&spec, std::slice::from_ref(&spec), "hetz", &refresh); + assert!(!set.carriers.iter().any(|carrier| carrier.label == "goal")); + assert!( + diagnostics.is_empty(), + "a silent profile must not execute its missing resolver: {diagnostics:?}" + ); + assert_eq!( + resource_coverage_with_profiles(&dir, &spec.resources[0], &refresh), + ResyncCoverage::Silent + ); +} diff --git a/src/resync/watch_set.rs b/src/resync/watch_set.rs new file mode 100644 index 00000000..0469623a --- /dev/null +++ b/src/resync/watch_set.rs @@ -0,0 +1,481 @@ +//! What a declaration makes this host watch, and how completely it is covered. +//! +//! Moved verbatim out of the parent module: carrier classification, the coalescing windows, the +//! per-agent watch set, and resource coverage. + +use std::collections::BTreeMap; +use std::path::{Component, Path, PathBuf}; +use std::time::Duration; + +use sha2::{Digest as _, Sha256}; + +pub(super) use agent_spec::profile::ProfileClass; +use agent_spec::profile::{ResourceProfileRefresh, ResourceProfileRegistry}; +use agent_spec::spec::{AgentSpec, Resource, decode_percent_path}; + +use crate::resource_profile::{MAX_FACTS, MAX_FACT_KEY_BYTES}; + + +/// Provisional coalescing windows (`RESYNC-T02`): tuned by observed notification volume. +pub(super) const IMMEDIATE_WINDOW: Duration = Duration::from_millis(500); +pub(super) const COALESCED_WINDOW: Duration = Duration::from_secs(5); + +/// How a carrier notifies (`RESYNC-R04`). Silent carriers never reach the watch set: +/// [`classify`] excludes them, so nothing about them is observed or emitted. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum CarrierClass { + Immediate, + Coalesced, +} + +impl CarrierClass { + pub(super) fn window(self) -> Duration { + match self { + CarrierClass::Immediate => IMMEDIATE_WINDOW, + CarrierClass::Coalesced => COALESCED_WINDOW, + } + } +} +/// Catalog-observable resync coverage for one declared Resource binding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ResyncCoverage { + Immediate, + Coalesced, + Silent, + Unsupported, + Inactive, +} + +impl ResyncCoverage { + pub const fn as_str(self) -> &'static str { + match self { + Self::Immediate => "immediate", + Self::Coalesced => "coalesced", + Self::Silent => "silent", + Self::Unsupported => "unsupported", + Self::Inactive => "inactive", + } + } + + pub(super) fn carrier_class(self) -> Option { + match self { + Self::Immediate => Some(CarrierClass::Immediate), + Self::Coalesced => Some(CarrierClass::Coalesced), + Self::Silent | Self::Unsupported | Self::Inactive => None, + } + } +} + + +/// One watchable local carrier: binding label, absolute path, notification class, and an optional +/// host root that must confine every read of a resolver-selected path. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WatchableCarrier { + pub label: String, + pub path: PathBuf, + pub class: CarrierClass, + pub containment_root: Option, +} +#[derive(Debug, Clone, PartialEq, Eq)] +pub(super) struct DeclarationSummary { + pub(super) bindings: BTreeMap, + pub(super) complete: bool, +} + +pub(super) fn declaration_summary(spec: &AgentSpec) -> DeclarationSummary { + let mut bindings = spec + .resources + .iter() + .map(|resource| { + ( + resource.name().to_owned(), + declaration_resource_digest(resource), + ) + }) + .collect::>(); + bindings.sort_by(|left, right| left.0.cmp(&right.0)); + let complete = bindings.len() <= MAX_FACTS + && bindings + .iter() + .all(|(label, _)| label.len() <= MAX_FACT_KEY_BYTES); + DeclarationSummary { + bindings: bindings.into_iter().take(MAX_FACTS).collect(), + complete, + } +} + +fn declaration_resource_digest(resource: &Resource) -> String { + let mut digest = Sha256::new(); + digest.update(b"st2.resync.resource-declaration.v1\0"); + update_digest_field(&mut digest, resource.uri().as_bytes()); + update_digest_field(&mut digest, resource.reason().as_bytes()); + match resource.inactive_reason() { + Some(reason) => { + digest.update([1]); + update_digest_field(&mut digest, reason.as_bytes()); + } + None => digest.update([0]), + } + let selector = serde_json::to_vec(&resource.selector()) + .expect("a parsed JSON selector always serializes"); + update_digest_field(&mut digest, &selector); + format!("{:x}", digest.finalize()) +} + +fn update_digest_field(digest: &mut Sha256, value: &[u8]) { + let length = u64::try_from(value.len()).expect("declaration fields fit in u64"); + digest.update(length.to_be_bytes()); + digest.update(value); +} + +/// The watchable carriers of one agent, keyed by its declaration path with current routing IDs. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct AgentWatchSet { + pub declaration_path: PathBuf, + pub bus_id: String, + pub seat_id: Option, + pub carriers: Vec, + pub(super) declaration_summary: Option, +} + +/// Resolve one spec's watch set: the declaration file plus every active resource binding whose +/// URI denotes a local file (`RESYNC-R01`) — directly (`file://`, catalog-relative) or through a +/// declared resource profile for a scheme URI. Bindings with an inactive reason are skipped; +/// schemes without a local denotation and silent carriers are simply absent. A failing profile +/// resolver is contained: its binding is skipped, the rest of the set survives. +pub fn watch_set_for( + spec: &AgentSpec, + this_host: &str, + profiles: &ResourceProfileRegistry, +) -> AgentWatchSet { + watch_set_for_in_catalog(spec, std::slice::from_ref(spec), this_host, profiles) +} + +/// [`watch_set_for`] with the catalog view a `notify-chain` profile needs to reach the carriers +/// this agent's `supervisor` ancestors declare. Without the other specs, chain carriers cannot be +/// resolved and only the agent's own carriers are produced. +pub fn watch_set_for_in_catalog( + spec: &AgentSpec, + specs: &[AgentSpec], + this_host: &str, + profiles: &ResourceProfileRegistry, +) -> AgentWatchSet { + let refresh = profiles.begin_refresh(); + resolve_watch_set(spec, specs, this_host, &refresh).0 +} + +pub(super) fn resolve_watch_set( + spec: &AgentSpec, + specs: &[AgentSpec], + this_host: &str, + profiles: &ResourceProfileRefresh<'_>, +) -> (AgentWatchSet, Vec) { + let declaration_path = lexical_clean(&spec.path); + let agent_dir = declaration_path.parent().unwrap_or(Path::new(".")); + let mut carriers = vec![WatchableCarrier { + label: "declaration".to_owned(), + path: declaration_path.clone(), + class: CarrierClass::Immediate, + containment_root: None, + }]; + let mut diagnostics = Vec::new(); + for resource in &spec.resources { + if resource.inactive_reason().is_some() { + continue; + } + // Silent profiles carry no observable transition, so never compile or execute their + // untrusted resolver merely to discard its result. + let registered_profile = resource + .uri() + .split_once(':') + .and_then(|(scheme, _)| profiles.get(scheme)); + if registered_profile.is_some_and(|profile| profile.class() == ProfileClass::Silent) { + continue; + } + // Declared profile schemes resolve through their wasm module; the declared class governs + // notification instead of the local-path defaults. + match profiles.try_resolve(agent_dir, resource.uri()) { + Ok(Some(resolution)) => { + let Some(class) = carrier_class(resolution.class) else { + continue; + }; + carriers.push(WatchableCarrier { + label: resource.name().to_owned(), + path: resolution.path, + class, + containment_root: Some(resolution.containment_root), + }); + continue; + } + Ok(None) => {} + Err(error) => { + diagnostics.push(format!( + "resync profile for {} resource '{}': {error}; binding is unwatchable", + spec.bus_id(this_host), + resource.name() + )); + continue; + } + } + let Some(class) = resource_coverage(agent_dir, resource).carrier_class() else { + continue; + }; + let path = resolve_local_path(agent_dir, resource.uri()) + .expect("watchable coverage must have a local path"); + carriers.push(WatchableCarrier { + label: resource.name().to_owned(), + path, + class, + containment_root: None, + }); + } + append_chain_carriers( + spec, + specs, + this_host, + profiles, + &mut carriers, + &mut diagnostics, + ); + // The supervisor's resolved logical host — not the OS hostname — decides the bus id, so an + // agent supervised under `st2 up --host ` without an explicit declaration host still + // produces a recipient `resolve_stream` can resolve. + ( + AgentWatchSet { + declaration_path, + bus_id: spec.bus_id(this_host), + seat_id: spec.tasks.iter().find(|task| task.name == "agent").map(|task| { + task.id + .clone() + .unwrap_or_else(|| format!("{}.{}", spec.bus_id(this_host), task.name)) + }), + carriers, + declaration_summary: Some(declaration_summary(spec)), + }, + diagnostics, + ) +} + +/// A URI denotes a local file when it is an absolute, authority-free `file://` URI or a +/// scheme-less catalog-relative path resolved against the agent directory. URI scheme syntax is +/// parsed before the `file` scheme name is matched ASCII-case-insensitively, as required by RFC +/// 3986. Unsupported file URI authorities, query/fragment components, +/// malformed escapes, encoded path separators, and encoded parent components have no local +/// denotation. +/// Carriers this agent's `supervisor` ancestors declare through a `notify-chain` profile. +/// +/// A profile whose layers compose along the supervisor edge leaves every descendant's effective +/// view dependent on carriers the descendant does not own. Resync notifies a carrier's owner, so +/// without this the descendant is never told its view changed. +/// +/// The walk deliberately reuses each ancestor's OWN declared URI rather than synthesizing one: +/// st2 does not own any profile's URI grammar, and a resolver is free to ignore the authority +/// component entirely, so a synthesized subject would be a guess. Resolving the ancestor's +/// declaration against the ancestor's directory is the identical call the ancestor's own +/// subscription makes, which is what keeps containment unchanged — the guest is still only ever +/// asked to resolve one agent's URI against that agent's own directory. +/// +/// Matching is by profile scheme, never by binding label: labels are agent-local and replaceable, +/// so keying on them would silently drop a layer whose owner renamed its binding. +fn append_chain_carriers( + spec: &AgentSpec, + specs: &[AgentSpec], + this_host: &str, + profiles: &ResourceProfileRefresh<'_>, + carriers: &mut Vec, + diagnostics: &mut Vec, +) { + let chain_schemes: Vec<&str> = spec + .resources + .iter() + .filter(|resource| resource.inactive_reason().is_none()) + .filter_map(|resource| resource.uri().split_once(':').map(|(scheme, _)| scheme)) + .filter(|scheme| profiles.get(scheme).is_some_and(|p| p.notify_chain())) + .collect(); + if chain_schemes.is_empty() { + return; + } + + let ancestors = match crate::supervisor_chain::ancestors(specs, spec, this_host) { + Ok(ancestors) => ancestors, + Err(error) => { + diagnostics.push(format!( + "resync notify-chain for {}: supervisor chain is unwalkable ({error:?}); \ + ancestor carriers are unwatchable", + spec.bus_id(this_host) + )); + return; + } + }; + + for ancestor in ancestors { + // Skip and continue, never sever: a retired ancestor contributes no layer, but its own + // ancestors still do. `is_retired` normalizes both declaration spellings. + if ancestor.desired_state.is_retired() { + continue; + } + let ancestor_declaration = lexical_clean(&ancestor.path); + let ancestor_dir = ancestor_declaration.parent().unwrap_or(Path::new(".")); + let ancestor_bus_id = ancestor.bus_id(this_host); + for resource in &ancestor.resources { + if resource.inactive_reason().is_some() { + continue; + } + let Some((scheme, _)) = resource.uri().split_once(':') else { + continue; + }; + if !chain_schemes.contains(&scheme) { + continue; + } + match profiles.try_resolve(ancestor_dir, resource.uri()) { + Ok(Some(resolution)) => { + let Some(class) = carrier_class(resolution.class) else { + continue; + }; + carriers.push(WatchableCarrier { + // Qualifying by owner keeps each ancestor's layer on its own supersession + // key, so a burst on one ancestor cannot collapse another's event. + label: format!("{}@{ancestor_bus_id}", resource.name()), + path: resolution.path, + class, + containment_root: Some(resolution.containment_root), + }); + } + Ok(None) => {} + Err(error) => diagnostics.push(format!( + "resync notify-chain for {}: ancestor {ancestor_bus_id} resource '{}': \ + {error}; that ancestor layer is unwatchable", + spec.bus_id(this_host), + resource.name() + )), + } + } + } +} + +pub(super) fn resolve_local_path(agent_dir: &Path, uri: &str) -> Option { + if let Some((scheme, scheme_specific)) = uri.split_once(':').filter(|(scheme, _)| { + !scheme.is_empty() + && !scheme.contains('/') + && scheme + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.')) + }) { + if !scheme.eq_ignore_ascii_case("file") { + return None; + } + let encoded_path = scheme_specific.strip_prefix("//")?; + if !encoded_path.starts_with('/') + || encoded_path.starts_with("//") + || encoded_path.contains(['?', '#']) + { + return None; + } + let path = PathBuf::from(decode_percent_path(encoded_path).ok()?); + return path.is_absolute().then(|| lexical_clean(&path)); + } + let path = PathBuf::from(decode_percent_path(uri).ok()?); + Some(lexical_clean(&agent_dir.join(path))) +} +/// Resolve the externally visible resync coverage for one Resource binding. +pub fn resource_coverage(agent_dir: &Path, resource: &agent_spec::spec::Resource) -> ResyncCoverage { + if resource.inactive_reason().is_some() { + return ResyncCoverage::Inactive; + } + let Some(path) = resolve_local_path(agent_dir, resource.uri()) else { + return ResyncCoverage::Unsupported; + }; + match classify(agent_dir, resource.name(), &path) { + Some(CarrierClass::Immediate) => ResyncCoverage::Immediate, + Some(CarrierClass::Coalesced) => ResyncCoverage::Coalesced, + None => ResyncCoverage::Silent, + } +} + + +/// Remove `.` and `..` components lexically. This deliberately does not inspect the filesystem: +/// classification follows the authored path structure without resolving symlinks. +pub(super) fn lexical_clean(path: &Path) -> PathBuf { + let mut clean = PathBuf::new(); + for component in path.components() { + match component { + Component::Prefix(_) | Component::RootDir | Component::Normal(_) => { + clean.push(component.as_os_str()); + } + Component::CurDir => {} + Component::ParentDir => { + let can_pop = matches!(clean.components().next_back(), Some(Component::Normal(_))); + if can_pop { + clean.pop(); + } else if !clean.has_root() { + clean.push(component.as_os_str()); + } + } + } + } + clean +} + +/// Class defaults for carriers resolved WITHOUT a declared profile (`RESYNC-R04`): goal carriers +/// are immediate; stores the agent itself authors are silent (None); everything else is coalesced. +/// The declaration carrier is immediate by construction in [`watch_set_for`]. Profile-resolved +/// carriers skip this sniffing entirely — their class is what the catalog declares. +pub(super) fn classify(agent_dir: &Path, binding_name: &str, normalized_path: &Path) -> Option { + let agent_relative = normalized_path.strip_prefix(agent_dir).ok(); + let authored_store = agent_relative.is_some_and(|rel| { + rel.starts_with("resources/context") + || rel.starts_with("resources/decisions") + || rel.starts_with("resources/friction") + }); + if authored_store { + return None; + } + let goal = + binding_name == "goal" || normalized_path.file_name().is_some_and(|n| n == "goal.md"); + Some(if goal { + CarrierClass::Immediate + } else { + CarrierClass::Coalesced + }) +} + +/// A declared profile class maps onto carrier notification: silent profiles are excluded from +/// the watch set exactly like sniffed agent-authored stores. +pub(super) fn carrier_class(class: ProfileClass) -> Option { + match class { + ProfileClass::Immediate => Some(CarrierClass::Immediate), + ProfileClass::Coalesced => Some(CarrierClass::Coalesced), + ProfileClass::Silent => None, + } +} + +/// Resolve coverage with the catalog's declared profile registry. Silent profiles report their +/// declared class without executing a guest; other registered schemes are watchable only when +/// their resolver succeeds. +pub fn resource_coverage_with_profiles( + agent_dir: &Path, + resource: &agent_spec::spec::Resource, + profiles: &ResourceProfileRefresh<'_>, +) -> ResyncCoverage { + if resource.inactive_reason().is_some() { + return ResyncCoverage::Inactive; + } + let registered = resource + .uri() + .split_once(':') + .and_then(|(scheme, _)| profiles.get(scheme)); + let Some(profile) = registered else { + return resource_coverage(agent_dir, resource); + }; + if profile.class() == ProfileClass::Silent { + return ResyncCoverage::Silent; + } + match profiles.try_resolve(agent_dir, resource.uri()) { + Ok(Some(resolution)) => match resolution.class { + ProfileClass::Immediate => ResyncCoverage::Immediate, + ProfileClass::Coalesced => ResyncCoverage::Coalesced, + ProfileClass::Silent => ResyncCoverage::Silent, + }, + Ok(None) | Err(_) => ResyncCoverage::Unsupported, + } +} diff --git a/src/run.rs b/src/run.rs index c8820d5c..f2a81246 100644 --- a/src/run.rs +++ b/src/run.rs @@ -14,12 +14,9 @@ use std::cell::RefCell; use std::collections::{BTreeMap, HashMap, HashSet}; use std::ffi::{OsStr, OsString}; use std::fs::File; -use std::io::{Read as _, Seek as _, Write as _}; -use std::os::fd::AsRawFd as _; use std::os::unix::fs::MetadataExt as _; -use std::os::unix::process::CommandExt as _; use std::path::{Path, PathBuf}; -use std::process::{Child, ChildStdin, Command, Output, Stdio}; +use std::process::Command; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::mpsc::{Receiver, RecvTimeoutError, Sender, channel}; use std::time::{Duration, Instant}; @@ -72,270 +69,8 @@ impl PresentationPatchCursor { } } -/// Per-stream cap for captured child diagnostics. Tail-preserving: when output exceeds the cap, -/// the LAST [`CAPTURE_CAP_BYTES`] bytes are kept — recent output is what a failure message needs, -/// and an uncapped capture lets one chatty child balloon sidecar memory without bound. -pub(crate) const CAPTURE_CAP_BYTES: usize = 256 * 1024; - -/// One captured child stream capped to [`CAPTURE_CAP_BYTES`], keeping the tail. -pub(crate) struct BoundedStream { - pub bytes: Vec, // last <= cap bytes - pub total: u64, // complete stream size before capping -} - -impl BoundedStream { - pub fn truncated(&self) -> bool { - self.total as usize > self.bytes.len() - } -} - -/// Read back at most `cap` bytes of a temp-file capture, preserving the tail. The file is stat'ed -/// and seek'ed straight to `len - cap`, so the cost is O(cap) no matter how much the child wrote. -pub(crate) fn read_bounded_tail( - file: &mut std::fs::File, - cap: usize, -) -> std::io::Result { - let total = file.metadata()?.len(); - let skip = total.saturating_sub(cap as u64); - file.seek(std::io::SeekFrom::Start(skip))?; - let mut bytes = Vec::with_capacity((total - skip) as usize); - file.take(cap as u64).read_to_end(&mut bytes)?; - Ok(BoundedStream { bytes, total }) -} - -/// Send an already-killed child to ONE shared reaper thread instead of spawning a detached thread -/// per timed-out child: under a timeout storm one-thread-per-child accumulates without bound. -/// The thread starts lazily on first use. -pub(crate) fn reap_detached(child: std::process::Child) { - static REAPER: std::sync::LazyLock> = - std::sync::LazyLock::new(|| { - let (sender, receiver) = std::sync::mpsc::channel::(); - // Thread-spawn exhaustion is the only failure mode; panicking here surfaces it at the - // call site instead of silently leaking unreaped children. - std::thread::Builder::new() - .name("st2-child-reaper".to_string()) - .spawn(move || { - for mut child in receiver { - let _ = child.wait(); - } - }) - .expect("spawn shared child reaper thread"); - sender - }); - let _ = REAPER.send(child); -} - -/// Run a non-interactive child with bounded output capture: each stream keeps at most its last -/// [`CAPTURE_CAP_BYTES`] bytes (tail-preserving, with a diagnostic line on truncation). Regular -/// temporary files keep an escaped descendant that inherited stdout/stderr from blocking cleanup -/// after the direct child times out. -/// The child still gets a fresh process group so the common wrapper-and-descendants case is reaped. -#[cfg(test)] -fn output_with_timeout(command: &mut Command, timeout: Duration) -> anyhow::Result { - output_with_input_timeout(command, timeout, None) -} - -fn terminate_and_reap_before(mut child: Child, pid: i32, deadline: Instant) { - unsafe { - libc::kill(-pid, libc::SIGKILL); - } - let _ = child.kill(); - loop { - match child.try_wait() { - Ok(Some(_)) => return, - Ok(None) if Instant::now() < deadline => { - std::thread::sleep( - deadline - .saturating_duration_since(Instant::now()) - .min(Duration::from_millis(20)), - ); - } - Ok(None) | Err(_) => { - reap_detached(child); - return; - } - } - } -} - -fn write_all_before( - mut stdin: ChildStdin, - mut input: &[u8], - deadline: Instant, -) -> anyhow::Result { - let fd = stdin.as_raw_fd(); - let flags = unsafe { libc::fcntl(fd, libc::F_GETFL) }; - if flags == -1 { - return Err(std::io::Error::last_os_error()).context("read metadata stdin flags"); - } - if unsafe { libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK) } == -1 { - return Err(std::io::Error::last_os_error()).context("make metadata stdin nonblocking"); - } - while !input.is_empty() { - if Instant::now() >= deadline { - return Ok(false); - } - match stdin.write(input) { - Ok(0) => { - return Err(std::io::Error::from(std::io::ErrorKind::WriteZero)) - .context("write metadata patch payload"); - } - Ok(written) => input = &input[written..], - Err(error) if error.kind() == std::io::ErrorKind::Interrupted => {} - Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => { - if Instant::now() >= deadline { - return Ok(false); - } - std::thread::sleep( - deadline - .saturating_duration_since(Instant::now()) - .min(Duration::from_millis(20)), - ); - } - Err(error) => return Err(error).context("write metadata patch payload"), - } - } - Ok(true) -} - -fn output_with_input_timeout( - command: &mut Command, - timeout: Duration, - input: Option>, -) -> anyhow::Result { - output_with_input_timeout_observed(command, timeout, input, |_| {}) -} - -/// `on_spawn` observes the direct child's pid at the moment it exists. The child is `setsid`, so -/// that pid is also its process group id — the group this function signals on every failure path. -/// Tests need it to assert the child was reaped, and the child cannot supply it: a test whose -/// deadline expires before the child is first scheduled would never see anything the child wrote. -/// -/// It runs BEFORE the child deadline starts, and tests rely on that: a lifecycle test blocks in -/// `on_spawn` until the fixture reached the state it wants to measure, so fork+exec scheduling is -/// paid outside the deadline instead of out of it. See -/// `tests::the_spawn_observer_runs_before_the_child_deadline_starts`. -fn output_with_input_timeout_observed( - command: &mut Command, - timeout: Duration, - input: Option>, - on_spawn: impl FnOnce(i32), -) -> anyhow::Result { - run_captured(command, timeout, input, on_spawn, false) -} - -/// Like [`output_with_timeout`], but returns the COMPLETE stdout: callers parse structured data -/// (e.g. `pty list --json`) that must be whole, and capping it would corrupt the parse for large -/// fleets. Stdout is therefore intentionally uncapped — one chatty child can balloon this buffer. -/// Stderr stays tail-capped at [`CAPTURE_CAP_BYTES`] with a diagnostic line on truncation, -/// because stderr is only surfaced inside error messages. -pub(crate) fn output_full_stdout_with_timeout( - command: &mut Command, - timeout: Duration, -) -> anyhow::Result { - run_captured(command, timeout, None, |_| {}, true) -} - -#[cfg(test)] -fn output_full_stdout_with_timeout_observed( - command: &mut Command, - timeout: Duration, - on_spawn: impl FnOnce(i32), -) -> anyhow::Result { - run_captured(command, timeout, None, on_spawn, true) -} - -/// Shared spawn/wait/read-back core. The child is `setsid`, so its pid is also its process group -/// id — the group this function signals on every failure path. -fn run_captured( - command: &mut Command, - timeout: Duration, - input: Option>, - on_spawn: impl FnOnce(i32), - full_stdout: bool, -) -> anyhow::Result { - let mut stdout = tempfile::tempfile()?; - let mut stderr = tempfile::tempfile()?; - command - .stdin(if input.is_some() { - Stdio::piped() - } else { - Stdio::null() - }) - .stdout(Stdio::from(stdout.try_clone()?)) - .stderr(Stdio::from(stderr.try_clone()?)); - unsafe { - command.pre_exec(|| { - if libc::setsid() == -1 { - Err(std::io::Error::last_os_error()) - } else { - Ok(()) - } - }); - } - let mut child = command.spawn()?; - let pid = child.id() as i32; - on_spawn(pid); - // Load-bearing order: the deadline starts after `on_spawn` returns, so a test that blocks there - // as a readiness barrier spends none of `timeout` on fork+exec. Moving this line above - // `on_spawn` is silent in production and makes every barrier test load-sensitive again. - let deadline = Instant::now() + timeout; - if let Some(input) = input { - let Some(stdin) = child.stdin.take() else { - terminate_and_reap_before(child, pid, deadline); - anyhow::bail!("metadata patch child has no piped stdin"); - }; - match write_all_before(stdin, &input, deadline) { - Ok(true) => {} - Ok(false) => { - terminate_and_reap_before(child, pid, deadline); - anyhow::bail!("timed out after {:.1}s", timeout.as_secs_f64()); - } - Err(error) => { - terminate_and_reap_before(child, pid, deadline); - return Err(error); - } - } - } - let status = loop { - if let Some(status) = child.try_wait()? { - break status; - } - if Instant::now() >= deadline { - terminate_and_reap_before(child, pid, deadline); - anyhow::bail!("timed out after {:.1}s", timeout.as_secs_f64()); - } - std::thread::sleep(Duration::from_millis(20)); - }; - let stdout_stream = if full_stdout { - // Intentionally uncapped: callers parse structured data that must be whole. - stdout.rewind()?; - let mut bytes = Vec::new(); - stdout.read_to_end(&mut bytes)?; - BoundedStream { - total: bytes.len() as u64, - bytes, - } - } else { - read_bounded_tail(&mut stdout, CAPTURE_CAP_BYTES)? - }; - let stderr_stream = read_bounded_tail(&mut stderr, CAPTURE_CAP_BYTES)?; - let program = command.get_program().to_string_lossy(); - for (stream, name) in [(&stdout_stream, "stdout"), (&stderr_stream, "stderr")] { - if stream.truncated() { - eprintln!( - "st2: truncated {name} capture of `{program}`: keeping last {} of {} bytes (cap {CAPTURE_CAP_BYTES})", - stream.bytes.len(), - stream.total, - ); - } - } - Ok(Output { - status, - stdout: stdout_stream.bytes, - stderr: stderr_stream.bytes, - }) -} +mod process; +pub(crate) use process::*; /// Resolve a task's working directory: declared `cwd` (expanded), else the agent's `workspace` /// (expanded), else the spec file's directory (spec.md §2). A relative value is joined to the spec @@ -3345,3705 +3080,5 @@ pub fn detect_host() -> String { "localhost".to_string() } -mod tests { - use super::*; - use agent_spec::spec::{ - AgentSpec, Driver, JobType, OmpDriver, Task, TaskKind, TaskLifecycle, - }; - use std::cell::{Cell, RefCell}; - use std::collections::{BTreeMap, BTreeSet}; - use std::ffi::OsStr; - use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering}; - use std::sync::mpsc; - - - /// The bound is derived from the resolved pty root, never a fixed maximum identity length. - /// - /// `pty` binds `/.sock`, so the separator plus the five-byte suffix is - /// the fixed overhead and the usable identity length is whatever remains of the limit. These - /// numbers are measured against the pty binary itself: with a 21-byte root it accepts a - /// 77-byte id and refuses a 78-byte one as "a socket path of 105 bytes, which exceeds the - /// 104-byte kernel limit by 1". - #[test] - fn session_socket_overage_is_derived_from_the_resolved_root() { - let short_root = Path::new("/tmp/ptyprobe-1960953"); - let fits = "a".repeat(77); - let over = "a".repeat(78); - - assert_eq!( - session_socket_path(short_root, &fits) - .as_os_str() - .as_encoded_bytes() - .len(), - PORTABLE_SOCKET_PATH_LIMIT, - "the accepted id must land exactly on the limit" - ); - assert!(session_socket_overage(short_root, &fits).is_none()); - - let (path, overage) = - session_socket_overage(short_root, &over).expect("one byte over is refused"); - assert_eq!(overage, 1); - assert_eq!(path, short_root.join(format!("{over}.sock"))); - - // A deeper root shrinks every identity's budget on that host: the same id that fitted - // above is now 26 bytes over. - let deep_root = Path::new("/home/user/.local/state/st2/default/catalog/pty"); - assert_eq!( - session_socket_overage(deep_root, &fits).map(|(_, over)| over), - Some(26) - ); - } - - #[cfg(target_os = "linux")] - fn linux_process_state(pid: i32) -> Option { - std::fs::read_to_string(format!("/proc/{pid}/stat")) - .ok()? - .rsplit_once(") ")? - .1 - .chars() - .next() - } - - /// Block until the fixture publishes `marker`, which its script creates by an atomic rename so - /// the barrier never observes a half-written file. Called from `on_spawn`, which runs before - /// [`run_captured`] starts the child deadline: fork+exec scheduling is therefore paid here and - /// not out of the deadline the test then measures. The ceiling is deliberately far larger than - /// any plausible fork+exec — it bounds a fixture that never ran at all, and is not itself the - /// behaviour under test, so a loaded host cannot turn it into a failure. - fn await_fixture_ready(pid: i32, marker: &Path, what: &str) { - const CEILING: Duration = Duration::from_secs(30); - let deadline = Instant::now() + CEILING; - while !marker.exists() { - if Instant::now() >= deadline { - // Do not leak the fixture's long sleeper into the test host on the way out. - unsafe { - libc::kill(-pid, libc::SIGKILL); - libc::kill(pid, libc::SIGKILL); - } - panic!( - "{what} within {CEILING:?}: {} never appeared", - marker.display() - ); - } - std::thread::sleep(Duration::from_millis(5)); - } - } - - fn fixture_barrier_path(executable: &Path, suffix: &str) -> PathBuf { - PathBuf::from(format!("{}.{suffix}", executable.display())) - } - - fn reset_fixture_barrier(executable: &Path) { - for suffix in ["ready", "release"] { - let marker = fixture_barrier_path(executable, suffix); - match std::fs::remove_file(&marker) { - Ok(()) => {} - Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} - Err(error) => panic!("remove stale fixture marker {}: {error}", marker.display()), - } - } - } - - fn release_ready_fixture(pid: i32, executable: &Path, what: &str) { - let ready = fixture_barrier_path(executable, "ready"); - await_fixture_ready(pid, &ready, what); - std::fs::write(fixture_barrier_path(executable, "release"), b"go\n").unwrap(); - } - - fn process_can_retain_cleanup_resources(pid: i32) -> bool { - #[cfg(target_os = "linux")] - if linux_process_state(pid) == Some('Z') { - return false; - } - crate::host_lock::process_alive(pid) - } - - fn target(id: &str, cmd: &str) -> TaskTarget { - TaskTarget { - kind: TaskKind::Pty, - pty_id: id.to_string(), - bus_id: "hetz.demo".to_string(), - name: "agent".to_string(), - derived: false, - launch: TaskLaunch::Shell(cmd.to_string()), - cwd: None, - workspace: None, - tags: BTreeMap::new(), - env: BTreeMap::new(), - keep: false, - presentation: None, - } - } - - struct GateRunner { - list_calls: Cell, - } - - impl Runner for GateRunner { - fn list_sessions(&self) -> anyhow::Result> { - self.list_calls.set(self.list_calls.get() + 1); - Ok(Vec::new()) - } - - fn spawn(&self, _target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { - panic!("gate runner must not spawn") - } - - fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { - panic!("gate runner must not kill") - } - - fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { - panic!("gate runner must not remove") - } - } - - #[derive(Default)] - struct PersistentPatchRunner { - patched: RefCell>, - } - - impl Runner for PersistentPatchRunner { - fn list_sessions(&self) -> anyhow::Result> { - unreachable!("presentation execution does not list sessions") - } - - fn spawn(&self, _target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { - unreachable!("presentation-only plan must not spawn") - } - - fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { - unreachable!("presentation-only plan must not kill") - } - - fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { - unreachable!("presentation-only plan must not remove") - } - - fn patch_presentation(&self, presentation: &PtyPresentation) -> anyhow::Result<()> { - self.patched.borrow_mut().push(presentation.pty_id.clone()); - if presentation.pty_id.as_str() < "host.presented.08" { - anyhow::bail!("simulated persistent metadata failure"); - } - Ok(()) - } - } - - #[test] - fn bounded_presentation_batches_are_deterministic_and_do_not_starve() { - let plan = ReconcilePlan { - presentation: (0..12) - .rev() - .map(|index| PtyPresentation { - pty_id: format!("host.presented.{index:02}"), - display_name: None, - tags: BTreeMap::new(), - }) - .collect(), - ..ReconcilePlan::default() - }; - let runner = PersistentPatchRunner::default(); - let mut cap = FlappingCap::default(); - let mut cursor = PresentationPatchCursor::default(); - - for _ in 0..2 { - execute_with_presentation_cursor( - &plan, - &runner, - &mut cap, - &mut cursor, - &mut UpReport::default(), - &mut |_| {}, - ); - } - - let attempted = runner.patched.borrow(); - assert_eq!( - &attempted[..8], - &(0..8) - .map(|index| format!("host.presented.{index:02}")) - .collect::>() - ); - assert_eq!(attempted.len(), 16); - for index in 8..12 { - assert!(attempted.contains(&format!("host.presented.{index:02}"))); - } - } - - #[test] - fn lifecycle_hook_consumer_is_a_closed_enum() { - let consumers = [ - lifecycle_hook_consumer(true, false, false), - lifecycle_hook_consumer(false, true, false), - lifecycle_hook_consumer(false, false, true), - lifecycle_hook_consumer(true, true, false), - lifecycle_hook_consumer(true, false, true), - lifecycle_hook_consumer(false, true, true), - lifecycle_hook_consumer(true, true, true), - ] - .into_iter() - .collect::>(); - assert_eq!( - consumers, - BTreeSet::from([ - "codex", - "pi", - "omp", - "codex+pi", - "codex+omp", - "pi+omp", - "codex+pi+omp", - ]) - ); - } - - #[test] - fn selected_codex_gate_suppresses_launch_on_stale_hooks() { - let spec = AgentSpec { - id: None, - address: None, - identity: "codex".into(), - name: None, - description: None, - host: None, - role: None, - job_type: JobType::Service, - workspace: None, - supervisor: None, - desired_state: crate::AgentDesiredState::Running, - keep: false, - restart: None, - delivery: None, - session_driver: None, - driver: None, - delivery_readiness: None, - resources: vec![], - streams: Vec::new(), - tasks: vec![Task { - kind: TaskKind::Pty, - derived: false, - name: "agent".into(), - id: Some("test.codex.agent".into()), - command: None, - argv: Some(vec!["$CATALOG/bin/codex".into(), "--version".into()]), - cwd: None, - tags: BTreeMap::new(), - env: BTreeMap::new(), - keep: false, - lifecycle: TaskLifecycle::Service, - }], - path: "/tmp/spec.kdl".into(), - }; - let runner = GateRunner { - list_calls: Cell::new(0), - }; - let report = up_once_selected_specs_with_gates( - Path::new("/tmp"), - &[spec], - "test.codex.agent", - "test", - &runner, - |consumer| { - assert_eq!(consumer, None); - anyhow::bail!("stale receipt") - }, - ) - .unwrap(); - assert_eq!(runner.list_calls.get(), 1); - assert!(report.launched.is_empty()); - assert!(report.errors.iter().any(|error| { - error.contains("stale receipt") && error.contains("launch suppressed") - })); - } - - #[test] - fn selected_identity_conflict_refuses_before_hook_verification_or_inventory() { - let mut spec = AgentSpec { - id: None, - address: None, - identity: "codex".into(), - name: None, - description: None, - host: None, - role: None, - job_type: JobType::Service, - workspace: None, - supervisor: None, - desired_state: crate::AgentDesiredState::Running, - keep: false, - restart: None, - delivery: None, - session_driver: None, - driver: None, - delivery_readiness: None, - resources: vec![], - streams: Vec::new(), - tasks: vec![Task { - kind: TaskKind::Pty, - derived: false, - name: "agent".into(), - id: Some("test.codex.agent".into()), - command: None, - argv: Some(vec!["$CATALOG/bin/codex".into(), "--version".into()]), - cwd: None, - tags: BTreeMap::new(), - env: BTreeMap::new(), - keep: false, - lifecycle: TaskLifecycle::Service, - }], - path: "/tmp/spec.kdl".into(), - }; - spec.tasks[0] - .env - .insert("ST_AGENT".into(), "wrong.actor".into()); - let runner = GateRunner { - list_calls: Cell::new(0), - }; - let verify_calls = Cell::new(0); - - let error = up_once_selected_specs_with_gates( - Path::new("/tmp"), - &[spec], - "test.codex.agent", - "test", - &runner, - |_| { - verify_calls.set(verify_calls.get() + 1); - Ok(()) - }, - ) - .unwrap_err(); - - assert!(error.to_string().contains("conflicting ST_AGENT")); - assert_eq!(verify_calls.get(), 0); - assert_eq!(runner.list_calls.get(), 0); - } - - #[cfg(target_os = "linux")] - #[test] - fn idle_supervisor_does_not_spin_on_its_own_catalog_reads() { - let catalog = tempfile::tempdir().unwrap(); - let stop = AtomicBool::new(false); - let mut passes = 0usize; - - std::thread::scope(|scope| { - scope.spawn(|| { - std::thread::sleep(Duration::from_millis(350)); - stop.store(true, Ordering::SeqCst); - }); - up_loop_until( - catalog.path(), - "test-host", - &GateRunner { - list_calls: Cell::new(0), - }, - Duration::from_secs(60), - &stop, - best_effort_catalog_watcher, - |_| passes += 1, - ) - .unwrap(); - }); - - assert!( - passes <= 2, - "idle supervisor must wait instead of reconciling its own read events: {passes} passes" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn failed_watch_installation_keeps_supervisor_on_timer_cadence() { - let catalog = tempfile::tempdir().unwrap(); - let agent = catalog.path().join("agents/test-host/live"); - std::fs::create_dir_all(&agent).unwrap(); - std::fs::write( - agent.join("agent.kdl"), - r#"agent "live" { host "test-host"; command "x" }"#, - ) - .unwrap(); - let stop = AtomicBool::new(false); - let mut passes = 0usize; - let (started_tx, started_rx) = mpsc::sync_channel(1); - - std::thread::scope(|scope| { - let stop = &stop; - scope.spawn(move || { - started_rx.recv().unwrap(); - std::thread::sleep(Duration::from_millis(350)); - stop.store(true, Ordering::SeqCst); - }); - up_loop_until( - catalog.path(), - "test-host", - &SpawnCountingRunner::default(), - Duration::from_millis(100), - &stop, - |_, _| None, // watcher installation fails, as it did on dev3's oversized catalog - |_| { - passes += 1; - let _ = started_tx.try_send(()); - }, - ) - .unwrap(); - }); - - assert!( - (2..=6).contains(&passes), - "a disconnected watcher channel must fall back to timer cadence, not spin: \ - {passes} passes in ~350ms at a 100ms interval" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn supervisor_wakes_and_launches_a_new_direct_declaration() { - let catalog = tempfile::tempdir().unwrap(); - let agent = catalog.path().join("agents/test-host/live"); - let stop = AtomicBool::new(false); - let runner = SpawnCountingRunner::default(); - let (done_tx, done_rx) = mpsc::sync_channel(1); - let mut passes = 0usize; - - std::thread::scope(|scope| { - let watchdog_stop = &stop; - scope.spawn(move || { - if done_rx.recv_timeout(Duration::from_secs(5)).is_err() { - watchdog_stop.store(true, Ordering::SeqCst); - } - }); - up_loop_until( - catalog.path(), - "test-host", - &runner, - Duration::from_secs(60), - &stop, - best_effort_catalog_watcher, - |report| { - passes += 1; - match passes { - 1 => { - std::fs::create_dir_all(&agent).unwrap(); - std::fs::write( - agent.join("agent.kdl"), - r#"agent "live" { host "test-host"; command "x" }"#, - ) - .unwrap(); - } - 2 => { - assert_eq!(report.launched, ["test-host.live"]); - let _ = done_tx.send(()); - stop.store(true, Ordering::SeqCst); - } - _ => panic!("one direct declaration must cause exactly one prompt pass"), - } - }, - ) - .unwrap(); - }); - - assert_eq!(passes, 2, "the 60s timer must not be the publication path"); - assert_eq!(runner.spawned.borrow().as_slice(), ["test-host.live"]); - } - - #[test] - fn resident_loop_reloads_added_changed_removed_and_malformed_profiles() { - let catalog = tempfile::tempdir().unwrap(); - let agent = catalog.path().join("agents/test-host/live"); - std::fs::create_dir_all(&agent).unwrap(); - std::fs::write( - agent.join("agent.kdl"), - r#"agent "live" { - host "test-host" - command "true" - resource "alpha" uri="alpha://test-host/live" reason="Alpha." - resource "beta" uri="beta://test-host/live" reason="Beta." -}"#, - ) - .unwrap(); - let missing = catalog.path().join("missing.wasm"); - let profile = |scheme: &str| { - format!( - "profile {scheme:?} {{ wasm {:?} }}\n", - missing.display().to_string() - ) - }; - let config = crate::catalog::config_path(catalog.path()); - let runner = SpawnCountingRunner::default(); - runner - .sessions - .borrow_mut() - .push(sess("test-host.live.agent", true)); - let stop = AtomicBool::new(false); - let mut reports = Vec::new(); - - up_loop_until( - catalog.path(), - "test-host", - &runner, - Duration::from_millis(5), - &stop, - |_, _| None, - |report| { - let pass = reports.len(); - reports.push((report.warnings.clone(), report.errors.clone())); - match pass { - 0 => std::fs::write(&config, profile("alpha")).unwrap(), - 1 => std::fs::write(&config, profile("beta")).unwrap(), - 2 => std::fs::write(&config, "").unwrap(), - 3 => stop.store(true, Ordering::SeqCst), - _ => unreachable!("profile removal run stops after four passes"), - } - }, - ) - .unwrap(); - - let profile_warnings = |reports: &Vec<(Vec, Vec)>, pass: usize| { - reports[pass] - .0 - .iter() - .filter(|warning| warning.contains("resync profile")) - .cloned() - .collect::>() - }; - assert!( - profile_warnings(&reports, 0).is_empty(), - "no profile is initially declared" - ); - assert!( - profile_warnings(&reports, 1) - .iter() - .any(|warning| warning.contains("resource 'alpha'")), - "an added profile takes effect: {:?}", - reports[1] - ); - assert!( - profile_warnings(&reports, 2) - .iter() - .any(|warning| warning.contains("resource 'beta'")) - && !profile_warnings(&reports, 2) - .iter() - .any(|warning| warning.contains("resource 'alpha'")), - "changing definitions replaces the registry: {:?}", - reports[2] - ); - assert!( - profile_warnings(&reports, 3).is_empty(), - "removing every profile removes the old resolution semantics: {:?}", - reports[3] - ); - - // A separate resident lifetime starts valid, then makes the envelope malformed. The - // initial hard parse still accepts the valid declaration; the later edit must clear its - // active semantics rather than silently carrying them forward. - std::fs::write(&config, profile("alpha")).unwrap(); - stop.store(false, Ordering::SeqCst); - let mut malformed_reports = Vec::new(); - up_loop_until( - catalog.path(), - "test-host", - &runner, - Duration::from_millis(5), - &stop, - |_, _| None, - |report| { - let pass = malformed_reports.len(); - malformed_reports.push((report.warnings.clone(), report.errors.clone())); - match pass { - 0 => std::fs::write( - &config, - r#"profiel "alpha" { wasm "missing.wasm" }"#, - ) - .unwrap(), - 1 => stop.store(true, Ordering::SeqCst), - _ => unreachable!("malformed profile run stops after two passes"), - } - }, - ) - .unwrap(); - assert!( - profile_warnings(&malformed_reports, 0) - .iter() - .any(|warning| warning.contains("resource 'alpha'")), - "the profile is active before the malformed edit: {:?}", - malformed_reports[0] - ); - assert!( - malformed_reports[1] - .1 - .iter() - .any(|error| error.contains("unknown catalog.kdl top-level node 'profiel'")) - && profile_warnings(&malformed_reports, 1).is_empty(), - "malformed catalog state is reported and fails closed instead of retaining alpha: {:?}", - malformed_reports[1] - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn disconnected_watcher_channel_waits_out_the_interval_instead_of_spinning() { - let (_tx, rx) = channel::<()>(); - let started = Instant::now(); - assert_eq!( - wait_for_reconcile(&rx, Duration::from_millis(80), &AtomicBool::new(false)), - ReconcileWake::Interval, - "disconnection must be treated as silence, not as a change" - ); - assert!(started.elapsed() >= Duration::from_millis(75)); - } - - // ── liveness debounce (R21c): a transient `pty list` not-alive flicker under load must not - // destructively GC/relaunch a HEALTHY agent; a stable death must still be reaped ────────────── - - use crate::reconcile::Launch; - fn sess(id: &str, alive: bool) -> Session { - Session { - pty_id: id.to_string(), - alive, - exit_code: None, - presentation: None, - } - } - - /// Records spawns and reports every launch as succeeding, so a pass can be driven repeatedly. - #[derive(Default)] - struct SpawnCountingRunner { - sessions: RefCell>, - spawned: RefCell>, - } - - impl Runner for SpawnCountingRunner { - fn list_sessions(&self) -> anyhow::Result> { - Ok(self.sessions.borrow().clone()) - } - - fn spawn(&self, target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { - self.spawned.borrow_mut().push(target.pty_id.clone()); - Ok(()) - } - - fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { - Ok(()) - } - - fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { - Ok(()) - } - - fn patch_presentation(&self, _presentation: &PtyPresentation) -> anyhow::Result<()> { - Ok(()) - } - } - - /// `execute` must close every pass, or a task that recovers is never forgiven and eventually - /// parks even though it is healthy — the opposite of the crash-loop bug the cap exists for. - /// - /// The unit tests in `flapping.rs` call `end_pass` by hand, so they cannot catch it never being - /// called from a reconcile pass. This one drives the real `execute` path. `interval = 0s` makes - /// any survived pass count as recovery, keeping the test free of wall-clock sleeping. - #[test] - fn execute_closes_each_pass_so_a_recovered_task_regains_its_fail_budget() { - let mut spec = spec_fixture(); - spec.restart = Some(agent_spec::spec::Restart { - attempts: 3, - interval: Duration::from_secs(0), - delay: Duration::from_secs(0), - mode: agent_spec::spec::RestartMode::Fail, - }); - let runner = SpawnCountingRunner::default(); - let mut cap = FlappingCap::default(); - - fn dying(spec: &AgentSpec) -> ReconcilePlan<'_> { - ReconcilePlan { - launch: vec![Launch { - spec, - tasks: vec![target("hetz.demo.agent", "x")], - live_derived: Vec::new(), - }], - ..ReconcilePlan::default() - } - } - - // Two failing passes: two of three launches spent. - for _ in 0..2 { - execute(&dying(&spec), &runner, &mut cap, &mut UpReport::default()); - } - assert_eq!(runner.spawned.borrow().len(), 2, "two launches spent"); - - // A pass that launches nothing because it found the task alive. That observation — not the - // empty launch set — is what forgives the budget. - execute( - &ReconcilePlan { - live: vec!["hetz.demo.agent".to_string()], - ..ReconcilePlan::default() - }, - &runner, - &mut cap, - &mut UpReport::default(), - ); - - // Having recovered, it gets the full budget back: three more launches, then parked. Without - // the pass being closed it would park after only one more. - let mut last = UpReport::default(); - for _ in 0..4 { - last = UpReport::default(); - execute(&dying(&spec), &runner, &mut cap, &mut last); - } - assert_eq!( - runner.spawned.borrow().len(), - 5, - "recovery must restore the full `attempts` budget, not leave it partly spent" - ); - assert_eq!( - last.flapping, - vec!["hetz.demo.agent".to_string()], - "and it still parks in the end" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn persistent_advisory_warnings_surface_once_not_per_pass() { - let catalog = tempfile::tempdir().unwrap(); - let agent = catalog.path().join("agents/test-host/live"); - std::fs::create_dir_all(&agent).unwrap(); - std::fs::create_dir_all(catalog.path().join("workspace")).unwrap(); - std::fs::write( - agent.join("agent.kdl"), - r#"agent "live" { - host "test-host" - command "true" - workspace "$CATALOG/workspace" - render { git-exclude "scratch.txt" } -}"#, - ) - .unwrap(); - let stop = AtomicBool::new(false); - let mut passes = 0usize; - let mut warnings_seen = 0usize; - let (started_tx, started_rx) = mpsc::sync_channel(1); - - std::thread::scope(|scope| { - let stop = &stop; - scope.spawn(move || { - started_rx.recv().unwrap(); - std::thread::sleep(Duration::from_millis(300)); - stop.store(true, Ordering::SeqCst); - }); - up_loop_until( - catalog.path(), - "test-host", - &SpawnCountingRunner::default(), - Duration::from_millis(50), - &stop, - |_, _| None, - |report| { - passes += 1; - warnings_seen += report.warnings.len(); - let _ = started_tx.try_send(()); - }, - ) - .unwrap(); - }); - - assert!( - passes >= 3, - "the loop must have run several passes for this to say anything: {passes}" - ); - assert_eq!( - warnings_seen, 1, - "an unchanged advisory failure must be diagnosed once across {passes} passes" - ); - } - /// A pass can execute a plan the task was never in: `up_once` drops an owner whose - /// materialization failed, `gate_harness_launches_on_hooks` strips gated launches, and - /// `defer_flickers` removes debounced ones — each after the pass is already committed to - /// running. Silence about a task is not evidence it is alive, and crediting uptime for it lets - /// a permanently-dead task refill its budget on every gated pass and never park. Identical to - /// the recovery test above except that the quiet pass does not report the task live. - #[test] - fn a_pass_that_omits_a_task_does_not_credit_it_with_uptime() { - let mut spec = spec_fixture(); - spec.restart = Some(agent_spec::spec::Restart { - attempts: 3, - interval: Duration::from_secs(0), - delay: Duration::from_secs(0), - mode: agent_spec::spec::RestartMode::Fail, - }); - let runner = SpawnCountingRunner::default(); - let mut cap = FlappingCap::default(); - - fn dying(spec: &AgentSpec) -> ReconcilePlan<'_> { - ReconcilePlan { - launch: vec![Launch { - spec, - tasks: vec![target("hetz.demo.agent", "x")], - live_derived: Vec::new(), - }], - ..ReconcilePlan::default() - } - } - - // Two failing passes: two of three launches spent. - for _ in 0..2 { - execute(&dying(&spec), &runner, &mut cap, &mut UpReport::default()); - } - assert_eq!(runner.spawned.borrow().len(), 2, "two launches spent"); - - // The task is dropped from this pass — not launched, and not observed alive either. - execute( - &ReconcilePlan::default(), - &runner, - &mut cap, - &mut UpReport::default(), - ); - - // The budget must be where the failures left it: one launch remains, then it parks. - let mut last = UpReport::default(); - for _ in 0..4 { - last = UpReport::default(); - execute(&dying(&spec), &runner, &mut cap, &mut last); - } - assert_eq!( - runner.spawned.borrow().len(), - 3, - "an unobserved pass must not forgive the failure budget" - ); - assert_eq!( - last.flapping, - vec!["hetz.demo.agent".to_string()], - "and the task must still park" - ); - } - - fn spec_fixture() -> AgentSpec { - AgentSpec { - id: None, - address: None, - identity: "demo".into(), - name: None, - description: None, - host: Some("hetz".into()), - role: None, - job_type: JobType::Service, - workspace: None, - supervisor: None, - desired_state: crate::AgentDesiredState::Running, - keep: false, - restart: None, - delivery: None, - session_driver: None, - driver: None, - delivery_readiness: None, - resources: vec![], - streams: Vec::new(), - tasks: vec![], - path: std::path::PathBuf::from("/x"), - } - } - - #[test] - fn driver_labels_include_typed_and_argv_omp_but_remain_bounded() { - let legacy_spec = spec_fixture(); - let legacy_launch = Launch { - spec: &legacy_spec, - tasks: Vec::new(), - live_derived: Vec::new(), - }; - let mut omp_argv = target("hetz.demo.agent", "unused"); - omp_argv.launch = TaskLaunch::Argv(vec![ - "st2".into(), - "driver".into(), - "omp-session".into(), - ]); - let mut exec = target("hetz.demo.agent", "codex"); - exec.kind = TaskKind::Exec; - let targets = [ - target("hetz.demo.agent", "codex"), - target("hetz.demo.agent", "claude"), - target("hetz.demo.agent", "opencode"), - target("hetz.demo.agent", "pi"), - omp_argv, - exec, - target("hetz.demo.agent", "unrecognized"), - ]; - let labels = targets - .iter() - .map(|target| driver_label(&legacy_launch, target)) - .collect::>(); - assert_eq!( - labels, - BTreeSet::from(["codex", "claude", "opencode", "pi", "omp", "exec", "other"]) - ); - - let mut typed_spec = spec_fixture(); - typed_spec.driver = Some(Driver::Omp(OmpDriver { - model: None, - effort: None, - prompt: String::new(), - args: Vec::new(), - })); - let typed_launch = Launch { - spec: &typed_spec, - tasks: Vec::new(), - live_derived: Vec::new(), - }; - assert_eq!( - driver_label(&typed_launch, &target("hetz.demo.agent", "claude")), - "omp", - "typed driver identity must take precedence over argv heuristics" - ); - } - - #[test] - fn resync_watch_eligibility_requires_a_proven_live_agent_seat() { - let spec = |identity: &str, explicit_id: Option<&str>| { - let mut spec = spec_fixture(); - spec.identity = identity.to_owned(); - spec.tasks = vec![Task { - kind: TaskKind::Pty, - derived: false, - name: "agent".into(), - id: explicit_id.map(str::to_owned), - command: Some("agent".into()), - argv: None, - cwd: None, - tags: BTreeMap::new(), - env: BTreeMap::new(), - keep: false, - lifecycle: TaskLifecycle::Service, - }]; - spec - }; - let specs = vec![ - spec("desired", None), - spec("dead-adopted", None), - spec("observed-live", None), - spec("launched", None), - spec("restarted", Some("custom-seat")), - ]; - let sessions = vec![ - sess("hetz.dead-adopted.agent", false), - // A live canonical seat remains eligible even when a missing companion means the - // whole spec was not adopted and the companion later fails to launch. - sess("hetz.observed-live.agent", true), - ]; - let report = UpReport { - adopted: vec!["dead-adopted".into()], - launched: vec![ - "hetz.launched.agent".into(), - // A successfully launched companion is not evidence of a live agent seat. - "hetz.desired.ding".into(), - ], - restarted: vec!["custom-seat".into()], - ..UpReport::default() - }; - - let eligible = live_resync_specs(&specs, "hetz", &sessions, &report) - .into_iter() - .map(|spec| spec.identity) - .collect::>(); - assert_eq!(eligible, vec!["observed-live", "launched", "restarted"]); - } - - #[test] - fn subscription_eligibility_excludes_non_running_agents_even_with_a_live_seat() { - // A retired or suspended agent whose canonical seat is still alive mid-teardown owns no - // live subscription work: its resync installs and resource-Profile bindings must be - // stripped this pass, not left running until the seat dies (dotfiles#1535). The declaration - // (including its `resource` bindings) is untouched — only the runtime work stops. - let seat = || Task { - kind: TaskKind::Pty, - derived: false, - name: "agent".into(), - id: None, - command: Some("agent".into()), - argv: None, - cwd: None, - tags: BTreeMap::new(), - env: BTreeMap::new(), - keep: false, - lifecycle: TaskLifecycle::Service, - }; - let with_state = |identity: &str, state: crate::AgentDesiredState| { - let mut spec = spec_fixture(); - spec.identity = identity.to_owned(); - spec.tasks = vec![seat()]; - spec.desired_state = state; - spec - }; - let specs = vec![ - with_state("running", crate::AgentDesiredState::Running), - with_state( - "retired", - crate::AgentDesiredState::Retired { - reason: Some("Mission complete".into()), - }, - ), - with_state( - "suspended", - crate::AgentDesiredState::Suspended { - reason: "Waiting for capacity".into(), - }, - ), - ]; - // Every seat is observed alive, so only desired state can distinguish them. - let sessions = vec![ - sess("hetz.running.agent", true), - sess("hetz.retired.agent", true), - sess("hetz.suspended.agent", true), - ]; - let eligible = live_resync_specs(&specs, "hetz", &sessions, &UpReport::default()) - .into_iter() - .map(|spec| spec.identity) - .collect::>(); - assert_eq!(eligible, vec!["running"]); - } - - struct BlockingLaunchRunner { - sessions: RefCell>, - fail_id: Option, - block_id: String, - entered: mpsc::SyncSender<()>, - release: RefCell>, - } - - impl Runner for BlockingLaunchRunner { - fn list_sessions(&self) -> anyhow::Result> { - Ok(self.sessions.borrow().clone()) - } - - fn spawn(&self, target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { - if self.fail_id.as_deref() == Some(&target.pty_id) { - anyhow::bail!("simulated launch failure"); - } - if target.pty_id == self.block_id { - self.entered.send(()).unwrap(); - self.release.borrow_mut().recv().unwrap(); - } - Ok(()) - } - - fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { - Ok(()) - } - - fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { - Ok(()) - } - } - - #[cfg(all(test, feature = "wasm-resolver"))] - struct SteadyChainRunner { - sessions: std::sync::Mutex>, - block_id: String, - entered: mpsc::SyncSender<()>, - release: std::sync::Mutex>, - } - - #[cfg(all(test, feature = "wasm-resolver"))] - impl Runner for SteadyChainRunner { - fn list_sessions(&self) -> anyhow::Result> { - Ok(self - .sessions - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) - .clone()) - } - - fn spawn(&self, target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { - if target.pty_id == self.block_id { - self.entered.send(()).unwrap(); - self.release - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) - .recv() - .unwrap(); - } - self.sessions - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner) - .push(sess(&target.pty_id, true)); - Ok(()) - } - - fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { - Ok(()) - } - - fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { - Ok(()) - } - } - - fn write_resync_agent(catalog: &Path, identity: &str) -> (PathBuf, PathBuf) { - let agent_dir = catalog.join("agents/hetz").join(identity); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - format!( - r#"agent "{identity}" {{ - host "hetz" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." -}}"# - ), - ) - .unwrap(); - let goal = resources.join("goal.md"); - std::fs::write(&goal, "before\n").unwrap(); - (agent_dir, goal) - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn write_notify_chain_profile(catalog: &Path) { - let resolver_dir = catalog.join("resolvers"); - std::fs::create_dir_all(&resolver_dir).unwrap(); - std::fs::copy( - Path::new(env!("CARGO_MANIFEST_DIR")) - .join("crates/agent-spec/tests/fixtures/demo_resolver.wasm"), - resolver_dir.join("goal.wasm"), - ) - .unwrap(); - std::fs::write( - crate::catalog::config_path(catalog), - r#"profile "dev.schickling.agent-goal" { - wasm "resolvers/goal.wasm" - class "immediate" - notify-chain #true -} -"#, - ) - .unwrap(); - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn write_notify_chain_agent( - catalog: &Path, - identity: &str, - supervisor: Option<&str>, - later_task: bool, - ) -> (PathBuf, PathBuf) { - write_notify_chain_agent_with_state(catalog, identity, supervisor, later_task, None) - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn write_notify_chain_agent_with_state( - catalog: &Path, - identity: &str, - supervisor: Option<&str>, - later_task: bool, - desired_state: Option<&str>, - ) -> (PathBuf, PathBuf) { - let agent_dir = catalog.join("agents/hetz").join(identity); - let resources = agent_dir.join("resources"); - std::fs::create_dir_all(&resources).unwrap(); - let supervisor = supervisor - .map(|supervisor| format!(" supervisor {supervisor:?}\n")) - .unwrap_or_default(); - let desired_state = desired_state - .map(|desired_state| format!(" {desired_state}\n")) - .unwrap_or_default(); - let later_task = if later_task { - " exec \"later\" { command \"true\" }\n" - } else { - "" - }; - std::fs::write( - agent_dir.join("agent.kdl"), - format!( - r#"agent "{identity}" {{ - host "hetz" -{supervisor}{desired_state} command "agent" -{later_task} resource "goal" uri="dev.schickling.agent-goal://hetz/{identity}" reason="Layer." -}} -"# - ), - ) - .unwrap(); - let goal = resources.join("goal.md"); - std::fs::write(&goal, "before\n").unwrap(); - (agent_dir, goal) - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn current_resync_event_for_key(agent_dir: &Path, key: &str) -> Option { - let expected = format!("key: {key}"); - std::fs::read_dir(agent_dir.join("resources/inbox")) - .ok()? - .filter_map(Result::ok) - .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) - .find(|body| { - body.lines().any(|line| line == "stream: resync") - && body.lines().any(|line| line == expected) - }) - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn wait_for_resync_event_for_key(agent_dir: &Path, key: &str) -> Option { - let deadline = Instant::now() + Duration::from_secs(5); - loop { - if let Some(body) = current_resync_event_for_key(agent_dir, key) { - return Some(body); - } - if Instant::now() >= deadline { - return None; - } - std::thread::sleep(Duration::from_millis(20)); - } - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn wait_for_resync_event_key_change( - agent_dir: &Path, - key: &str, - prior: &str, - ) -> Option { - let deadline = Instant::now() + Duration::from_secs(5); - loop { - if let Some(body) = current_resync_event_for_key(agent_dir, key) - && body != prior - { - return Some(body); - } - if Instant::now() >= deadline { - return None; - } - std::thread::sleep(Duration::from_millis(20)); - } - } - - fn current_resync_event(agent_dir: &Path) -> Option { - std::fs::read_dir(agent_dir.join("resources/inbox")) - .ok()? - .filter_map(Result::ok) - .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) - .find(|body| body.lines().any(|line| line == "stream: resync")) - } - - fn wait_for_resync_event(agent_dir: &Path) -> Option { - let deadline = Instant::now() + Duration::from_secs(5); - loop { - if let Some(body) = current_resync_event(agent_dir) { - return Some(body); - } - if Instant::now() >= deadline { - return None; - } - std::thread::sleep(Duration::from_millis(20)); - } - } - - fn wait_for_resync_event_change(agent_dir: &Path, prior: &str) -> Option { - let deadline = Instant::now() + Duration::from_secs(5); - loop { - if let Some(body) = current_resync_event(agent_dir) - && body != prior - { - return Some(body); - } - if Instant::now() >= deadline { - return None; - } - std::thread::sleep(Duration::from_millis(20)); - } - } - - #[cfg(feature = "wasm-resolver")] - #[test] - fn up_loop_keeps_complete_notify_chain_sets_during_steady_reconcile() { - let catalog = tempfile::tempdir().unwrap(); - write_notify_chain_profile(catalog.path()); - let (root_dir, root_goal) = - write_notify_chain_agent(catalog.path(), "root", None, false); - let (lead_dir, lead_goal) = - write_notify_chain_agent(catalog.path(), "lead", Some("hetz.root"), false); - let (worker_dir, _worker_goal) = - write_notify_chain_agent(catalog.path(), "worker", Some("hetz.lead"), false); - let specs = crate::discover_strict(catalog.path()).specs; - let task_id = |spec: &AgentSpec, task: &Task| { - task.id - .clone() - .unwrap_or_else(|| format!("{}.{}", spec.bus_id("hetz"), task.name)) - }; - let mut sessions = Vec::new(); - for spec in &specs { - for task in &spec.tasks { - sessions.push(sess(&task_id(spec, task), true)); - } - } - let (entered_tx, entered_rx) = mpsc::sync_channel(0); - let (release_tx, release_rx) = mpsc::channel(); - let runner = SteadyChainRunner { - sessions: std::sync::Mutex::new(sessions), - block_id: "hetz.worker.later".to_owned(), - entered: entered_tx, - release: std::sync::Mutex::new(release_rx), - }; - let stop = AtomicBool::new(false); - let (first_report_tx, first_report_rx) = mpsc::sync_channel(1); - let observer_catalog = catalog.path().to_path_buf(); - - let evidence = std::thread::scope(|scope| { - let observer_stop = &stop; - let observer = scope.spawn(move || { - first_report_rx.recv().unwrap(); - - std::fs::write(&root_goal, "steady baseline transition\n").unwrap(); - let root_initial = wait_for_resync_event_for_key(&root_dir, "goal"); - let lead_initial = - wait_for_resync_event_for_key(&lead_dir, "goal@hetz.root"); - let worker_initial = - wait_for_resync_event_for_key(&worker_dir, "goal@hetz.root"); - - write_notify_chain_agent( - &observer_catalog, - "worker", - Some("hetz.lead"), - true, - ); - let entered = entered_rx.recv_timeout(Duration::from_secs(5)).is_ok(); - let root_after_reconcile = root_initial.as_deref().and_then(|prior| { - std::fs::write(&root_goal, "transition during steady reconcile\n").unwrap(); - wait_for_resync_event_key_change(&root_dir, "goal", prior) - }); - let lead_after_reconcile = lead_initial.as_deref().and_then(|prior| { - wait_for_resync_event_key_change(&lead_dir, "goal@hetz.root", prior) - }); - let worker_after_reconcile = worker_initial.as_deref().and_then(|prior| { - wait_for_resync_event_key_change(&worker_dir, "goal@hetz.root", prior) - }); - - std::fs::write(&lead_goal, "lead transition during steady reconcile\n").unwrap(); - let lead_own = wait_for_resync_event_for_key(&lead_dir, "goal"); - let worker_from_lead = - wait_for_resync_event_for_key(&worker_dir, "goal@hetz.lead"); - - let _ = release_tx.send(()); - observer_stop.store(true, Ordering::SeqCst); - ( - entered, - root_initial, - lead_initial, - worker_initial, - root_after_reconcile, - lead_after_reconcile, - worker_after_reconcile, - lead_own, - worker_from_lead, - ) - }); - up_loop_until( - catalog.path(), - "hetz", - &runner, - Duration::from_millis(25), - &stop, - |_, _| None, - |_| { - let _ = first_report_tx.try_send(()); - }, - ) - .unwrap(); - observer.join().unwrap() - }); - - assert!(evidence.0, "the steady-state reconcile must reach its later task"); - assert!(evidence.1.is_some(), "root must receive its own transition"); - assert!( - evidence.2.is_some() && evidence.3.is_some(), - "root transition must fan out through lead and worker" - ); - assert!( - evidence.4.is_some() && evidence.5.is_some() && evidence.6.is_some(), - "a steady reconcile must not replace chain sets with self-only sets" - ); - assert!( - evidence.7.is_some() && evidence.8.is_some(), - "lead transition must reach lead and worker" - ); - assert_up_loop_full_refresh_keeps_a_retired_middle_as_live_child_topology(); - } - - #[cfg(all(test, feature = "wasm-resolver"))] - fn assert_up_loop_full_refresh_keeps_a_retired_middle_as_live_child_topology() { - for retirement in [ - "retired #true", - "desired-state \"retired\" reason=\"fixture\"", - ] { - let catalog = tempfile::tempdir().unwrap(); - write_notify_chain_profile(catalog.path()); - let (root_dir, root_goal) = - write_notify_chain_agent(catalog.path(), "root", None, false); - let (middle_dir, _middle_goal) = write_notify_chain_agent_with_state( - catalog.path(), - "middle", - Some("hetz.root"), - false, - Some(retirement), - ); - let (child_dir, _child_goal) = - write_notify_chain_agent(catalog.path(), "child", Some("hetz.middle"), false); - let specs = crate::discover_strict(catalog.path()).specs; - let sessions = specs - .iter() - .filter(|spec| spec.desired_state.is_running()) - .flat_map(|spec| { - spec.tasks.iter().map(|task| { - let id = task - .id - .clone() - .unwrap_or_else(|| format!("{}.{}", spec.bus_id("hetz"), task.name)); - sess(&id, true) - }) - }) - .collect(); - let (entered_tx, _entered_rx) = mpsc::sync_channel(1); - let (_release_tx, release_rx) = mpsc::channel(); - let runner = SteadyChainRunner { - sessions: std::sync::Mutex::new(sessions), - block_id: "never-block".to_owned(), - entered: entered_tx, - release: std::sync::Mutex::new(release_rx), - }; - let stop = AtomicBool::new(false); - let missing_supervisor = AtomicBool::new(false); - let (first_report_tx, first_report_rx) = mpsc::sync_channel(1); - - let evidence = std::thread::scope(|scope| { - let observer_stop = &stop; - let observer = scope.spawn(move || { - first_report_rx.recv().unwrap(); - // Let the asynchronous full refresh replace the synchronous install before - // mutating the root carrier. The child must retain the complete catalog chain. - std::thread::sleep(Duration::from_millis(300)); - std::fs::write(&root_goal, "root transition after full refresh\n").unwrap(); - let root_event = wait_for_resync_event_for_key(&root_dir, "goal"); - let child_event = - wait_for_resync_event_for_key(&child_dir, "goal@hetz.root"); - let middle_event = current_resync_event_for_key(&middle_dir, "goal@hetz.root"); - observer_stop.store(true, Ordering::SeqCst); - (root_event, child_event, middle_event) - }); - up_loop_until( - catalog.path(), - "hetz", - &runner, - Duration::from_millis(25), - &stop, - |_, _| None, - |report| { - if report - .errors - .iter() - .chain(&report.warnings) - .any(|message| message.contains("MissingSupervisor")) - { - missing_supervisor.store(true, Ordering::SeqCst); - } - let _ = first_report_tx.try_send(()); - }, - ) - .unwrap(); - observer.join().unwrap() - }); - - assert!( - evidence.0.is_some(), - "root must receive its own event ({retirement})" - ); - assert!( - evidence.1.is_some(), - "the live child must receive exactly its owner-qualified root event through the \ - retired middle after full refresh ({retirement})" - ); - assert!( - evidence.2.is_none(), - "the retired middle must own no active subscription ({retirement})" - ); - assert!( - !missing_supervisor.load(Ordering::SeqCst), - "the complete catalog graph must prevent MissingSupervisor ({retirement})" - ); - } - } - - #[test] - fn compile_invalid_seat_does_not_block_existing_live_resync_watch() { - let catalog = tempfile::tempdir().unwrap(); - let (live_dir, live_goal) = write_resync_agent(catalog.path(), "live"); - let broken_dir = catalog.path().join("agents/hetz/broken"); - let broken_resources = broken_dir.join("resources"); - std::fs::create_dir_all(&broken_resources).unwrap(); - std::fs::create_dir_all(catalog.path().join("broken-workspace")).unwrap(); - let broken_declaration = broken_dir.join("agent.kdl"); - std::fs::write( - &broken_declaration, - r#"agent "broken" { - host "hetz" - deliver "mcp" - workspace "$CATALOG/broken-workspace" - exec "agent" { command "true" } - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let broken_goal = broken_resources.join("goal.md"); - std::fs::write(&broken_goal, "before\n").unwrap(); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - - let runner = SpawnCountingRunner { - sessions: RefCell::new(vec![ - sess("hetz.live", true), - sess("hetz.broken.agent", true), - ]), - ..SpawnCountingRunner::default() - }; - let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - let mut cap = FlappingCap::default(); - let mut debounce = LivenessDebounce::new(DEBOUNCE_GRACE); - let mut presentation_cursor = PresentationPatchCursor::default(); - - let first = reconcile_pass(catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), None); - assert!( - first.errors.iter().any(|error| { - error.contains("compile generated tasks") - && error.contains("non-PTY canonical task") - }), - "{first:#?}" - ); - assert!(!first.skipped, "the valid subset completed its pass"); - assert!( - runner.spawned.borrow().is_empty(), - "the compile-invalid seat must not launch" - ); - - std::fs::write(&live_goal, "changed while compile failed\n").unwrap(); - let first_event = wait_for_resync_event(&live_dir) - .expect("the already-live valid seat must stay watched across the compile error"); - assert!(first_event.contains(r#""binding":"goal""#), "{first_event}"); - - std::fs::write(&broken_goal, "invalid seat changed\n").unwrap(); - std::thread::sleep(Duration::from_millis(750)); - assert!( - current_resync_event(&broken_dir).is_none(), - "a compile-invalid seat must not be watched even when its canonical task is live" - ); - - std::fs::write(&live_goal, "changed while declaration is corrected\n").unwrap(); - std::fs::write( - &broken_declaration, - r#"agent "broken" { - host "hetz" - deliver "mcp" - workspace "$CATALOG/broken-workspace" - pty "agent" { command "true" } - resource "goal" uri="resources/goal.md" reason="Mission." -}"#, - ) - .unwrap(); - let corrected = reconcile_pass(catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), None); - assert!( - corrected - .errors - .iter() - .all(|error| !error.contains("compile generated tasks")), - "{corrected:#?}" - ); - assert!(corrected.launched.is_empty(), "{corrected:#?}"); - assert!( - corrected.adopted.iter().any(|identity| identity == "broken"), - "the corrected already-live seat should be adopted: {corrected:#?}" - ); - let corrected_event = wait_for_resync_event_change(&live_dir, &first_event) - .expect("correcting another declaration must not reseed and hide the live transition"); - assert!(corrected_event.contains(r#""binding":"goal""#), "{corrected_event}"); - } - - #[test] - fn materialization_failure_retains_only_the_observed_live_resync_watch() { - let catalog = tempfile::tempdir().unwrap(); - let write_broken_agent = |identity: &str| { - let (agent_dir, goal) = write_resync_agent(catalog.path(), identity); - let workspace = catalog.path().join(format!("{identity}-workspace")); - std::fs::create_dir_all(&workspace).unwrap(); - std::fs::write( - agent_dir.join("agent.kdl"), - format!( - r#"agent "{identity}" {{ - host "hetz" - workspace "{}" - command "agent" - resource "goal" uri="resources/goal.md" reason="Mission." - render {{ - copy "_templates/{identity}.md" "AGENTS.md" - }} -}}"#, - workspace.display() - ), - ) - .unwrap(); - (agent_dir, goal) - }; - let (live_dir, live_goal) = write_broken_agent("live"); - let (dormant_dir, dormant_goal) = write_broken_agent("dormant"); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - - let runner = SpawnCountingRunner { - sessions: RefCell::new(vec![sess("hetz.live", true)]), - ..SpawnCountingRunner::default() - }; - let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - let mut cap = FlappingCap::default(); - let mut debounce = LivenessDebounce::new(DEBOUNCE_GRACE); - let mut presentation_cursor = PresentationPatchCursor::default(); - - let failed = reconcile_pass(catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), None); - assert!( - failed - .errors - .iter() - .filter(|error| error.contains("copy source")) - .count() - >= 2, - "{failed:#?}" - ); - assert!( - runner.spawned.borrow().is_empty(), - "materialization-failed seats must not launch" - ); - - std::fs::write(&live_goal, "changed while materialization failed\n").unwrap(); - std::fs::write(&dormant_goal, "unwatched while materialization failed\n").unwrap(); - let first_event = wait_for_resync_event(&live_dir) - .expect("the observed live seat must remain watched through materialization failure"); - assert!(first_event.contains(r#""binding":"goal""#), "{first_event}"); - std::thread::sleep(Duration::from_millis(750)); - assert!( - current_resync_event(&dormant_dir).is_none(), - "a materialization-failed seat without an observed live session must stay unwatched" - ); - - std::fs::write(&live_goal, "changed immediately before recovery\n").unwrap(); - std::fs::create_dir_all(catalog.path().join("_templates")).unwrap(); - std::fs::write(catalog.path().join("_templates/live.md"), "rendered\n").unwrap(); - let recovered = reconcile_pass(catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), None); - assert!( - recovered - .errors - .iter() - .all(|error| !error.contains("_templates/live.md")), - "{recovered:#?}" - ); - assert!(recovered.launched.is_empty(), "{recovered:#?}"); - let recovered_event = wait_for_resync_event_change(&live_dir, &first_event) - .expect("recovery must preserve the pending transition instead of silently reseeding"); - assert!( - recovered_event.contains(r#""binding":"goal""#), - "{recovered_event}" - ); - } - - fn execute_resync_plan( - plan: &ReconcilePlan<'_>, - runner: &dyn Runner, - specs: &[AgentSpec], - resync: &crate::resync::ResyncSupervisor, - ) -> UpReport { - let mut report = UpReport::default(); - let mut install_count = 0; - execute_with_presentation_cursor( - plan, - runner, - &mut FlappingCap::default(), - &mut PresentationPatchCursor::default(), - &mut report, - &mut |spec| { - install_count += 1; - assert!(resync.install_live(spec, specs, "hetz").is_empty()); - }, - ); - assert!( - resync - .refresh( - specs, - &live_resync_specs(specs, "hetz", &[], &report), - "hetz", - &[], - &[], - ) - .is_empty() - ); - assert!(install_count > 0 || report.launched.is_empty()); - report - } - - #[cfg(feature = "wasm-resolver")] - #[test] - fn notify_chain_launch_boundary_installs_ancestors_before_a_later_task_finishes() { - let catalog = tempfile::tempdir().unwrap(); - write_notify_chain_profile(catalog.path()); - let (_root_dir, root_goal) = - write_notify_chain_agent(catalog.path(), "root", None, false); - write_notify_chain_agent(catalog.path(), "lead", Some("hetz.root"), false); - let (worker_dir, _worker_goal) = - write_notify_chain_agent(catalog.path(), "worker", Some("hetz.lead"), false); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let specs = crate::discover_strict(catalog.path()).specs; - let worker = specs - .iter() - .find(|spec| spec.identity == "worker") - .unwrap(); - let mut later = target("hetz.worker.later", "later"); - later.name = "later".into(); - later.derived = true; - let plan = ReconcilePlan { - launch: vec![Launch { - spec: worker, - tasks: vec![target("hetz.worker.agent", "agent"), later], - live_derived: Vec::new(), - }], - ..ReconcilePlan::default() - }; - let (entered_tx, entered_rx) = mpsc::sync_channel(0); - let (release_tx, release_rx) = mpsc::channel(); - let runner = BlockingLaunchRunner { - sessions: RefCell::new(Vec::new()), - fail_id: None, - block_id: "hetz.worker.later".to_owned(), - entered: entered_tx, - release: RefCell::new(release_rx), - }; - let resync = crate::resync::ResyncSupervisor::with_profiles( - catalog.path().to_path_buf(), - "hetz".into(), - crate::catalog::declared_profiles(catalog.path()).unwrap(), - ); - - let event = std::thread::scope(|scope| { - let observer = scope.spawn(move || { - entered_rx.recv().unwrap(); - std::fs::write(&root_goal, "changed while later task launches\n").unwrap(); - let event = - wait_for_resync_event_for_key(&worker_dir, "goal@hetz.root"); - release_tx.send(()).unwrap(); - event - }); - let report = execute_resync_plan(&plan, &runner, &specs, &resync); - assert_eq!( - report.launched, - ["hetz.worker.agent", "hetz.worker.later"] - ); - observer.join().unwrap() - }) - .expect("the fresh worker must receive its ancestor transition before full refresh"); - assert!(event.contains("key: goal@hetz.root"), "{event}"); - } - - #[test] - fn resync_launch_boundary_seeds_first_seat_before_later_seat_finishes() { - let catalog = tempfile::tempdir().unwrap(); - let (first_dir, first_goal) = write_resync_agent(catalog.path(), "first"); - write_resync_agent(catalog.path(), "second"); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let specs = crate::discover_strict(catalog.path()).specs; - let first = specs.iter().find(|spec| spec.identity == "first").unwrap(); - let second = specs.iter().find(|spec| spec.identity == "second").unwrap(); - let plan = ReconcilePlan { - launch: vec![ - Launch { - spec: first, - tasks: vec![target("hetz.first.agent", "agent")], - live_derived: Vec::new(), - }, - Launch { - spec: second, - tasks: vec![target("hetz.second.agent", "agent")], - live_derived: Vec::new(), - }, - ], - ..ReconcilePlan::default() - }; - let (entered_tx, entered_rx) = mpsc::sync_channel(0); - let (release_tx, release_rx) = mpsc::channel(); - let runner = BlockingLaunchRunner { - sessions: RefCell::new(Vec::new()), - fail_id: None, - block_id: "hetz.second.agent".to_owned(), - entered: entered_tx, - release: RefCell::new(release_rx), - }; - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - - std::thread::scope(|scope| { - scope.spawn(move || { - entered_rx.recv().unwrap(); - std::fs::write(&first_goal, "changed while second launches\n").unwrap(); - std::thread::sleep(Duration::from_secs(1)); - release_tx.send(()).unwrap(); - }); - let report = execute_resync_plan(&plan, &runner, &specs, &resync); - assert_eq!( - report.launched, - ["hetz.first.agent", "hetz.second.agent"] - ); - }); - - let event = wait_for_resync_event(&first_dir) - .expect("the first seat must observe a carrier transition during the later launch"); - assert!(event.contains(r#""binding":"goal""#), "{event}"); - } - - #[test] - fn resync_launch_boundary_excludes_failed_canonical_seat() { - let catalog = tempfile::tempdir().unwrap(); - let (first_dir, first_goal) = write_resync_agent(catalog.path(), "first"); - write_resync_agent(catalog.path(), "second"); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let specs = crate::discover_strict(catalog.path()).specs; - let first = specs.iter().find(|spec| spec.identity == "first").unwrap(); - let second = specs.iter().find(|spec| spec.identity == "second").unwrap(); - let plan = ReconcilePlan { - launch: vec![ - Launch { - spec: first, - tasks: vec![target("hetz.first.agent", "agent")], - live_derived: Vec::new(), - }, - Launch { - spec: second, - tasks: vec![target("hetz.second.agent", "agent")], - live_derived: Vec::new(), - }, - ], - ..ReconcilePlan::default() - }; - let (entered_tx, entered_rx) = mpsc::sync_channel(0); - let (release_tx, release_rx) = mpsc::channel(); - let runner = BlockingLaunchRunner { - sessions: RefCell::new(Vec::new()), - fail_id: Some("hetz.first.agent".to_owned()), - block_id: "hetz.second.agent".to_owned(), - entered: entered_tx, - release: RefCell::new(release_rx), - }; - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - - std::thread::scope(|scope| { - scope.spawn(move || { - entered_rx.recv().unwrap(); - std::fs::write(&first_goal, "changed after failed launch\n").unwrap(); - std::thread::sleep(Duration::from_secs(1)); - release_tx.send(()).unwrap(); - }); - let report = execute_resync_plan(&plan, &runner, &specs, &resync); - assert_eq!(report.launched, ["hetz.second.agent"]); - assert!(report.errors.iter().any(|error| { - error.contains("hetz.first.agent") && error.contains("simulated launch failure") - })); - }); - - std::thread::sleep(Duration::from_millis(750)); - assert!( - current_resync_event(&first_dir).is_none(), - "desired-but-failed canonical seats must remain unwatched" - ); - } - - #[test] - fn dead_resync_seat_is_deactivated_before_its_relaunch_blocks() { - let catalog = tempfile::tempdir().unwrap(); - let (agent_dir, goal) = write_resync_agent(catalog.path(), "worker"); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let (entered_tx, entered_rx) = mpsc::sync_channel(0); - let (release_tx, release_rx) = mpsc::channel(); - let runner = BlockingLaunchRunner { - sessions: RefCell::new(vec![sess("hetz.worker", true)]), - fail_id: None, - block_id: "hetz.worker".to_owned(), - entered: entered_tx, - release: RefCell::new(release_rx), - }; - let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - let mut cap = FlappingCap::default(); - let mut debounce = LivenessDebounce::new(Duration::ZERO); - let mut presentation_cursor = PresentationPatchCursor::default(); - - let seeded = reconcile_pass(catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), None); - assert!(seeded.adopted.iter().any(|identity| identity == "worker")); - *runner.sessions.borrow_mut() = vec![sess("hetz.worker", false)]; - let blocked_goal = goal.clone(); - - let relaunched = std::thread::scope(|scope| { - scope.spawn(move || { - entered_rx.recv().unwrap(); - std::fs::write(&blocked_goal, "changed while replacement launch blocks\n").unwrap(); - std::thread::sleep(Duration::from_secs(1)); - release_tx.send(()).unwrap(); - }); - reconcile_pass(catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), None) - }); - assert_eq!(relaunched.restarted, ["hetz.worker"]); - std::thread::sleep(Duration::from_millis(750)); - assert!( - current_resync_event(&agent_dir).is_none(), - "a carrier mutation while no canonical seat is live must not emit" - ); - - std::fs::write(&goal, "changed after replacement launch\n").unwrap(); - let event = wait_for_resync_event(&agent_dir) - .expect("the successful replacement must receive a fresh silent baseline"); - assert!(event.contains(r#""binding":"goal""#), "{event}"); - } - - /// A publication the resync worker cannot finish must not hold up a reconcile pass. - /// - /// Every per-seat `install_live` handshake is answered by the same worker thread that runs - /// publications, so a publication in progress serializes the whole pass behind it. That is the - /// coupling which let a terminal-refusal loop keep every pass from completing for two hours - /// (#431): the refusals only had power because they denied the pass that would have ended - /// them. Blocking one real publication on the recipient's stream lock is the sharpest form of - /// the same coupling — a slow publication makes a pass late, a stuck one makes it never - /// finish — and it holds the pass at exactly the point `emit_admitted` serializes. - #[test] - fn reconcile_pass_completes_while_a_resync_publication_is_blocked() { - use std::os::fd::AsRawFd as _; - use std::sync::atomic::{AtomicBool, Ordering as AtomicOrdering}; - - let catalog = tempfile::tempdir().unwrap(); - let (agent_dir, goal) = write_resync_agent(catalog.path(), "worker"); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let runner = SpawnCountingRunner { - sessions: RefCell::new(vec![sess("hetz.worker", true)]), - ..Default::default() - }; - let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - let mut cap = FlappingCap::default(); - let mut debounce = LivenessDebounce::new(DEBOUNCE_GRACE); - let mut presentation_cursor = PresentationPatchCursor::default(); - - let seeded = reconcile_pass( - catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), - None, - ); - assert!( - seeded.adopted.iter().any(|identity| identity == "worker"), - "{seeded:#?}" - ); - - // One completed publication first: it creates the recipient's resync stream state - // directory, whose `.lock` is the gate below, and proves the publication path is live. - std::fs::write(&goal, "changed before the gate closes\n").unwrap(); - let published = wait_for_resync_event(&agent_dir) - .expect("the live seat must observe its first carrier transition"); - - let gate = std::fs::OpenOptions::new() - .read(true) - .write(true) - .open(agent_dir.join("resources/streams/resync/.lock")) - .unwrap(); - assert_eq!( - unsafe { libc::flock(gate.as_raw_fd(), libc::LOCK_EX) }, - 0, - "the test must own the stream lock the publication path takes" - ); - let gate_fd = gate.as_raw_fd(); - - // The worker has no other work and `IMMEDIATE_WINDOW` is 500 ms, so it is inside the - // blocked publication well before this wait ends; the unchanged event is the positive - // evidence that the publication has not completed. - std::fs::write(&goal, "changed while the gate is closed\n").unwrap(); - std::thread::sleep(Duration::from_secs(2)); - assert_eq!( - current_resync_event(&agent_dir).as_deref(), - Some(published.as_str()), - "the gate must hold the second publication open" - ); - - // The watchdog releases the gate only when the pass fails to complete on its own, which - // is what separates a decoupled pass from one that merely finished after the rescue. - let rescued = AtomicBool::new(false); - let rescued_flag = &rescued; - let (finished_tx, finished_rx) = mpsc::channel::<()>(); - let pass = std::thread::scope(|scope| { - scope.spawn(move || { - if finished_rx.recv_timeout(Duration::from_secs(20)).is_err() { - rescued_flag.store(true, AtomicOrdering::SeqCst); - unsafe { libc::flock(gate_fd, libc::LOCK_UN) }; - } - }); - let pass = reconcile_pass( - catalog.path(), - "hetz", - &task_context, - &runner, - &mut cap, - &mut debounce, - &mut presentation_cursor, - Some(&resync), - None, - ); - let _ = finished_tx.send(()); - pass - }); - assert!( - !rescued.load(AtomicOrdering::SeqCst), - "the pass only completed after the blocked publication was released: {pass:#?}" - ); - assert!( - pass.adopted.iter().any(|identity| identity == "worker"), - "{pass:#?}" - ); - drop(gate); - } - - #[test] - fn resync_launch_boundary_preserves_baseline_across_derived_companion() { - let catalog = tempfile::tempdir().unwrap(); - let (agent_dir, goal) = write_resync_agent(catalog.path(), "worker"); - crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); - let specs = crate::discover_strict(catalog.path()).specs; - let spec = &specs[0]; - let mut derived = target("hetz.worker.ding", "ding"); - derived.name = "ding".into(); - derived.derived = true; - let plan = ReconcilePlan { - launch: vec![Launch { - spec, - tasks: vec![target("hetz.worker.agent", "agent"), derived], - live_derived: Vec::new(), - }], - ..ReconcilePlan::default() - }; - let (entered_tx, entered_rx) = mpsc::sync_channel(0); - let (release_tx, release_rx) = mpsc::channel(); - let runner = BlockingLaunchRunner { - sessions: RefCell::new(Vec::new()), - fail_id: None, - block_id: "hetz.worker.ding".to_owned(), - entered: entered_tx, - release: RefCell::new(release_rx), - }; - let resync = - crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); - let installs = AtomicUsize::new(0); - let mut report = UpReport::default(); - - std::thread::scope(|scope| { - scope.spawn(move || { - entered_rx.recv().unwrap(); - std::fs::write(&goal, "changed while companion launches\n").unwrap(); - std::thread::sleep(Duration::from_secs(1)); - release_tx.send(()).unwrap(); - }); - execute_with_presentation_cursor( - &plan, - &runner, - &mut FlappingCap::default(), - &mut PresentationPatchCursor::default(), - &mut report, - &mut |spec| { - installs.fetch_add(1, AtomicOrdering::SeqCst); - assert!(resync.install_live(spec, &specs, "hetz").is_empty()); - }, - ); - }); - assert!( - resync - .refresh( - &specs, - &live_resync_specs(&specs, "hetz", &[], &report), - "hetz", - &[], - &[], - ) - .is_empty() - ); - - assert_eq!( - installs.load(AtomicOrdering::SeqCst), - 1, - "only the canonical task transition may install its watch set" - ); - let event = wait_for_resync_event(&agent_dir) - .expect("the companion launch and final refresh must preserve the canonical baseline"); - assert!(event.contains(r#""binding":"goal""#), "{event}"); - } - - #[test] - fn debounce_absorbs_a_gc_flicker_but_reaps_a_stable_death() { - let t0 = Instant::now(); - let mut db = LivenessDebounce::new(Duration::from_secs(10)); - db.observe(&[sess("hetz.demo.agent", true)], t0); - - // Flicker: reads not-alive 1s later but was alive within the grace → deferred (left running). - let mut plan = ReconcilePlan::default(); - plan.gc.push("hetz.demo.agent".into()); - let deferred = db.defer_flickers(&mut plan, t0 + Duration::from_secs(1)); - assert!( - plan.gc.is_empty(), - "a recently-alive flicker must NOT be GC'd" - ); - assert_eq!(deferred, vec!["hetz.demo.agent".to_string()]); - - // CENTRAL anti-over-correction check: a STABLE death past the grace IS still reaped — the - // debounce must never MASK a real death. - let mut plan = ReconcilePlan::default(); - plan.gc.push("hetz.demo.agent".into()); - let deferred = db.defer_flickers(&mut plan, t0 + Duration::from_secs(11)); - assert_eq!( - plan.gc, - vec!["hetz.demo.agent".to_string()], - "a stable death must still be reaped" - ); - assert!(deferred.is_empty()); - } - - #[test] - fn effective_pty_root_prefers_an_exported_ambient_root_else_catalog_pty() { - let cat = std::path::Path::new("/deep/sandbox/st-root"); - // No ambient PTY_ROOT → the rendered default `/pty`. - assert_eq!(effective_pty_root_from(cat, None), cat.join("pty")); - assert_eq!( - effective_pty_root_from(cat, Some("".into())), - cat.join("pty"), - "empty is treated as unset" - ); - // An exported ambient PTY_ROOT (e.g. an eval's short decoupled root) WINS — so a deep catalog - // path can't blow the unix-socket limit, and spawn agrees with list/kill. - let short = std::ffi::OsString::from("/tmp/stev-abc123"); - assert_eq!( - effective_pty_root_from(cat, Some(short)), - std::path::PathBuf::from("/tmp/stev-abc123") - ); - } - - #[test] - fn a_catalog_declared_root_outranks_the_default_but_never_an_ambient_one() { - let tmp = tempfile::tempdir().unwrap(); - let cat = tmp.path(); - std::fs::write( - cat.join(crate::catalog::CONFIG_FILE), - "catalog { pty-root \"/run/agents/pty\" }\n", - ) - .unwrap(); - - // The declaration replaces the `/pty` default for every st2 pty op — so a reader - // that resolves the catalog finds the sessions without being handed an env var. - assert_eq!( - effective_pty_root_from(cat, None), - std::path::PathBuf::from("/run/agents/pty") - ); - // An explicit ambient root still wins: an eval run's short decoupled partition must be able - // to override a catalog it copied from. - assert_eq!( - effective_pty_root_from(cat, Some("/tmp/stev-abc123".into())), - std::path::PathBuf::from("/tmp/stev-abc123") - ); - } - - #[test] - fn debounce_never_defers_a_never_seen_task() { - let t0 = Instant::now(); - let db = LivenessDebounce::new(Duration::from_secs(10)); - // A genuinely-new target (never observed alive) is handled immediately, not deferred. - let mut plan = ReconcilePlan::default(); - plan.gc.push("hetz.brandnew.agent".into()); - let deferred = db.defer_flickers(&mut plan, t0); - assert_eq!(plan.gc, vec!["hetz.brandnew.agent".to_string()]); - assert!(deferred.is_empty()); - } - - #[test] - fn debounce_defers_a_flickering_launch_target_too() { - let t0 = Instant::now(); - let mut db = LivenessDebounce::new(Duration::from_secs(10)); - db.observe(&[sess("hetz.demo.agent", true)], t0); - - // The same recently-alive id showing up as a launch target (Absent/Dead) is also deferred — - // no noisy "already in use" re-launch of a live session. - let spec = spec_fixture(); - let mut plan = ReconcilePlan::default(); - plan.launch.push(Launch { - spec: &spec, - tasks: vec![target("hetz.demo.agent", "x")], - live_derived: Vec::new(), - }); - let deferred = db.defer_flickers(&mut plan, t0 + Duration::from_secs(2)); - assert!( - plan.launch.is_empty(), - "a recently-alive flicker must NOT be re-launched" - ); - assert_eq!(deferred, vec!["hetz.demo.agent".to_string()]); - } - - #[test] - fn codex_hook_gate_accepts_new_agents_without_mutating_the_launch_plan() { - let mut left = spec_fixture(); - left.identity = "left".into(); - left.path = PathBuf::from("/catalog/node/left/agent.kdl"); - let mut right = spec_fixture(); - right.identity = "right".into(); - right.path = PathBuf::from("/catalog/node/right/agent.kdl"); - let mut left_agent = target("node.left.agent", "exec codex --model gpt-5"); - left_agent.workspace = Some("/workspaces/shared".into()); - let mut right_agent = target("node.right.agent", "/opt/bin/codex --model gpt-5"); - right_agent.workspace = Some("/workspaces/shared".into()); - let mut plan = ReconcilePlan::default(); - plan.launch.push(Launch { - spec: &left, - tasks: vec![left_agent], - live_derived: Vec::new(), - }); - plan.launch.push(Launch { - spec: &right, - tasks: vec![right_agent], - live_derived: Vec::new(), - }); - let expected = plan - .launch - .iter() - .map(|launch| launch.spec.identity.clone()) - .collect::>(); - let mut report = UpReport::default(); - - gate_harness_launches_on_hooks(&mut plan, Path::new("/catalog"), &mut report, |_| Ok(())); - - assert_eq!( - plan.launch - .iter() - .map(|launch| launch.spec.identity.clone()) - .collect::>(), - expected, - "successful hook verification must leave the launch plan unchanged" - ); - assert_eq!(plan.launch.len(), 2); - assert!(report.errors.is_empty()); - } - - #[test] - fn codex_hook_gate_does_not_touch_adopted_agents_or_sidecar_only_repairs() { - let mut spec = spec_fixture(); - spec.identity = "root".into(); - let mut ding = target("node.root.ding", "st2 ding"); - ding.name = "ding".into(); - let mut plan = ReconcilePlan::default(); - plan.adopt.push(&spec); - plan.launch.push(Launch { - spec: &spec, - tasks: vec![ding], - live_derived: Vec::new(), - }); - let mut report = UpReport::default(); - - gate_harness_launches_on_hooks(&mut plan, Path::new("/catalog"), &mut report, |_| { - panic!("an already-live Codex agent must not enter the hook gate") - }); - - assert_eq!(plan.adopt, [&spec]); - assert_eq!(plan.launch.len(), 1); - assert_eq!(plan.launch[0].tasks[0].name, "ding"); - assert!(report.errors.is_empty()); - } - - #[test] - fn hook_verification_failure_suppresses_only_new_codex_agents() { - let mut codex = spec_fixture(); - codex.identity = "codex".into(); - codex.path = PathBuf::from("/catalog/node/codex/agent.kdl"); - let mut claude = spec_fixture(); - claude.identity = "claude".into(); - claude.path = PathBuf::from("/catalog/node/claude/agent.kdl"); - let mut codex_agent = target("node.codex.agent", "exec codex"); - codex_agent.workspace = Some("/workspaces/codex".into()); - let claude_agent = target("node.claude.agent", "exec claude"); - let mut plan = ReconcilePlan::default(); - plan.launch.push(Launch { - spec: &codex, - tasks: vec![codex_agent], - live_derived: Vec::new(), - }); - plan.launch.push(Launch { - spec: &claude, - tasks: vec![claude_agent], - live_derived: Vec::new(), - }); - let mut report = UpReport::default(); - - gate_harness_launches_on_hooks(&mut plan, Path::new("/catalog"), &mut report, |_| { - anyhow::bail!("stale receipt") - }); - - assert_eq!( - plan.launch - .iter() - .map(|launch| launch.spec.identity.as_str()) - .collect::>(), - ["claude"] - ); - assert_eq!(report.errors.len(), 1); - assert!(report.errors[0].contains("stale receipt")); - assert!(report.errors[0].contains("launch suppressed")); - } - - /// The built `pty run` argv runs the command verbatim under `sh -c`, detached, with the pinned id - /// and the established fallback presentation when no Agent Spec name is projected. - #[test] - fn build_run_command_wraps_command_in_sh_c() { - let cli = PtyCli::default(); - let t = target( - "hetz.demo.agent", - "exec claude --permission-mode bypassPermissions 'boot'", - ); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - - assert_eq!(cmd.get_program(), OsStr::new("pty")); - let args: Vec = cmd - .get_args() - .map(|a| a.to_string_lossy().into_owned()) - .collect(); - // Stable launch/config arguments precede the persisted environment and command separator. - assert_eq!(&args[0..2], &["run", "-d"]); - assert!(args.contains(&"--force".to_string())); - let id_pos = args.iter().position(|a| a == "--id").unwrap(); - assert_eq!(args[id_pos + 1], "hetz.demo.agent"); - let name_pos = args.iter().position(|a| a == "--name").unwrap(); - assert_eq!(args[name_pos + 1], "hetz.demo"); - let sep = args.iter().position(|a| a == "--").unwrap(); - assert_eq!( - &args[sep + 1..], - &[ - "sh", - "-c", - "exec claude --permission-mode bypassPermissions 'boot'" - ] - ); - } - - #[test] - fn build_run_command_projects_primary_name_and_owned_tags_at_spawn() { - let key = "ST2_TEST_PRESENTATION_LITERAL_71c"; - unsafe { std::env::set_var(key, "expanded") } - - let cli = PtyCli::default(); - let mut t = target("hetz.demo", "codex"); - t.bus_id = "hetz.demo".to_owned(); - t.tags - .insert("unrelated".to_owned(), "preserved".to_owned()); - t.presentation = Some(PtyPresentation { - pty_id: "hetz.demo".to_owned(), - display_name: Some(Some("Build owner".to_owned())), - tags: BTreeMap::from([ - ("agent.presentation.schema".to_owned(), Some("1".to_owned())), - ("agent.actor.path".to_owned(), Some("hetz.demo".to_owned())), - ( - "agent.presentation.description".to_owned(), - Some(format!("${key}")), - ), - ]), - }); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - let args = cmd - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect::>(); - - let name = args.iter().position(|arg| arg == "--name").unwrap(); - assert_eq!(args[name + 1], "Build owner"); - let tags = args - .windows(2) - .filter(|pair| pair[0] == "--tag") - .map(|pair| pair[1].as_str()) - .collect::>(); - assert!(tags.contains("unrelated=preserved")); - assert!(tags.contains("agent.presentation.schema=1")); - assert!(tags.contains("agent.actor.path=hetz.demo")); - assert!(tags.contains("agent.presentation.description=$ST2_TEST_PRESENTATION_LITERAL_71c")); - } - - #[test] - fn metadata_patch_uses_exact_id_and_one_json_stdin_payload() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("pty-capture"); - std::fs::write( - &executable, - r#"#!/bin/sh -printf '%s\n' "$@" > "$0.args" -printf '' > "$0.ready.tmp" -mv "$0.ready.tmp" "$0.ready" -while [ ! -e "$0.release" ]; do sleep 0.01; done -cat > "$0.stdin" -"#, - ) - .unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - reset_fixture_barrier(&executable); - let observed_executable = executable.clone(); - let cli = PtyCli { - bin: executable.display().to_string(), - catalog_root: temporary.path().to_path_buf(), - on_command_spawn: Some(std::sync::Arc::new(move |pid| { - release_ready_fixture( - pid, - &observed_executable, - "fake PTY metadata command was not ready", - ); - })), - }; - let presentation = PtyPresentation { - pty_id: "stable.agent.id".to_owned(), - display_name: Some(None), - tags: BTreeMap::from([ - ("agent.presentation.schema".to_owned(), Some("1".to_owned())), - ("agent.presentation.description".to_owned(), None), - ]), - }; - - cli.patch_presentation(&presentation).unwrap(); - - assert_eq!( - std::fs::read_to_string(executable.with_extension("args")).unwrap(), - "metadata\npatch\n--id\nstable.agent.id\n" - ); - let payload: serde_json::Value = - serde_json::from_slice(&std::fs::read(executable.with_extension("stdin")).unwrap()) - .unwrap(); - assert_eq!(payload["displayName"], serde_json::Value::Null); - assert_eq!(payload["tags"]["agent.presentation.schema"], "1"); - assert_eq!( - payload["tags"]["agent.presentation.description"], - serde_json::Value::Null - ); - } - - #[test] - fn input_write_failure_terminates_and_reaps_the_child() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("close-stdin"); - let stdin_closed = temporary.path().join("stdin-closed"); - // The script signals only AFTER closing its stdin, so the barrier below returns exactly when - // the read end is gone and the parent's very next write must fail with EPIPE. - std::fs::write( - &executable, - "#!/bin/sh\nexec 0<&-\n: > \"$READY.tmp\"\nmv \"$READY.tmp\" \"$READY\"\nsleep 60\n", - ) - .unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - let input = vec![b'x'; 1024 * 1024]; - // The pid comes from the parent at spawn, and the barrier makes the deadline measure only - // the behaviour under test. Without it the 1s budget also had to cover fork+exec of the - // shell, so a loaded host reported `timed out after 1.0s` instead of `Broken pipe` — the - // fixture's scheduling consumed the deadline the assertion is about. - let mut spawned = None; - let error = output_with_input_timeout_observed( - Command::new(&executable).env("READY", &stdin_closed), - Duration::from_secs(1), - Some(input), - |pid| { - spawned = Some(pid); - await_fixture_ready(pid, &stdin_closed, "the child never closed its stdin"); - }, - ) - .unwrap_err(); - let pid = spawned.expect("the child was spawned before the input write failed"); - - assert!( - format!("{error:#}").contains("Broken pipe"), - "unexpected write error: {error:#}" - ); - assert!( - !crate::host_lock::process_alive(pid), - "failed metadata child {pid} was not terminated and reaped" - ); - } - - /// The process-group kill is the entire stated reason [`terminate_and_reap_before`] exists — its - /// docstring is about an escaped descendant that inherited stdout/stderr and would otherwise - /// block cleanup. Nothing constructed such a descendant, so `kill(-pid, SIGKILL)` was asserted - /// by no test: removing it alone left the suite green, because `child.kill()` already satisfies - /// every assertion that only looks at the direct child. - #[test] - fn the_group_kill_reaps_a_descendant_that_outlives_the_direct_child() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("spawn-descendant"); - let descendant_pidfile = temporary.path().join("descendant.pid"); - // The descendant inherits stdout/stderr and outlives the direct child, which is exactly the - // shape the docstring describes. `child.kill()` cannot reach it; only the group signal can. - // It publishes its pid by atomic rename, so the barrier never reads a truncated file. - std::fs::write( - &executable, - "#!/bin/sh\nsh -c 'printf \"%s\" \"$$\" > \"$DESCENDANT_PIDFILE.tmp\"; mv \"$DESCENDANT_PIDFILE.tmp\" \"$DESCENDANT_PIDFILE\"; sleep 60' &\nsleep 60\n", - ) - .unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - - // This test *requires* the child to have run — a descendant it never forked is nothing to - // reap. Waiting for the pidfile inside `on_spawn` makes that a barrier instead of a race: - // the deadline then only has to outlast a `sleep`, never a fork+exec, so a loaded host can - // no longer end the run before the fixture has built the thing under test. - let error = output_with_input_timeout_observed( - Command::new(&executable).env("DESCENDANT_PIDFILE", &descendant_pidfile), - Duration::from_millis(500), - None, - |pid| { - await_fixture_ready( - pid, - &descendant_pidfile, - "the child never forked a descendant, so this case would test nothing", - ) - }, - ) - .unwrap_err(); - assert!( - format!("{error:#}").contains("timed out"), - "unexpected error: {error:#}" - ); - - let descendant = std::fs::read_to_string(&descendant_pidfile) - .expect("the readiness barrier returned without a pidfile") - .parse::() - .unwrap(); - - // Generous on purpose: the descendant is orphaned by the same group kill, so its exit is - // observable only once the reparenting init reaps it. That latency is not the behaviour - // under test, and waiting longer costs nothing when the kill did reach it. - let deadline = Instant::now() + Duration::from_secs(5); - while process_can_retain_cleanup_resources(descendant) && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(10)); - } - let survived = process_can_retain_cleanup_resources(descendant); - if survived { - // Do not leak a 60s sleeper into the test host when the assertion is about to fail. - unsafe { libc::kill(descendant, libc::SIGKILL) }; - } - assert!( - !survived, - "escaped descendant {descendant} survived cleanup: the process-group kill did not reach it" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn a_zombie_cannot_retain_cleanup_resources() { - let mut child = Command::new("sh").arg("-c").arg("exit 0").spawn().unwrap(); - let pid = child.id() as i32; - let deadline = Instant::now() + Duration::from_secs(1); - let mut state = None; - while Instant::now() < deadline { - state = linux_process_state(pid); - if state == Some('Z') { - break; - } - std::thread::sleep(Duration::from_millis(10)); - } - let kill_probe_considered_alive = crate::host_lock::process_alive(pid); - let retained_cleanup_resources = process_can_retain_cleanup_resources(pid); - let _ = child.wait(); - - assert_eq!(state, Some('Z'), "child did not become a zombie"); - assert!( - kill_probe_considered_alive, - "the fixture must expose kill(pid, 0) treating a zombie as alive" - ); - assert!( - !retained_cleanup_resources, - "a terminated zombie cannot retain cleanup resources" - ); - } - - #[test] - fn input_write_obeys_the_child_deadline() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("ignore-stdin"); - std::fs::write(&executable, "#!/bin/sh\nsleep 60\n").unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - let input = vec![b'x'; 1024 * 1024]; - // The pid comes from the parent at spawn, not from the child, and this case cannot use a - // readiness barrier: it is precisely the one where the child may never be scheduled. The - // write blocks as soon as the pipe buffer fills, which needs no execution by the child at - // all, and the deadline then terminates the whole group. Anything the child was supposed to - // record would never be written, so a test that waits for it fails on exactly the condition - // it exists to cover. The observed spawn instant is therefore also the clock: timing from - // before the call would charge fork+exec to the 1s budget this assertion polices. - let mut spawned = None; - let error = output_with_input_timeout_observed( - &mut Command::new(&executable), - Duration::from_millis(100), - Some(input), - |pid| spawned = Some((pid, Instant::now())), - ) - .unwrap_err(); - let (pid, started) = - spawned.expect("the child was spawned before the input deadline expired"); - - assert!( - format!("{error:#}").contains("timed out"), - "unexpected write error: {error:#}" - ); - assert!( - started.elapsed() < Duration::from_secs(1), - "blocked stdin write ignored the child deadline" - ); - let reap_deadline = Instant::now() + Duration::from_secs(1); - while crate::host_lock::process_alive(pid) && Instant::now() < reap_deadline { - std::thread::sleep(Duration::from_millis(10)); - } - assert!(!crate::host_lock::process_alive(pid)); - } - - /// The two lifecycle tests above block in `on_spawn` until their fixture reached the state under - /// test, which only keeps them load-insensitive because [`run_captured`] starts the child - /// deadline AFTER `on_spawn` returns. Nothing else proves that order: reversing it leaves every - /// other test green on an idle host and silently puts both back on a race with the scheduler. - #[test] - fn the_spawn_observer_runs_before_the_child_deadline_starts() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("close-stdin"); - let stdin_closed = temporary.path().join("stdin-closed"); - std::fs::write( - &executable, - "#!/bin/sh\nexec 0<&-\n: > \"$READY.tmp\"\nmv \"$READY.tmp\" \"$READY\"\nsleep 60\n", - ) - .unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - let timeout = Duration::from_millis(200); - - // The barrier deliberately outlasts `timeout`, so the outcome depends on the order alone and - // on nothing the host's scheduler does. Deadline after `on_spawn`: the write meets a closed - // read end and fails with EPIPE at once. Deadline before `on_spawn`: it has already expired - // when the barrier returns, so the write never runs and the call reports a timeout instead. - let error = output_with_input_timeout_observed( - Command::new(&executable).env("READY", &stdin_closed), - timeout, - Some(vec![b'x'; 1024]), - |pid| { - await_fixture_ready(pid, &stdin_closed, "the child never closed its stdin"); - std::thread::sleep(timeout * 2); - }, - ) - .unwrap_err(); - - assert!( - format!("{error:#}").contains("Broken pipe"), - "the child deadline started before `on_spawn` returned: {error:#}" - ); - } - - #[test] - fn bounded_capture_keeps_the_tail_of_an_oversized_stream() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("flood"); - // Start marker, 1 MiB of filler (4x the cap, so both streams truncate), end marker. - std::fs::write( - &executable, - "#!/bin/sh\nprintf START; head -c 1048576 /dev/zero; printf END\n", - ) - .unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - - let output = - output_with_timeout(&mut Command::new(&executable), Duration::from_secs(5)).unwrap(); - - assert_eq!(output.stdout.len(), CAPTURE_CAP_BYTES); - assert!( - output.stdout.ends_with(b"END"), - "capped stdout lost the tail" - ); - assert!( - !output.stdout.starts_with(b"START"), - "capped stdout kept the head instead of the tail" - ); - // stderr is empty here, so only the stdout read-back may have been capped. - } - - #[test] - fn full_stdout_variant_returns_complete_output_larger_than_the_cap() { - use std::os::unix::fs::PermissionsExt as _; - - let temporary = tempfile::tempdir().unwrap(); - let executable = temporary.path().join("flood"); - std::fs::write( - &executable, - "#!/bin/sh\nprintf START; head -c 1048576 /dev/zero; printf END\n", - ) - .unwrap(); - std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); - - let output = - output_full_stdout_with_timeout(&mut Command::new(&executable), Duration::from_secs(5)) - .unwrap(); - - assert!(output.stdout.len() > CAPTURE_CAP_BYTES); - assert!( - output.stdout.starts_with(b"START") && output.stdout.ends_with(b"END"), - "full-stdout variant truncated structured output: {} bytes", - output.stdout.len() - ); - } - - /// Proves the shared reaper actually waits: the killed child is observed as a zombie BEFORE - /// `reap_detached` runs, so only the reaper's `wait()` can clear that state. - #[cfg(target_os = "linux")] - #[test] - fn the_shared_reaper_reaps_a_killed_child() { - let mut child = Command::new("sh") - .arg("-c") - .arg("sleep 60") - .spawn() - .unwrap(); - let pid = child.id() as i32; - unsafe { - libc::kill(pid, libc::SIGKILL); - } - let _ = child.kill(); - let deadline = Instant::now() + Duration::from_secs(1); - while linux_process_state(pid) != Some('Z') && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(10)); - } - assert_eq!( - linux_process_state(pid), - Some('Z'), - "fixture did not produce a zombie" - ); - - reap_detached(child); - let deadline = Instant::now() + Duration::from_secs(2); - while linux_process_state(pid) == Some('Z') && Instant::now() < deadline { - std::thread::sleep(Duration::from_millis(10)); - } - assert_ne!( - linux_process_state(pid), - Some('Z'), - "the shared reaper did not reap the killed child {pid}" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn undrained_reader_does_not_retain_the_nonblocking_writer() { - use std::os::fd::{FromRawFd as _, OwnedFd}; - - let mut pipe_fds = [0; 2]; - assert_eq!( - unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC) }, - 0 - ); - let reader = unsafe { OwnedFd::from_raw_fd(pipe_fds[0]) }; - let writer = unsafe { OwnedFd::from_raw_fd(pipe_fds[1]) }; - let pipe = std::fs::read_link(format!("/proc/self/fd/{}", reader.as_raw_fd())).unwrap(); - let started = Instant::now(); - - assert!( - !write_all_before( - ChildStdin::from(writer), - &vec![b'x'; 1024 * 1024], - Instant::now() + Duration::from_millis(100), - ) - .unwrap() - ); - let retained_writers = std::fs::read_dir("/proc/self/fd") - .unwrap() - .filter_map(Result::ok) - .filter_map(|entry| std::fs::read_link(entry.path()).ok()) - .filter(|target| target == &pipe) - .count(); - - assert!(started.elapsed() < Duration::from_secs(1)); - assert_eq!( - retained_writers, 1, - "the undrained pipe retained a writer after the deadline" - ); - } - - #[cfg(target_os = "linux")] - #[test] - fn expired_write_deadline_prevents_further_progress() { - use std::os::fd::{FromRawFd as _, OwnedFd}; - - let mut pipe_fds = [0; 2]; - assert_eq!( - unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC) }, - 0 - ); - let _reader = unsafe { OwnedFd::from_raw_fd(pipe_fds[0]) }; - let writer = unsafe { OwnedFd::from_raw_fd(pipe_fds[1]) }; - - assert!( - !write_all_before(ChildStdin::from(writer), b"x", Instant::now()).unwrap(), - "an expired child deadline still allowed stdin progress" - ); - } - - #[test] - fn expired_cleanup_deadline_hands_reaping_off_without_blocking() { - let child = Command::new("sleep").arg("60").spawn().unwrap(); - let pid = child.id() as i32; - let started = Instant::now(); - terminate_and_reap_before(child, pid, Instant::now()); - - assert!( - started.elapsed() < Duration::from_secs(1), - "expired cleanup deadline blocked the caller" - ); - let reap_deadline = Instant::now() + Duration::from_secs(1); - while crate::host_lock::process_alive(pid) && Instant::now() < reap_deadline { - std::thread::sleep(Duration::from_millis(10)); - } - assert!( - !crate::host_lock::process_alive(pid), - "background reaper did not collect child {pid}" - ); - } - - #[test] - fn build_run_command_passes_direct_argv_without_a_shell() { - let cli = PtyCli::new(PathBuf::from("/my/catalog")); - let mut t = target("hetz.demo.agent", "unused"); - t.launch = TaskLaunch::Argv(vec![ - "axe".into(), - "agent".into(), - "exec".into(), - "--".into(), - "claude".into(), - "--resume".into(), - "$CATALOG/session id".into(), - ]); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - let args = cmd - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect::>(); - let sep = args.iter().position(|arg| arg == "--").unwrap(); - - assert_eq!( - &args[sep + 1..], - [ - "axe", - "agent", - "exec", - "--", - "claude", - "--resume", - "/my/catalog/session id" - ] - ); - assert!(!args[sep + 1..].iter().any(|arg| arg == "sh")); - } - - #[test] - fn build_run_command_expands_direct_argv_with_the_managed_agent_environment() { - let cli = PtyCli::new(PathBuf::from("/eval/catalog")); - let mut t = target("local.worker", "unused"); - t.env.insert("ST_AGENT".into(), "local.worker".into()); - t.env.insert("ST_ROOT".into(), "/eval/catalog".into()); - t.launch = TaskLaunch::Argv(vec![ - "claude".into(), - "$ST_AGENT reads $ST_ROOT and $CATALOG".into(), - ]); - - let cmd = cli.build_run_command(&t, Path::new("/eval/catalog/local/worker")); - let args = cmd - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect::>(); - let separator = args.iter().position(|arg| arg == "--").unwrap(); - - assert_eq!( - &args[separator + 1..], - [ - "claude", - "local.worker reads /eval/catalog and /eval/catalog" - ] - ); - } - - #[test] - fn build_run_command_persists_the_complete_managed_environment_before_the_command() { - let cli = PtyCli::new(PathBuf::from("/my/catalog")); - let mut t = target("hetz.demo.agent", "exec codex 'boot'"); - t.env.insert("CUSTOM".into(), "task-value".into()); - t.env.insert("ST_AGENT".into(), "hetz.demo".into()); - t.env.insert("ST_ROOT".into(), "$CATALOG/custom-bus".into()); - t.env.insert("TERM".into(), "screen-256color".into()); - t.env - .insert("PTY_ROOT".into(), "/declared/root/must-not-win".into()); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - - let args: Vec = cmd - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect(); - let separator = args.iter().position(|arg| arg == "--").unwrap(); - let mut persisted = BTreeMap::new(); - let mut index = 0; - while index < separator { - if args[index] == "--env" { - let (key, value) = args[index + 1].split_once('=').unwrap(); - assert!( - persisted - .insert(key.to_string(), value.to_string()) - .is_none(), - "the final managed overlay needs only one persisted value per key" - ); - index += 2; - } else { - index += 1; - } - } - let inherited = cmd - .get_envs() - .filter_map(|(key, value)| { - value.map(|value| { - ( - key.to_string_lossy().into_owned(), - value.to_string_lossy().into_owned(), - ) - }) - }) - .collect::>(); - assert_eq!( - persisted, inherited, - "initial process env and restart-persisted env must be the same resolved overlay" - ); - assert_eq!( - persisted.get("CATALOG").map(String::as_str), - Some("/my/catalog") - ); - assert_eq!( - persisted.get("ST_ROOT").map(String::as_str), - Some("/my/catalog/custom-bus") - ); - assert_eq!( - persisted.get("PTY_ROOT").map(String::as_str), - Some( - effective_pty_root(&cli.catalog_root) - .to_string_lossy() - .as_ref() - ) - ); - assert_eq!( - persisted.get("TERM").map(String::as_str), - Some("screen-256color") - ); - assert_eq!( - persisted.get("ST_AGENT").map(String::as_str), - Some("hetz.demo") - ); - assert_eq!( - persisted.get("CUSTOM").map(String::as_str), - Some("task-value") - ); - assert!(persisted.contains_key("ST_HOOKS")); - } - - #[test] - fn build_run_command_omits_an_alias_equal_to_the_lifecycle_id() { - let cli = PtyCli::default(); - let mut t = target("hetz.demo", "exec codex 'boot'"); - t.bus_id = t.pty_id.clone(); - t.presentation = Some(PtyPresentation { - pty_id: t.pty_id.clone(), - display_name: Some(Some(t.pty_id.clone())), - tags: BTreeMap::new(), - }); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - let args: Vec = cmd - .get_args() - .map(|a| a.to_string_lossy().into_owned()) - .collect(); - - assert_eq!( - args.iter() - .position(|arg| arg == "--id") - .map(|position| args[position + 1].as_str()), - Some("hetz.demo") - ); - assert!( - !args.iter().any(|arg| arg == "--name"), - "pty rejects a display name equal to the stable session id" - ); - assert!( - args.iter().any(|arg| arg == "--no-display-name"), - "without this flag pty would create an unrelated automatic alias" - ); - } - - #[test] - fn build_run_command_defaults_cwd_to_spec_dir_and_passes_tags_and_env() { - let cli = PtyCli::default(); - let mut t = target("hetz.demo.agent", "exec claude 'boot'"); - t.tags.insert("role".into(), "agent".into()); - t.env.insert("ST_AGENT".into(), "hetz.demo-claude".into()); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - - let args: Vec = cmd - .get_args() - .map(|a| a.to_string_lossy().into_owned()) - .collect(); - let cwd_pos = args.iter().position(|a| a == "--cwd").unwrap(); - assert_eq!(args[cwd_pos + 1], "/cat/hetz/demo"); // no cwd, no workspace → spec dir - let tag_pos = args.iter().position(|a| a == "--tag").unwrap(); - assert_eq!(args[tag_pos + 1], "role=agent"); - - // env injected onto the child process - let envs: BTreeMap> = cmd - .get_envs() - .map(|(k, v)| { - ( - k.to_string_lossy().into_owned(), - v.map(|v| v.to_string_lossy().into_owned()), - ) - }) - .collect(); - assert_eq!( - envs.get("ST_AGENT"), - Some(&Some("hetz.demo-claude".to_string())) - ); - assert_eq!( - envs.get("TERM"), - Some(&Some("xterm-256color".to_string())), - "headless st2 launches must not pass TERM=dumb into an interactive harness" - ); - assert!( - envs.get("ST_HOOKS") - .and_then(Option::as_deref) - .is_some_and(|path| !path.contains("/sets/sha256-")), - "managed tasks keep ST_HOOKS at the receipt-bearing root; only rendered hook commands use a versioned set" - ); - } - - #[test] - fn managed_agent_scrubs_ambient_no_color_unless_explicitly_declared() { - let cli = PtyCli::default(); - let agent = target("hetz.demo.agent", "exec claude 'boot'"); - let command = cli.build_run_command(&agent, Path::new("/cat/hetz/demo")); - assert_eq!( - command - .get_envs() - .find(|(key, _)| *key == OsStr::new("NO_COLOR")) - .map(|(_, value)| value), - Some(None), - "ambient NO_COLOR must not silently disable an interactive agent's color" - ); - let args = command - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect::>(); - assert!( - args.windows(2) - .any(|pair| pair == ["--unset-env", "NO_COLOR"]), - "the removal must be persisted for PTY restart" - ); - - let mut explicit = target("hetz.explicit.agent", "exec claude 'boot'"); - explicit.env.insert("NO_COLOR".into(), "1".into()); - let command = cli.build_run_command(&explicit, Path::new("/cat/hetz/explicit")); - assert_eq!( - command - .get_envs() - .find(|(key, _)| *key == OsStr::new("NO_COLOR")) - .and_then(|(_, value)| value), - Some(OsStr::new("1")), - "an explicit Agent Spec NO_COLOR remains authoritative" - ); - let args = command - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect::>(); - assert!( - !args - .windows(2) - .any(|pair| pair == ["--unset-env", "NO_COLOR"]), - "an explicit assignment must not also persist a removal" - ); - } - - #[test] - fn non_agent_task_does_not_claim_no_color_policy() { - let cli = PtyCli::default(); - let mut task = target("hetz.demo.sidecar", "exec sleep 1"); - task.name = "sidecar".into(); - let command = cli.build_run_command(&task, Path::new("/cat/hetz/demo")); - - assert!( - command - .get_envs() - .all(|(key, _)| key != OsStr::new("NO_COLOR")), - "non-agent services keep the caller's ambient NO_COLOR semantics" - ); - let args = command - .get_args() - .map(|arg| arg.to_string_lossy().into_owned()) - .collect::>(); - assert!( - !args - .windows(2) - .any(|pair| pair == ["--unset-env", "NO_COLOR"]), - "non-agent services must not persist an st2-owned removal" - ); - } - - #[test] - fn isolation_wrapper_preserves_environment_removals() { - let mut inner = Command::new("pty"); - inner.env("TERM", "xterm-256color").env_remove("NO_COLOR"); - let mut outer = Command::new("systemd-run"); - - apply_command_env(&inner, &mut outer); - - let env = outer - .get_envs() - .map(|(key, value)| (key.to_os_string(), value.map(OsStr::to_os_string))) - .collect::>(); - assert_eq!( - env.get(OsStr::new("TERM")).and_then(Option::as_deref), - Some(OsStr::new("xterm-256color")) - ); - assert_eq!(env.get(OsStr::new("NO_COLOR")), Some(&None)); - } - - #[test] - fn build_run_command_allows_a_task_to_override_the_default_term() { - let cli = PtyCli::default(); - let mut t = target("hetz.demo.agent", "exec codex 'boot'"); - t.env.insert("TERM".into(), "screen-256color".into()); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - let term = cmd - .get_envs() - .find(|(key, _)| *key == OsStr::new("TERM")) - .and_then(|(_, value)| value) - .map(|value| value.to_string_lossy().into_owned()); - assert_eq!(term.as_deref(), Some("screen-256color")); - } - - #[test] - fn build_run_command_defaults_cwd_to_workspace_when_task_cwd_absent() { - let cli = PtyCli::default(); - let mut t = target("hetz.demo.agent", "exec claude 'boot'"); - t.workspace = Some("/repos/demo".into()); // no task cwd → workspace (spec.md §2) - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - let args: Vec = cmd - .get_args() - .map(|a| a.to_string_lossy().into_owned()) - .collect(); - let cwd_pos = args.iter().position(|a| a == "--cwd").unwrap(); - assert_eq!(args[cwd_pos + 1], "/repos/demo"); - } - - #[test] - fn build_run_command_expands_catalog_var_and_sets_it_in_env() { - let cli = PtyCli::new(PathBuf::from("/my/catalog")); - let mut t = target("hetz.demo.agent", "run"); - t.env.insert("DATA".into(), "$CATALOG/evals/x".into()); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - let envs: BTreeMap> = cmd - .get_envs() - .map(|(k, v)| { - ( - k.to_string_lossy().into_owned(), - v.map(|v| v.to_string_lossy().into_owned()), - ) - }) - .collect(); - assert_eq!( - envs.get("DATA"), - Some(&Some("/my/catalog/evals/x".to_string())) - ); - assert_eq!(envs.get("CATALOG"), Some(&Some("/my/catalog".to_string()))); - } - - #[test] - fn build_run_command_expands_vars_in_env_cwd_and_tags_but_not_command() { - // Unique var name so the process-global set_var can't collide with a parallel test. - let key = "ST2_TEST_EXPAND_NET_9f3"; - unsafe { std::env::set_var(key, "/net/xyz") } - - let cli = PtyCli::default(); - let mut t = target("hetz.demo.agent", "exec claude $ST2_TEST_EXPAND_NET_9f3/go"); - t.cwd = Some(format!("${key}/work")); - t.tags.insert("net".into(), format!("${key}")); - t.env.insert("ST_ROOT".into(), format!("${key}/custom-bus")); - let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); - - let args: Vec = cmd - .get_args() - .map(|a| a.to_string_lossy().into_owned()) - .collect(); - // cwd expanded (absolute → replaces the spec dir) - let cwd_pos = args.iter().position(|a| a == "--cwd").unwrap(); - assert_eq!(args[cwd_pos + 1], "/net/xyz/work"); - // tag value expanded - let tag_pos = args.iter().position(|a| a == "--tag").unwrap(); - assert_eq!(args[tag_pos + 1], "net=/net/xyz"); - // command left verbatim for sh -c to expand at spawn - assert_eq!( - args.last().unwrap(), - "exec claude $ST2_TEST_EXPAND_NET_9f3/go" - ); - - // env value expanded - let envs: std::collections::BTreeMap> = cmd - .get_envs() - .map(|(k, v)| { - ( - k.to_string_lossy().into_owned(), - v.map(|v| v.to_string_lossy().into_owned()), - ) - }) - .collect(); - assert_eq!( - envs.get("ST_ROOT"), - Some(&Some("/net/xyz/custom-bus".to_string())) - ); - - unsafe { std::env::remove_var(key) } - } - - #[test] - fn resolve_cwd_honors_relative_absolute_workspace_and_default() { - let cli = PtyCli::default(); - let mut t = target("x", "y"); - // relative cwd → joined onto the spec dir - t.cwd = Some("sub".into()); - assert_eq!( - cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), - Path::new("/cat/hetz/demo/sub") - ); - // absolute cwd → replaces - t.cwd = Some("/repos/fabric".into()); - assert_eq!( - cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), - Path::new("/repos/fabric") - ); - // no cwd but a workspace → workspace - t.cwd = None; - t.workspace = Some("/repos/ws".into()); - assert_eq!( - cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), - Path::new("/repos/ws") - ); - // neither → spec dir - t.workspace = None; - assert_eq!( - cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), - Path::new("/cat/hetz/demo") - ); - } - - #[test] - fn detect_host_returns_a_nonempty_short_name() { - let h = detect_host(); - assert!(!h.is_empty()); - assert!(!h.contains('.'), "short name only, got {h}"); - } - - #[test] - fn task_observation_of_missing_pty_root_is_complete_and_does_not_create_it() { - let tmp = tempfile::tempdir().unwrap(); - let catalog = tmp.path().join("catalog"); - std::fs::create_dir(&catalog).unwrap(); - std::fs::write( - catalog.join("catalog.kdl"), - format!( - "catalog {{ pty-root {:?} }}\n", - tmp.path().join("missing-pty").display().to_string() - ), - ) - .unwrap(); - let root = effective_pty_root_from(&catalog, None); - assert!(!root.exists()); - let batch = PtyCli::new(catalog).task_observations(&HashSet::from(["h.worker"])); - assert!(batch.complete, "{:?}", batch.errors); - assert!(batch.observations.is_empty()); - assert!(!root.exists(), "read-only observation created the PTY root"); - } - - #[test] - fn unreadable_pty_root_evidence_is_indeterminate_not_absent() { - let tmp = tempfile::tempdir().unwrap(); - let catalog = tmp.path().join("catalog"); - let loop_path = tmp.path().join("pty-loop"); - std::fs::create_dir(&catalog).unwrap(); - std::os::unix::fs::symlink(&loop_path, &loop_path).unwrap(); - std::fs::write( - catalog.join("catalog.kdl"), - format!( - "catalog {{ pty-root {:?} }}\n", - loop_path.display().to_string() - ), - ) - .unwrap(); - let batch = PtyCli::new(catalog) - .task_observations_at_root(&HashSet::from(["h.worker"]), &loop_path); - assert!(!batch.complete); - assert!(batch.observations.is_empty()); - assert!( - batch.errors[0].contains("cannot inspect PTY root"), - "{:?}", - batch.errors - ); - } - - #[test] - fn removed_and_recreated_pty_root_is_indeterminate_not_absent() { - use std::os::unix::fs::PermissionsExt; - - let tmp = tempfile::tempdir().unwrap(); - let root = tmp.path().join("pty"); - std::fs::create_dir(&root).unwrap(); - let fake = tmp.path().join("pty-bin"); - std::fs::write( - &fake, - r#"#!/bin/sh -rmdir "$PTY_ROOT" -mkdir "$PTY_ROOT" -printf '%s\n' '[]' -printf '' > "$0.ready.tmp" -mv "$0.ready.tmp" "$0.ready" -while [ ! -e "$0.release" ]; do sleep 0.01; done -"#, - ) - .unwrap(); - let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); - permissions.set_mode(0o755); - std::fs::set_permissions(&fake, permissions).unwrap(); - - reset_fixture_barrier(&fake); - let observed_fake = fake.clone(); - let batch = PtyCli { - bin: fake.display().to_string(), - catalog_root: tmp.path().join("catalog"), - on_command_spawn: Some(std::sync::Arc::new(move |pid| { - release_ready_fixture( - pid, - &observed_fake, - "fake PTY inventory was not published", - ); - })), - } - .task_observations_at_root(&HashSet::from(["h.worker"]), &root); - assert!(!batch.complete); - assert!(batch.observations.is_empty()); - assert!( - batch.errors[0].contains("changed identity during observation"), - "{:?}", - batch.errors - ); - } - - #[test] - fn pty_stats_rejects_pid_reuse_between_registry_snapshot_and_start_token_capture() { - let initial = PtyListEntry { - name: "h.worker".into(), - status: "running".into(), - exit_code: None, - pid: Some(41), - created_at: Some("2026-09-05T10:00:00.000Z".into()), - display_name: None, - tags: BTreeMap::new(), - }; - let stats = |alive: bool, created_at: &str| PtyStatsEntry { - name: "h.worker".into(), - process: Some(PtyStatsProcess { alive }), - daemon: Some(PtyStatsDaemon { pid: 41 }), - created_at: Some(created_at.into()), - }; - - // PID 41 has been reused. A token captured after the registry snapshot - // would describe the replacement, but its live socket reports a new - // creation generation and prevents that token from being admitted. - let replacement = stats(true, "2026-09-05T10:00:01.000Z"); - assert_eq!( - confirm_pty_generation(&initial, &[replacement]), - Err(ResourceTargetUnavailableReason::GenerationChanged) - ); - - let exited = stats(false, "2026-09-05T10:00:00.000Z"); - assert_eq!( - confirm_pty_generation(&initial, &[exited]), - Err(ResourceTargetUnavailableReason::ProcessUnavailable) - ); - let stable = stats(true, "2026-09-05T10:00:00.000Z"); - assert_eq!(confirm_pty_generation(&initial, &[stable]), Ok(())); - } - - #[test] - fn pty_resource_observation_uses_one_stats_snapshot_and_preserves_generation_id() { - use std::os::unix::fs::PermissionsExt; - - let tmp = tempfile::tempdir().unwrap(); - let root = tmp.path().join("pty"); - std::fs::create_dir(&root).unwrap(); - let invocations = tmp.path().join("invocations"); - let failed_once = tmp.path().join("stats-failed-once"); - let pid = std::process::id(); - let fake = tmp.path().join("pty-bin"); - std::fs::write( - &fake, - format!( - r#"#!/bin/sh -printf '%s\n' "$*" >> {invocations:?} -case "$1" in - list) - printf '%s\n' '[{{"name":"h.a","status":"running","pid":{pid},"createdAt":"2026-09-05T10:00:00.000Z"}},{{"name":"h.b","status":"running","pid":{pid},"createdAt":"2026-09-05T10:00:01.000Z"}}]' - ;; - stats) - if [ ! -e {failed_once:?} ]; then - : > {failed_once:?} - exit 1 - fi - printf '%s\n' '[{{"name":"h.a","process":{{"alive":true}},"daemon":{{"pid":{pid}}},"createdAt":"2026-09-05T10:00:00.000Z"}},{{"name":"h.b","process":{{"alive":true}},"daemon":{{"pid":{pid}}},"createdAt":"2026-09-05T10:00:01.000Z"}}]' - ;; -esac -"#, - invocations = invocations, - failed_once = failed_once, - ), - ) - .unwrap(); - let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); - permissions.set_mode(0o755); - std::fs::set_permissions(&fake, permissions).unwrap(); - - let cli = PtyCli { - bin: fake.display().to_string(), - catalog_root: tmp.path().join("catalog"), - on_command_spawn: None, - }; - let desired_ids = HashSet::from(["h.a", "h.b"]); - let unavailable = cli.task_observations_at_root(&desired_ids, &root); - let available = cli.task_observations_at_root(&desired_ids, &root); - assert!(unavailable.complete, "{:?}", unavailable.errors); - assert!(available.complete, "{:?}", available.errors); - assert_eq!(available.observations.len(), 2); - assert_eq!( - std::fs::read_to_string(invocations).unwrap(), - "list --json\nstats --json\nlist --json\nstats --json\n" - ); - for (before, after) in unavailable - .observations - .iter() - .zip(&available.observations) - { - let (ObservedState::Running(before), ObservedState::Running(after)) = - (&before.state, &after.state) - else { - panic!("live PTY lost generation"); - }; - assert!(matches!( - before.resource_target(), - ResourceTarget::Unavailable { .. } - )); - assert!(!matches!( - after.resource_target(), - ResourceTarget::Unavailable { .. } - )); - assert_eq!( - before.generation_id(), - after.generation_id(), - "transient target proof changed stable PTY generation identity" - ); - } - } - - #[test] - fn pty_task_observation_preserves_exact_generation_and_closed_states() { - use std::os::unix::fs::PermissionsExt; - - let tmp = tempfile::tempdir().unwrap(); - let catalog = tmp.path().join("catalog"); - let pty_root = tmp.path().join("pty"); - std::fs::create_dir_all(&catalog).unwrap(); - std::fs::create_dir(&pty_root).unwrap(); - std::fs::write( - catalog.join("catalog.kdl"), - format!( - "catalog {{ pty-root {:?} }}\n", - pty_root.display().to_string() - ), - ) - .unwrap(); - let fake = tmp.path().join("pty-bin"); - std::fs::write( - &fake, - r#"#!/bin/sh -printf '%s\n' '[{"name":"h.live","status":"running","pid":41,"createdAt":"2026-07-31T10:00:00.000Z","displayName":"Build owner","tags":{"agent.presentation.schema":"1","unrelated":"preserved"}},{"name":"h.exit","status":"exited","exitCode":0,"pid":42,"createdAt":"2026-07-31T09:00:00.000Z"},{"name":"h.gone","status":"vanished","pid":43,"createdAt":"2026-07-31T08:00:00.000Z"}]' -printf '' > "$0.ready.tmp" -mv "$0.ready.tmp" "$0.ready" -while [ ! -e "$0.release" ]; do sleep 0.01; done -"#, - ) - .unwrap(); - let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); - permissions.set_mode(0o755); - std::fs::set_permissions(&fake, permissions).unwrap(); - - let observed_fake = fake.clone(); - let cli = PtyCli { - bin: fake.display().to_string(), - catalog_root: catalog, - on_command_spawn: Some(std::sync::Arc::new(move |pid| { - release_ready_fixture( - pid, - &observed_fake, - "fake PTY inventory was not published", - ); - })), - }; - let desired = HashSet::from(["h.live", "h.exit", "h.gone"]); - reset_fixture_barrier(&fake); - let first = cli.task_observations(&desired); - reset_fixture_barrier(&fake); - let second = cli.task_observations(&desired); - assert!(first.complete, "{:?}", first.errors); - assert_eq!(first, second, "same PTY evidence changed generation"); - let ObservedState::Running(generation) = &first.observations[0].state else { - panic!("running PTY lost generation: {:?}", first.observations[0]); - }; - assert_eq!(generation.pid(), 41); - assert_eq!(generation.created_at(), "2026-07-31T10:00:00.000Z"); - assert!(generation.generation_id().starts_with("sha256:")); - assert_eq!(first.observations[1].state, ObservedState::Exited); - assert_eq!(first.observations[2].state, ObservedState::Vanished); - - reset_fixture_barrier(&fake); - let sessions = cli.list_sessions().unwrap(); - let presentation = sessions[0].presentation.as_ref().unwrap(); - assert_eq!(presentation.display_name.as_deref(), Some("Build owner")); - assert_eq!( - presentation - .tags - .get("agent.presentation.schema") - .map(String::as_str), - Some("1") - ); - assert_eq!( - presentation.tags.get("unrelated").map(String::as_str), - Some("preserved") - ); - } - - #[test] - fn running_pty_without_complete_generation_is_indeterminate() { - use std::os::unix::fs::PermissionsExt; - - let tmp = tempfile::tempdir().unwrap(); - let catalog = tmp.path().join("catalog"); - let pty_root = tmp.path().join("pty"); - std::fs::create_dir_all(&catalog).unwrap(); - std::fs::create_dir(&pty_root).unwrap(); - std::fs::write( - catalog.join("catalog.kdl"), - format!( - "catalog {{ pty-root {:?} }}\n", - pty_root.display().to_string() - ), - ) - .unwrap(); - let fake = tmp.path().join("pty-bin"); - std::fs::write( - &fake, - r#"#!/bin/sh -printf '%s\n' '[{"name":"h.live","status":"running"}]' -printf '' > "$0.ready.tmp" -mv "$0.ready.tmp" "$0.ready" -while [ ! -e "$0.release" ]; do sleep 0.01; done -"#, - ) - .unwrap(); - let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); - permissions.set_mode(0o755); - std::fs::set_permissions(&fake, permissions).unwrap(); - reset_fixture_barrier(&fake); - let observed_fake = fake.clone(); - let batch = PtyCli { - bin: fake.display().to_string(), - catalog_root: catalog, - on_command_spawn: Some(std::sync::Arc::new(move |pid| { - release_ready_fixture( - pid, - &observed_fake, - "fake PTY inventory was not published", - ); - })), - } - .task_observations(&HashSet::from(["h.live"])); - assert!(!batch.complete); - assert!(matches!( - batch.observations[0].state, - ObservedState::Indeterminate(_) - )); - } -} +#[cfg(test)] +mod tests; diff --git a/src/run/process.rs b/src/run/process.rs new file mode 100644 index 00000000..b8050e58 --- /dev/null +++ b/src/run/process.rs @@ -0,0 +1,277 @@ +//! Child-process execution with bounded output capture. +//! +//! Moved verbatim out of the parent module: the capture cap, the bounded reader, and every +//! deadline-bounded `Command` runner the supervisor uses. + +use std::io::{Read as _, Seek as _, Write as _}; +use std::os::fd::AsRawFd as _; +use std::os::unix::process::CommandExt as _; +use std::process::{Child, ChildStdin, Command, Output, Stdio}; +use std::time::{Duration, Instant}; + +use super::*; + +/// Per-stream cap for captured child diagnostics. Tail-preserving: when output exceeds the cap, +/// the LAST [`CAPTURE_CAP_BYTES`] bytes are kept — recent output is what a failure message needs, +/// and an uncapped capture lets one chatty child balloon sidecar memory without bound. +pub(crate) const CAPTURE_CAP_BYTES: usize = 256 * 1024; + +/// One captured child stream capped to [`CAPTURE_CAP_BYTES`], keeping the tail. +pub(crate) struct BoundedStream { + pub bytes: Vec, // last <= cap bytes + pub total: u64, // complete stream size before capping +} + +impl BoundedStream { + pub fn truncated(&self) -> bool { + self.total as usize > self.bytes.len() + } +} + +/// Read back at most `cap` bytes of a temp-file capture, preserving the tail. The file is stat'ed +/// and seek'ed straight to `len - cap`, so the cost is O(cap) no matter how much the child wrote. +pub(crate) fn read_bounded_tail( + file: &mut std::fs::File, + cap: usize, +) -> std::io::Result { + let total = file.metadata()?.len(); + let skip = total.saturating_sub(cap as u64); + file.seek(std::io::SeekFrom::Start(skip))?; + let mut bytes = Vec::with_capacity((total - skip) as usize); + file.take(cap as u64).read_to_end(&mut bytes)?; + Ok(BoundedStream { bytes, total }) +} + +/// Send an already-killed child to ONE shared reaper thread instead of spawning a detached thread +/// per timed-out child: under a timeout storm one-thread-per-child accumulates without bound. +/// The thread starts lazily on first use. +pub(crate) fn reap_detached(child: std::process::Child) { + static REAPER: std::sync::LazyLock> = + std::sync::LazyLock::new(|| { + let (sender, receiver) = std::sync::mpsc::channel::(); + // Thread-spawn exhaustion is the only failure mode; panicking here surfaces it at the + // call site instead of silently leaking unreaped children. + std::thread::Builder::new() + .name("st2-child-reaper".to_string()) + .spawn(move || { + for mut child in receiver { + let _ = child.wait(); + } + }) + .expect("spawn shared child reaper thread"); + sender + }); + let _ = REAPER.send(child); +} + +/// Run a non-interactive child with bounded output capture: each stream keeps at most its last +/// [`CAPTURE_CAP_BYTES`] bytes (tail-preserving, with a diagnostic line on truncation). Regular +/// temporary files keep an escaped descendant that inherited stdout/stderr from blocking cleanup +/// after the direct child times out. +/// The child still gets a fresh process group so the common wrapper-and-descendants case is reaped. +#[cfg(test)] +pub(super) fn output_with_timeout(command: &mut Command, timeout: Duration) -> anyhow::Result { + output_with_input_timeout(command, timeout, None) +} + +pub(super) fn terminate_and_reap_before(mut child: Child, pid: i32, deadline: Instant) { + unsafe { + libc::kill(-pid, libc::SIGKILL); + } + let _ = child.kill(); + loop { + match child.try_wait() { + Ok(Some(_)) => return, + Ok(None) if Instant::now() < deadline => { + std::thread::sleep( + deadline + .saturating_duration_since(Instant::now()) + .min(Duration::from_millis(20)), + ); + } + Ok(None) | Err(_) => { + reap_detached(child); + return; + } + } + } +} + +pub(super) fn write_all_before( + mut stdin: ChildStdin, + mut input: &[u8], + deadline: Instant, +) -> anyhow::Result { + let fd = stdin.as_raw_fd(); + let flags = unsafe { libc::fcntl(fd, libc::F_GETFL) }; + if flags == -1 { + return Err(std::io::Error::last_os_error()).context("read metadata stdin flags"); + } + if unsafe { libc::fcntl(fd, libc::F_SETFL, flags | libc::O_NONBLOCK) } == -1 { + return Err(std::io::Error::last_os_error()).context("make metadata stdin nonblocking"); + } + while !input.is_empty() { + if Instant::now() >= deadline { + return Ok(false); + } + match stdin.write(input) { + Ok(0) => { + return Err(std::io::Error::from(std::io::ErrorKind::WriteZero)) + .context("write metadata patch payload"); + } + Ok(written) => input = &input[written..], + Err(error) if error.kind() == std::io::ErrorKind::Interrupted => {} + Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => { + if Instant::now() >= deadline { + return Ok(false); + } + std::thread::sleep( + deadline + .saturating_duration_since(Instant::now()) + .min(Duration::from_millis(20)), + ); + } + Err(error) => return Err(error).context("write metadata patch payload"), + } + } + Ok(true) +} + +pub(super) fn output_with_input_timeout( + command: &mut Command, + timeout: Duration, + input: Option>, +) -> anyhow::Result { + output_with_input_timeout_observed(command, timeout, input, |_| {}) +} + +/// `on_spawn` observes the direct child's pid at the moment it exists. The child is `setsid`, so +/// that pid is also its process group id — the group this function signals on every failure path. +/// Tests need it to assert the child was reaped, and the child cannot supply it: a test whose +/// deadline expires before the child is first scheduled would never see anything the child wrote. +/// +/// It runs BEFORE the child deadline starts, and tests rely on that: a lifecycle test blocks in +/// `on_spawn` until the fixture reached the state it wants to measure, so fork+exec scheduling is +/// paid outside the deadline instead of out of it. See +/// `tests::the_spawn_observer_runs_before_the_child_deadline_starts`. +pub(super) fn output_with_input_timeout_observed( + command: &mut Command, + timeout: Duration, + input: Option>, + on_spawn: impl FnOnce(i32), +) -> anyhow::Result { + run_captured(command, timeout, input, on_spawn, false) +} + +/// Like [`output_with_timeout`], but returns the COMPLETE stdout: callers parse structured data +/// (e.g. `pty list --json`) that must be whole, and capping it would corrupt the parse for large +/// fleets. Stdout is therefore intentionally uncapped — one chatty child can balloon this buffer. +/// Stderr stays tail-capped at [`CAPTURE_CAP_BYTES`] with a diagnostic line on truncation, +/// because stderr is only surfaced inside error messages. +pub(crate) fn output_full_stdout_with_timeout( + command: &mut Command, + timeout: Duration, +) -> anyhow::Result { + run_captured(command, timeout, None, |_| {}, true) +} + +#[cfg(test)] +pub(super) fn output_full_stdout_with_timeout_observed( + command: &mut Command, + timeout: Duration, + on_spawn: impl FnOnce(i32), +) -> anyhow::Result { + run_captured(command, timeout, None, on_spawn, true) +} + +/// Shared spawn/wait/read-back core. The child is `setsid`, so its pid is also its process group +/// id — the group this function signals on every failure path. +pub(super) fn run_captured( + command: &mut Command, + timeout: Duration, + input: Option>, + on_spawn: impl FnOnce(i32), + full_stdout: bool, +) -> anyhow::Result { + let mut stdout = tempfile::tempfile()?; + let mut stderr = tempfile::tempfile()?; + command + .stdin(if input.is_some() { + Stdio::piped() + } else { + Stdio::null() + }) + .stdout(Stdio::from(stdout.try_clone()?)) + .stderr(Stdio::from(stderr.try_clone()?)); + unsafe { + command.pre_exec(|| { + if libc::setsid() == -1 { + Err(std::io::Error::last_os_error()) + } else { + Ok(()) + } + }); + } + let mut child = command.spawn()?; + let pid = child.id() as i32; + on_spawn(pid); + // Load-bearing order: the deadline starts after `on_spawn` returns, so a test that blocks there + // as a readiness barrier spends none of `timeout` on fork+exec. Moving this line above + // `on_spawn` is silent in production and makes every barrier test load-sensitive again. + let deadline = Instant::now() + timeout; + if let Some(input) = input { + let Some(stdin) = child.stdin.take() else { + terminate_and_reap_before(child, pid, deadline); + anyhow::bail!("metadata patch child has no piped stdin"); + }; + match write_all_before(stdin, &input, deadline) { + Ok(true) => {} + Ok(false) => { + terminate_and_reap_before(child, pid, deadline); + anyhow::bail!("timed out after {:.1}s", timeout.as_secs_f64()); + } + Err(error) => { + terminate_and_reap_before(child, pid, deadline); + return Err(error); + } + } + } + let status = loop { + if let Some(status) = child.try_wait()? { + break status; + } + if Instant::now() >= deadline { + terminate_and_reap_before(child, pid, deadline); + anyhow::bail!("timed out after {:.1}s", timeout.as_secs_f64()); + } + std::thread::sleep(Duration::from_millis(20)); + }; + let stdout_stream = if full_stdout { + // Intentionally uncapped: callers parse structured data that must be whole. + stdout.rewind()?; + let mut bytes = Vec::new(); + stdout.read_to_end(&mut bytes)?; + BoundedStream { + total: bytes.len() as u64, + bytes, + } + } else { + read_bounded_tail(&mut stdout, CAPTURE_CAP_BYTES)? + }; + let stderr_stream = read_bounded_tail(&mut stderr, CAPTURE_CAP_BYTES)?; + let program = command.get_program().to_string_lossy(); + for (stream, name) in [(&stdout_stream, "stdout"), (&stderr_stream, "stderr")] { + if stream.truncated() { + eprintln!( + "st2: truncated {name} capture of `{program}`: keeping last {} of {} bytes (cap {CAPTURE_CAP_BYTES})", + stream.bytes.len(), + stream.total, + ); + } + } + Ok(Output { + status, + stdout: stdout_stream.bytes, + stderr: stderr_stream.bytes, + }) +} diff --git a/src/run/tests.rs b/src/run/tests.rs new file mode 100644 index 00000000..b20c9c19 --- /dev/null +++ b/src/run/tests.rs @@ -0,0 +1,3702 @@ +use super::*; +use std::os::fd::AsRawFd as _; +use std::process::ChildStdin; +use agent_spec::spec::{ + AgentSpec, Driver, JobType, OmpDriver, Task, TaskKind, TaskLifecycle, +}; +use std::cell::{Cell, RefCell}; +use std::collections::{BTreeMap, BTreeSet}; +use std::ffi::OsStr; +use std::sync::atomic::{AtomicUsize, Ordering as AtomicOrdering}; +use std::sync::mpsc; + + +/// The bound is derived from the resolved pty root, never a fixed maximum identity length. +/// +/// `pty` binds `/.sock`, so the separator plus the five-byte suffix is +/// the fixed overhead and the usable identity length is whatever remains of the limit. These +/// numbers are measured against the pty binary itself: with a 21-byte root it accepts a +/// 77-byte id and refuses a 78-byte one as "a socket path of 105 bytes, which exceeds the +/// 104-byte kernel limit by 1". +#[test] +fn session_socket_overage_is_derived_from_the_resolved_root() { + let short_root = Path::new("/tmp/ptyprobe-1960953"); + let fits = "a".repeat(77); + let over = "a".repeat(78); + + assert_eq!( + session_socket_path(short_root, &fits) + .as_os_str() + .as_encoded_bytes() + .len(), + PORTABLE_SOCKET_PATH_LIMIT, + "the accepted id must land exactly on the limit" + ); + assert!(session_socket_overage(short_root, &fits).is_none()); + + let (path, overage) = + session_socket_overage(short_root, &over).expect("one byte over is refused"); + assert_eq!(overage, 1); + assert_eq!(path, short_root.join(format!("{over}.sock"))); + + // A deeper root shrinks every identity's budget on that host: the same id that fitted + // above is now 26 bytes over. + let deep_root = Path::new("/home/user/.local/state/st2/default/catalog/pty"); + assert_eq!( + session_socket_overage(deep_root, &fits).map(|(_, over)| over), + Some(26) + ); +} + +#[cfg(target_os = "linux")] +fn linux_process_state(pid: i32) -> Option { + std::fs::read_to_string(format!("/proc/{pid}/stat")) + .ok()? + .rsplit_once(") ")? + .1 + .chars() + .next() +} + +/// Block until the fixture publishes `marker`, which its script creates by an atomic rename so +/// the barrier never observes a half-written file. Called from `on_spawn`, which runs before +/// [`run_captured`] starts the child deadline: fork+exec scheduling is therefore paid here and +/// not out of the deadline the test then measures. The ceiling is deliberately far larger than +/// any plausible fork+exec — it bounds a fixture that never ran at all, and is not itself the +/// behaviour under test, so a loaded host cannot turn it into a failure. +fn await_fixture_ready(pid: i32, marker: &Path, what: &str) { + const CEILING: Duration = Duration::from_secs(30); + let deadline = Instant::now() + CEILING; + while !marker.exists() { + if Instant::now() >= deadline { + // Do not leak the fixture's long sleeper into the test host on the way out. + unsafe { + libc::kill(-pid, libc::SIGKILL); + libc::kill(pid, libc::SIGKILL); + } + panic!( + "{what} within {CEILING:?}: {} never appeared", + marker.display() + ); + } + std::thread::sleep(Duration::from_millis(5)); + } +} + +fn fixture_barrier_path(executable: &Path, suffix: &str) -> PathBuf { + PathBuf::from(format!("{}.{suffix}", executable.display())) +} + +fn reset_fixture_barrier(executable: &Path) { + for suffix in ["ready", "release"] { + let marker = fixture_barrier_path(executable, suffix); + match std::fs::remove_file(&marker) { + Ok(()) => {} + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => panic!("remove stale fixture marker {}: {error}", marker.display()), + } + } +} + +fn release_ready_fixture(pid: i32, executable: &Path, what: &str) { + let ready = fixture_barrier_path(executable, "ready"); + await_fixture_ready(pid, &ready, what); + std::fs::write(fixture_barrier_path(executable, "release"), b"go\n").unwrap(); +} + +fn process_can_retain_cleanup_resources(pid: i32) -> bool { + #[cfg(target_os = "linux")] + if linux_process_state(pid) == Some('Z') { + return false; + } + crate::host_lock::process_alive(pid) +} + +fn target(id: &str, cmd: &str) -> TaskTarget { + TaskTarget { + kind: TaskKind::Pty, + pty_id: id.to_string(), + bus_id: "hetz.demo".to_string(), + name: "agent".to_string(), + derived: false, + launch: TaskLaunch::Shell(cmd.to_string()), + cwd: None, + workspace: None, + tags: BTreeMap::new(), + env: BTreeMap::new(), + keep: false, + presentation: None, + } +} + +struct GateRunner { + list_calls: Cell, +} + +impl Runner for GateRunner { + fn list_sessions(&self) -> anyhow::Result> { + self.list_calls.set(self.list_calls.get() + 1); + Ok(Vec::new()) + } + + fn spawn(&self, _target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { + panic!("gate runner must not spawn") + } + + fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { + panic!("gate runner must not kill") + } + + fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { + panic!("gate runner must not remove") + } +} + +#[derive(Default)] +struct PersistentPatchRunner { + patched: RefCell>, +} + +impl Runner for PersistentPatchRunner { + fn list_sessions(&self) -> anyhow::Result> { + unreachable!("presentation execution does not list sessions") + } + + fn spawn(&self, _target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { + unreachable!("presentation-only plan must not spawn") + } + + fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { + unreachable!("presentation-only plan must not kill") + } + + fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { + unreachable!("presentation-only plan must not remove") + } + + fn patch_presentation(&self, presentation: &PtyPresentation) -> anyhow::Result<()> { + self.patched.borrow_mut().push(presentation.pty_id.clone()); + if presentation.pty_id.as_str() < "host.presented.08" { + anyhow::bail!("simulated persistent metadata failure"); + } + Ok(()) + } +} + +#[test] +fn bounded_presentation_batches_are_deterministic_and_do_not_starve() { + let plan = ReconcilePlan { + presentation: (0..12) + .rev() + .map(|index| PtyPresentation { + pty_id: format!("host.presented.{index:02}"), + display_name: None, + tags: BTreeMap::new(), + }) + .collect(), + ..ReconcilePlan::default() + }; + let runner = PersistentPatchRunner::default(); + let mut cap = FlappingCap::default(); + let mut cursor = PresentationPatchCursor::default(); + + for _ in 0..2 { + execute_with_presentation_cursor( + &plan, + &runner, + &mut cap, + &mut cursor, + &mut UpReport::default(), + &mut |_| {}, + ); + } + + let attempted = runner.patched.borrow(); + assert_eq!( + &attempted[..8], + &(0..8) + .map(|index| format!("host.presented.{index:02}")) + .collect::>() + ); + assert_eq!(attempted.len(), 16); + for index in 8..12 { + assert!(attempted.contains(&format!("host.presented.{index:02}"))); + } +} + +#[test] +fn lifecycle_hook_consumer_is_a_closed_enum() { + let consumers = [ + lifecycle_hook_consumer(true, false, false), + lifecycle_hook_consumer(false, true, false), + lifecycle_hook_consumer(false, false, true), + lifecycle_hook_consumer(true, true, false), + lifecycle_hook_consumer(true, false, true), + lifecycle_hook_consumer(false, true, true), + lifecycle_hook_consumer(true, true, true), + ] + .into_iter() + .collect::>(); + assert_eq!( + consumers, + BTreeSet::from([ + "codex", + "pi", + "omp", + "codex+pi", + "codex+omp", + "pi+omp", + "codex+pi+omp", + ]) + ); +} + +#[test] +fn selected_codex_gate_suppresses_launch_on_stale_hooks() { + let spec = AgentSpec { + id: None, + address: None, + identity: "codex".into(), + name: None, + description: None, + host: None, + role: None, + job_type: JobType::Service, + workspace: None, + supervisor: None, + desired_state: crate::AgentDesiredState::Running, + keep: false, + restart: None, + delivery: None, + session_driver: None, + driver: None, + delivery_readiness: None, + resources: vec![], + streams: Vec::new(), + tasks: vec![Task { + kind: TaskKind::Pty, + derived: false, + name: "agent".into(), + id: Some("test.codex.agent".into()), + command: None, + argv: Some(vec!["$CATALOG/bin/codex".into(), "--version".into()]), + cwd: None, + tags: BTreeMap::new(), + env: BTreeMap::new(), + keep: false, + lifecycle: TaskLifecycle::Service, + }], + path: "/tmp/spec.kdl".into(), + }; + let runner = GateRunner { + list_calls: Cell::new(0), + }; + let report = up_once_selected_specs_with_gates( + Path::new("/tmp"), + &[spec], + "test.codex.agent", + "test", + &runner, + |consumer| { + assert_eq!(consumer, None); + anyhow::bail!("stale receipt") + }, + ) + .unwrap(); + assert_eq!(runner.list_calls.get(), 1); + assert!(report.launched.is_empty()); + assert!(report.errors.iter().any(|error| { + error.contains("stale receipt") && error.contains("launch suppressed") + })); +} + +#[test] +fn selected_identity_conflict_refuses_before_hook_verification_or_inventory() { + let mut spec = AgentSpec { + id: None, + address: None, + identity: "codex".into(), + name: None, + description: None, + host: None, + role: None, + job_type: JobType::Service, + workspace: None, + supervisor: None, + desired_state: crate::AgentDesiredState::Running, + keep: false, + restart: None, + delivery: None, + session_driver: None, + driver: None, + delivery_readiness: None, + resources: vec![], + streams: Vec::new(), + tasks: vec![Task { + kind: TaskKind::Pty, + derived: false, + name: "agent".into(), + id: Some("test.codex.agent".into()), + command: None, + argv: Some(vec!["$CATALOG/bin/codex".into(), "--version".into()]), + cwd: None, + tags: BTreeMap::new(), + env: BTreeMap::new(), + keep: false, + lifecycle: TaskLifecycle::Service, + }], + path: "/tmp/spec.kdl".into(), + }; + spec.tasks[0] + .env + .insert("ST_AGENT".into(), "wrong.actor".into()); + let runner = GateRunner { + list_calls: Cell::new(0), + }; + let verify_calls = Cell::new(0); + + let error = up_once_selected_specs_with_gates( + Path::new("/tmp"), + &[spec], + "test.codex.agent", + "test", + &runner, + |_| { + verify_calls.set(verify_calls.get() + 1); + Ok(()) + }, + ) + .unwrap_err(); + + assert!(error.to_string().contains("conflicting ST_AGENT")); + assert_eq!(verify_calls.get(), 0); + assert_eq!(runner.list_calls.get(), 0); +} + +#[cfg(target_os = "linux")] +#[test] +fn idle_supervisor_does_not_spin_on_its_own_catalog_reads() { + let catalog = tempfile::tempdir().unwrap(); + let stop = AtomicBool::new(false); + let mut passes = 0usize; + + std::thread::scope(|scope| { + scope.spawn(|| { + std::thread::sleep(Duration::from_millis(350)); + stop.store(true, Ordering::SeqCst); + }); + up_loop_until( + catalog.path(), + "test-host", + &GateRunner { + list_calls: Cell::new(0), + }, + Duration::from_secs(60), + &stop, + best_effort_catalog_watcher, + |_| passes += 1, + ) + .unwrap(); + }); + + assert!( + passes <= 2, + "idle supervisor must wait instead of reconciling its own read events: {passes} passes" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn failed_watch_installation_keeps_supervisor_on_timer_cadence() { + let catalog = tempfile::tempdir().unwrap(); + let agent = catalog.path().join("agents/test-host/live"); + std::fs::create_dir_all(&agent).unwrap(); + std::fs::write( + agent.join("agent.kdl"), + r#"agent "live" { host "test-host"; command "x" }"#, + ) + .unwrap(); + let stop = AtomicBool::new(false); + let mut passes = 0usize; + let (started_tx, started_rx) = mpsc::sync_channel(1); + + std::thread::scope(|scope| { + let stop = &stop; + scope.spawn(move || { + started_rx.recv().unwrap(); + std::thread::sleep(Duration::from_millis(350)); + stop.store(true, Ordering::SeqCst); + }); + up_loop_until( + catalog.path(), + "test-host", + &SpawnCountingRunner::default(), + Duration::from_millis(100), + &stop, + |_, _| None, // watcher installation fails, as it did on dev3's oversized catalog + |_| { + passes += 1; + let _ = started_tx.try_send(()); + }, + ) + .unwrap(); + }); + + assert!( + (2..=6).contains(&passes), + "a disconnected watcher channel must fall back to timer cadence, not spin: \ + {passes} passes in ~350ms at a 100ms interval" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn supervisor_wakes_and_launches_a_new_direct_declaration() { + let catalog = tempfile::tempdir().unwrap(); + let agent = catalog.path().join("agents/test-host/live"); + let stop = AtomicBool::new(false); + let runner = SpawnCountingRunner::default(); + let (done_tx, done_rx) = mpsc::sync_channel(1); + let mut passes = 0usize; + + std::thread::scope(|scope| { + let watchdog_stop = &stop; + scope.spawn(move || { + if done_rx.recv_timeout(Duration::from_secs(5)).is_err() { + watchdog_stop.store(true, Ordering::SeqCst); + } + }); + up_loop_until( + catalog.path(), + "test-host", + &runner, + Duration::from_secs(60), + &stop, + best_effort_catalog_watcher, + |report| { + passes += 1; + match passes { + 1 => { + std::fs::create_dir_all(&agent).unwrap(); + std::fs::write( + agent.join("agent.kdl"), + r#"agent "live" { host "test-host"; command "x" }"#, + ) + .unwrap(); + } + 2 => { + assert_eq!(report.launched, ["test-host.live"]); + let _ = done_tx.send(()); + stop.store(true, Ordering::SeqCst); + } + _ => panic!("one direct declaration must cause exactly one prompt pass"), + } + }, + ) + .unwrap(); + }); + + assert_eq!(passes, 2, "the 60s timer must not be the publication path"); + assert_eq!(runner.spawned.borrow().as_slice(), ["test-host.live"]); +} + +#[test] +fn resident_loop_reloads_added_changed_removed_and_malformed_profiles() { + let catalog = tempfile::tempdir().unwrap(); + let agent = catalog.path().join("agents/test-host/live"); + std::fs::create_dir_all(&agent).unwrap(); + std::fs::write( + agent.join("agent.kdl"), + r#"agent "live" { + host "test-host" + command "true" + resource "alpha" uri="alpha://test-host/live" reason="Alpha." + resource "beta" uri="beta://test-host/live" reason="Beta." +}"#, + ) + .unwrap(); + let missing = catalog.path().join("missing.wasm"); + let profile = |scheme: &str| { + format!( + "profile {scheme:?} {{ wasm {:?} }}\n", + missing.display().to_string() + ) + }; + let config = crate::catalog::config_path(catalog.path()); + let runner = SpawnCountingRunner::default(); + runner + .sessions + .borrow_mut() + .push(sess("test-host.live.agent", true)); + let stop = AtomicBool::new(false); + let mut reports = Vec::new(); + + up_loop_until( + catalog.path(), + "test-host", + &runner, + Duration::from_millis(5), + &stop, + |_, _| None, + |report| { + let pass = reports.len(); + reports.push((report.warnings.clone(), report.errors.clone())); + match pass { + 0 => std::fs::write(&config, profile("alpha")).unwrap(), + 1 => std::fs::write(&config, profile("beta")).unwrap(), + 2 => std::fs::write(&config, "").unwrap(), + 3 => stop.store(true, Ordering::SeqCst), + _ => unreachable!("profile removal run stops after four passes"), + } + }, + ) + .unwrap(); + + let profile_warnings = |reports: &Vec<(Vec, Vec)>, pass: usize| { + reports[pass] + .0 + .iter() + .filter(|warning| warning.contains("resync profile")) + .cloned() + .collect::>() + }; + assert!( + profile_warnings(&reports, 0).is_empty(), + "no profile is initially declared" + ); + assert!( + profile_warnings(&reports, 1) + .iter() + .any(|warning| warning.contains("resource 'alpha'")), + "an added profile takes effect: {:?}", + reports[1] + ); + assert!( + profile_warnings(&reports, 2) + .iter() + .any(|warning| warning.contains("resource 'beta'")) + && !profile_warnings(&reports, 2) + .iter() + .any(|warning| warning.contains("resource 'alpha'")), + "changing definitions replaces the registry: {:?}", + reports[2] + ); + assert!( + profile_warnings(&reports, 3).is_empty(), + "removing every profile removes the old resolution semantics: {:?}", + reports[3] + ); + + // A separate resident lifetime starts valid, then makes the envelope malformed. The + // initial hard parse still accepts the valid declaration; the later edit must clear its + // active semantics rather than silently carrying them forward. + std::fs::write(&config, profile("alpha")).unwrap(); + stop.store(false, Ordering::SeqCst); + let mut malformed_reports = Vec::new(); + up_loop_until( + catalog.path(), + "test-host", + &runner, + Duration::from_millis(5), + &stop, + |_, _| None, + |report| { + let pass = malformed_reports.len(); + malformed_reports.push((report.warnings.clone(), report.errors.clone())); + match pass { + 0 => std::fs::write( + &config, + r#"profiel "alpha" { wasm "missing.wasm" }"#, + ) + .unwrap(), + 1 => stop.store(true, Ordering::SeqCst), + _ => unreachable!("malformed profile run stops after two passes"), + } + }, + ) + .unwrap(); + assert!( + profile_warnings(&malformed_reports, 0) + .iter() + .any(|warning| warning.contains("resource 'alpha'")), + "the profile is active before the malformed edit: {:?}", + malformed_reports[0] + ); + assert!( + malformed_reports[1] + .1 + .iter() + .any(|error| error.contains("unknown catalog.kdl top-level node 'profiel'")) + && profile_warnings(&malformed_reports, 1).is_empty(), + "malformed catalog state is reported and fails closed instead of retaining alpha: {:?}", + malformed_reports[1] + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn disconnected_watcher_channel_waits_out_the_interval_instead_of_spinning() { + let (_tx, rx) = channel::<()>(); + let started = Instant::now(); + assert_eq!( + wait_for_reconcile(&rx, Duration::from_millis(80), &AtomicBool::new(false)), + ReconcileWake::Interval, + "disconnection must be treated as silence, not as a change" + ); + assert!(started.elapsed() >= Duration::from_millis(75)); +} + +// ── liveness debounce (R21c): a transient `pty list` not-alive flicker under load must not +// destructively GC/relaunch a HEALTHY agent; a stable death must still be reaped ────────────── + +use crate::reconcile::Launch; +fn sess(id: &str, alive: bool) -> Session { + Session { + pty_id: id.to_string(), + alive, + exit_code: None, + presentation: None, + } +} + +/// Records spawns and reports every launch as succeeding, so a pass can be driven repeatedly. +#[derive(Default)] +struct SpawnCountingRunner { + sessions: RefCell>, + spawned: RefCell>, +} + +impl Runner for SpawnCountingRunner { + fn list_sessions(&self) -> anyhow::Result> { + Ok(self.sessions.borrow().clone()) + } + + fn spawn(&self, target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { + self.spawned.borrow_mut().push(target.pty_id.clone()); + Ok(()) + } + + fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { + Ok(()) + } + + fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { + Ok(()) + } + + fn patch_presentation(&self, _presentation: &PtyPresentation) -> anyhow::Result<()> { + Ok(()) + } +} + +/// `execute` must close every pass, or a task that recovers is never forgiven and eventually +/// parks even though it is healthy — the opposite of the crash-loop bug the cap exists for. +/// +/// The unit tests in `flapping.rs` call `end_pass` by hand, so they cannot catch it never being +/// called from a reconcile pass. This one drives the real `execute` path. `interval = 0s` makes +/// any survived pass count as recovery, keeping the test free of wall-clock sleeping. +#[test] +fn execute_closes_each_pass_so_a_recovered_task_regains_its_fail_budget() { + let mut spec = spec_fixture(); + spec.restart = Some(agent_spec::spec::Restart { + attempts: 3, + interval: Duration::from_secs(0), + delay: Duration::from_secs(0), + mode: agent_spec::spec::RestartMode::Fail, + }); + let runner = SpawnCountingRunner::default(); + let mut cap = FlappingCap::default(); + + fn dying(spec: &AgentSpec) -> ReconcilePlan<'_> { + ReconcilePlan { + launch: vec![Launch { + spec, + tasks: vec![target("hetz.demo.agent", "x")], + live_derived: Vec::new(), + }], + ..ReconcilePlan::default() + } + } + + // Two failing passes: two of three launches spent. + for _ in 0..2 { + execute(&dying(&spec), &runner, &mut cap, &mut UpReport::default()); + } + assert_eq!(runner.spawned.borrow().len(), 2, "two launches spent"); + + // A pass that launches nothing because it found the task alive. That observation — not the + // empty launch set — is what forgives the budget. + execute( + &ReconcilePlan { + live: vec!["hetz.demo.agent".to_string()], + ..ReconcilePlan::default() + }, + &runner, + &mut cap, + &mut UpReport::default(), + ); + + // Having recovered, it gets the full budget back: three more launches, then parked. Without + // the pass being closed it would park after only one more. + let mut last = UpReport::default(); + for _ in 0..4 { + last = UpReport::default(); + execute(&dying(&spec), &runner, &mut cap, &mut last); + } + assert_eq!( + runner.spawned.borrow().len(), + 5, + "recovery must restore the full `attempts` budget, not leave it partly spent" + ); + assert_eq!( + last.flapping, + vec!["hetz.demo.agent".to_string()], + "and it still parks in the end" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn persistent_advisory_warnings_surface_once_not_per_pass() { + let catalog = tempfile::tempdir().unwrap(); + let agent = catalog.path().join("agents/test-host/live"); + std::fs::create_dir_all(&agent).unwrap(); + std::fs::create_dir_all(catalog.path().join("workspace")).unwrap(); + std::fs::write( + agent.join("agent.kdl"), + r#"agent "live" { + host "test-host" + command "true" + workspace "$CATALOG/workspace" + render { git-exclude "scratch.txt" } +}"#, + ) + .unwrap(); + let stop = AtomicBool::new(false); + let mut passes = 0usize; + let mut warnings_seen = 0usize; + let (started_tx, started_rx) = mpsc::sync_channel(1); + + std::thread::scope(|scope| { + let stop = &stop; + scope.spawn(move || { + started_rx.recv().unwrap(); + std::thread::sleep(Duration::from_millis(300)); + stop.store(true, Ordering::SeqCst); + }); + up_loop_until( + catalog.path(), + "test-host", + &SpawnCountingRunner::default(), + Duration::from_millis(50), + &stop, + |_, _| None, + |report| { + passes += 1; + warnings_seen += report.warnings.len(); + let _ = started_tx.try_send(()); + }, + ) + .unwrap(); + }); + + assert!( + passes >= 3, + "the loop must have run several passes for this to say anything: {passes}" + ); + assert_eq!( + warnings_seen, 1, + "an unchanged advisory failure must be diagnosed once across {passes} passes" + ); +} +/// A pass can execute a plan the task was never in: `up_once` drops an owner whose +/// materialization failed, `gate_harness_launches_on_hooks` strips gated launches, and +/// `defer_flickers` removes debounced ones — each after the pass is already committed to +/// running. Silence about a task is not evidence it is alive, and crediting uptime for it lets +/// a permanently-dead task refill its budget on every gated pass and never park. Identical to +/// the recovery test above except that the quiet pass does not report the task live. +#[test] +fn a_pass_that_omits_a_task_does_not_credit_it_with_uptime() { + let mut spec = spec_fixture(); + spec.restart = Some(agent_spec::spec::Restart { + attempts: 3, + interval: Duration::from_secs(0), + delay: Duration::from_secs(0), + mode: agent_spec::spec::RestartMode::Fail, + }); + let runner = SpawnCountingRunner::default(); + let mut cap = FlappingCap::default(); + + fn dying(spec: &AgentSpec) -> ReconcilePlan<'_> { + ReconcilePlan { + launch: vec![Launch { + spec, + tasks: vec![target("hetz.demo.agent", "x")], + live_derived: Vec::new(), + }], + ..ReconcilePlan::default() + } + } + + // Two failing passes: two of three launches spent. + for _ in 0..2 { + execute(&dying(&spec), &runner, &mut cap, &mut UpReport::default()); + } + assert_eq!(runner.spawned.borrow().len(), 2, "two launches spent"); + + // The task is dropped from this pass — not launched, and not observed alive either. + execute( + &ReconcilePlan::default(), + &runner, + &mut cap, + &mut UpReport::default(), + ); + + // The budget must be where the failures left it: one launch remains, then it parks. + let mut last = UpReport::default(); + for _ in 0..4 { + last = UpReport::default(); + execute(&dying(&spec), &runner, &mut cap, &mut last); + } + assert_eq!( + runner.spawned.borrow().len(), + 3, + "an unobserved pass must not forgive the failure budget" + ); + assert_eq!( + last.flapping, + vec!["hetz.demo.agent".to_string()], + "and the task must still park" + ); +} + +fn spec_fixture() -> AgentSpec { + AgentSpec { + id: None, + address: None, + identity: "demo".into(), + name: None, + description: None, + host: Some("hetz".into()), + role: None, + job_type: JobType::Service, + workspace: None, + supervisor: None, + desired_state: crate::AgentDesiredState::Running, + keep: false, + restart: None, + delivery: None, + session_driver: None, + driver: None, + delivery_readiness: None, + resources: vec![], + streams: Vec::new(), + tasks: vec![], + path: std::path::PathBuf::from("/x"), + } +} + +#[test] +fn driver_labels_include_typed_and_argv_omp_but_remain_bounded() { + let legacy_spec = spec_fixture(); + let legacy_launch = Launch { + spec: &legacy_spec, + tasks: Vec::new(), + live_derived: Vec::new(), + }; + let mut omp_argv = target("hetz.demo.agent", "unused"); + omp_argv.launch = TaskLaunch::Argv(vec![ + "st2".into(), + "driver".into(), + "omp-session".into(), + ]); + let mut exec = target("hetz.demo.agent", "codex"); + exec.kind = TaskKind::Exec; + let targets = [ + target("hetz.demo.agent", "codex"), + target("hetz.demo.agent", "claude"), + target("hetz.demo.agent", "opencode"), + target("hetz.demo.agent", "pi"), + omp_argv, + exec, + target("hetz.demo.agent", "unrecognized"), + ]; + let labels = targets + .iter() + .map(|target| driver_label(&legacy_launch, target)) + .collect::>(); + assert_eq!( + labels, + BTreeSet::from(["codex", "claude", "opencode", "pi", "omp", "exec", "other"]) + ); + + let mut typed_spec = spec_fixture(); + typed_spec.driver = Some(Driver::Omp(OmpDriver { + model: None, + effort: None, + prompt: String::new(), + args: Vec::new(), + })); + let typed_launch = Launch { + spec: &typed_spec, + tasks: Vec::new(), + live_derived: Vec::new(), + }; + assert_eq!( + driver_label(&typed_launch, &target("hetz.demo.agent", "claude")), + "omp", + "typed driver identity must take precedence over argv heuristics" + ); +} + +#[test] +fn resync_watch_eligibility_requires_a_proven_live_agent_seat() { + let spec = |identity: &str, explicit_id: Option<&str>| { + let mut spec = spec_fixture(); + spec.identity = identity.to_owned(); + spec.tasks = vec![Task { + kind: TaskKind::Pty, + derived: false, + name: "agent".into(), + id: explicit_id.map(str::to_owned), + command: Some("agent".into()), + argv: None, + cwd: None, + tags: BTreeMap::new(), + env: BTreeMap::new(), + keep: false, + lifecycle: TaskLifecycle::Service, + }]; + spec + }; + let specs = vec![ + spec("desired", None), + spec("dead-adopted", None), + spec("observed-live", None), + spec("launched", None), + spec("restarted", Some("custom-seat")), + ]; + let sessions = vec![ + sess("hetz.dead-adopted.agent", false), + // A live canonical seat remains eligible even when a missing companion means the + // whole spec was not adopted and the companion later fails to launch. + sess("hetz.observed-live.agent", true), + ]; + let report = UpReport { + adopted: vec!["dead-adopted".into()], + launched: vec![ + "hetz.launched.agent".into(), + // A successfully launched companion is not evidence of a live agent seat. + "hetz.desired.ding".into(), + ], + restarted: vec!["custom-seat".into()], + ..UpReport::default() + }; + + let eligible = live_resync_specs(&specs, "hetz", &sessions, &report) + .into_iter() + .map(|spec| spec.identity) + .collect::>(); + assert_eq!(eligible, vec!["observed-live", "launched", "restarted"]); +} + +#[test] +fn subscription_eligibility_excludes_non_running_agents_even_with_a_live_seat() { + // A retired or suspended agent whose canonical seat is still alive mid-teardown owns no + // live subscription work: its resync installs and resource-Profile bindings must be + // stripped this pass, not left running until the seat dies (dotfiles#1535). The declaration + // (including its `resource` bindings) is untouched — only the runtime work stops. + let seat = || Task { + kind: TaskKind::Pty, + derived: false, + name: "agent".into(), + id: None, + command: Some("agent".into()), + argv: None, + cwd: None, + tags: BTreeMap::new(), + env: BTreeMap::new(), + keep: false, + lifecycle: TaskLifecycle::Service, + }; + let with_state = |identity: &str, state: crate::AgentDesiredState| { + let mut spec = spec_fixture(); + spec.identity = identity.to_owned(); + spec.tasks = vec![seat()]; + spec.desired_state = state; + spec + }; + let specs = vec![ + with_state("running", crate::AgentDesiredState::Running), + with_state( + "retired", + crate::AgentDesiredState::Retired { + reason: Some("Mission complete".into()), + }, + ), + with_state( + "suspended", + crate::AgentDesiredState::Suspended { + reason: "Waiting for capacity".into(), + }, + ), + ]; + // Every seat is observed alive, so only desired state can distinguish them. + let sessions = vec![ + sess("hetz.running.agent", true), + sess("hetz.retired.agent", true), + sess("hetz.suspended.agent", true), + ]; + let eligible = live_resync_specs(&specs, "hetz", &sessions, &UpReport::default()) + .into_iter() + .map(|spec| spec.identity) + .collect::>(); + assert_eq!(eligible, vec!["running"]); +} + +struct BlockingLaunchRunner { + sessions: RefCell>, + fail_id: Option, + block_id: String, + entered: mpsc::SyncSender<()>, + release: RefCell>, +} + +impl Runner for BlockingLaunchRunner { + fn list_sessions(&self) -> anyhow::Result> { + Ok(self.sessions.borrow().clone()) + } + + fn spawn(&self, target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { + if self.fail_id.as_deref() == Some(&target.pty_id) { + anyhow::bail!("simulated launch failure"); + } + if target.pty_id == self.block_id { + self.entered.send(()).unwrap(); + self.release.borrow_mut().recv().unwrap(); + } + Ok(()) + } + + fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { + Ok(()) + } + + fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { + Ok(()) + } +} + +#[cfg(all(test, feature = "wasm-resolver"))] +struct SteadyChainRunner { + sessions: std::sync::Mutex>, + block_id: String, + entered: mpsc::SyncSender<()>, + release: std::sync::Mutex>, +} + +#[cfg(all(test, feature = "wasm-resolver"))] +impl Runner for SteadyChainRunner { + fn list_sessions(&self) -> anyhow::Result> { + Ok(self + .sessions + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .clone()) + } + + fn spawn(&self, target: &TaskTarget, _spec_dir: &Path) -> anyhow::Result<()> { + if target.pty_id == self.block_id { + self.entered.send(()).unwrap(); + self.release + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .recv() + .unwrap(); + } + self.sessions + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .push(sess(&target.pty_id, true)); + Ok(()) + } + + fn kill(&self, _pty_id: &str) -> anyhow::Result<()> { + Ok(()) + } + + fn remove(&self, _pty_id: &str) -> anyhow::Result<()> { + Ok(()) + } +} + +fn write_resync_agent(catalog: &Path, identity: &str) -> (PathBuf, PathBuf) { + let agent_dir = catalog.join("agents/hetz").join(identity); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + format!( + r#"agent "{identity}" {{ + host "hetz" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." +}}"# + ), + ) + .unwrap(); + let goal = resources.join("goal.md"); + std::fs::write(&goal, "before\n").unwrap(); + (agent_dir, goal) +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn write_notify_chain_profile(catalog: &Path) { + let resolver_dir = catalog.join("resolvers"); + std::fs::create_dir_all(&resolver_dir).unwrap(); + std::fs::copy( + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("crates/agent-spec/tests/fixtures/demo_resolver.wasm"), + resolver_dir.join("goal.wasm"), + ) + .unwrap(); + std::fs::write( + crate::catalog::config_path(catalog), + r#"profile "dev.schickling.agent-goal" { + wasm "resolvers/goal.wasm" + class "immediate" + notify-chain #true +} +"#, + ) + .unwrap(); +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn write_notify_chain_agent( + catalog: &Path, + identity: &str, + supervisor: Option<&str>, + later_task: bool, +) -> (PathBuf, PathBuf) { + write_notify_chain_agent_with_state(catalog, identity, supervisor, later_task, None) +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn write_notify_chain_agent_with_state( + catalog: &Path, + identity: &str, + supervisor: Option<&str>, + later_task: bool, + desired_state: Option<&str>, +) -> (PathBuf, PathBuf) { + let agent_dir = catalog.join("agents/hetz").join(identity); + let resources = agent_dir.join("resources"); + std::fs::create_dir_all(&resources).unwrap(); + let supervisor = supervisor + .map(|supervisor| format!(" supervisor {supervisor:?}\n")) + .unwrap_or_default(); + let desired_state = desired_state + .map(|desired_state| format!(" {desired_state}\n")) + .unwrap_or_default(); + let later_task = if later_task { + " exec \"later\" { command \"true\" }\n" + } else { + "" + }; + std::fs::write( + agent_dir.join("agent.kdl"), + format!( + r#"agent "{identity}" {{ + host "hetz" +{supervisor}{desired_state} command "agent" +{later_task} resource "goal" uri="dev.schickling.agent-goal://hetz/{identity}" reason="Layer." +}} +"# + ), + ) + .unwrap(); + let goal = resources.join("goal.md"); + std::fs::write(&goal, "before\n").unwrap(); + (agent_dir, goal) +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn current_resync_event_for_key(agent_dir: &Path, key: &str) -> Option { + let expected = format!("key: {key}"); + std::fs::read_dir(agent_dir.join("resources/inbox")) + .ok()? + .filter_map(Result::ok) + .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) + .find(|body| { + body.lines().any(|line| line == "stream: resync") + && body.lines().any(|line| line == expected) + }) +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn wait_for_resync_event_for_key(agent_dir: &Path, key: &str) -> Option { + let deadline = Instant::now() + Duration::from_secs(5); + loop { + if let Some(body) = current_resync_event_for_key(agent_dir, key) { + return Some(body); + } + if Instant::now() >= deadline { + return None; + } + std::thread::sleep(Duration::from_millis(20)); + } +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn wait_for_resync_event_key_change( + agent_dir: &Path, + key: &str, + prior: &str, +) -> Option { + let deadline = Instant::now() + Duration::from_secs(5); + loop { + if let Some(body) = current_resync_event_for_key(agent_dir, key) + && body != prior + { + return Some(body); + } + if Instant::now() >= deadline { + return None; + } + std::thread::sleep(Duration::from_millis(20)); + } +} + +fn current_resync_event(agent_dir: &Path) -> Option { + std::fs::read_dir(agent_dir.join("resources/inbox")) + .ok()? + .filter_map(Result::ok) + .filter_map(|entry| std::fs::read_to_string(entry.path()).ok()) + .find(|body| body.lines().any(|line| line == "stream: resync")) +} + +fn wait_for_resync_event(agent_dir: &Path) -> Option { + let deadline = Instant::now() + Duration::from_secs(5); + loop { + if let Some(body) = current_resync_event(agent_dir) { + return Some(body); + } + if Instant::now() >= deadline { + return None; + } + std::thread::sleep(Duration::from_millis(20)); + } +} + +fn wait_for_resync_event_change(agent_dir: &Path, prior: &str) -> Option { + let deadline = Instant::now() + Duration::from_secs(5); + loop { + if let Some(body) = current_resync_event(agent_dir) + && body != prior + { + return Some(body); + } + if Instant::now() >= deadline { + return None; + } + std::thread::sleep(Duration::from_millis(20)); + } +} + +#[cfg(feature = "wasm-resolver")] +#[test] +fn up_loop_keeps_complete_notify_chain_sets_during_steady_reconcile() { + let catalog = tempfile::tempdir().unwrap(); + write_notify_chain_profile(catalog.path()); + let (root_dir, root_goal) = + write_notify_chain_agent(catalog.path(), "root", None, false); + let (lead_dir, lead_goal) = + write_notify_chain_agent(catalog.path(), "lead", Some("hetz.root"), false); + let (worker_dir, _worker_goal) = + write_notify_chain_agent(catalog.path(), "worker", Some("hetz.lead"), false); + let specs = crate::discover_strict(catalog.path()).specs; + let task_id = |spec: &AgentSpec, task: &Task| { + task.id + .clone() + .unwrap_or_else(|| format!("{}.{}", spec.bus_id("hetz"), task.name)) + }; + let mut sessions = Vec::new(); + for spec in &specs { + for task in &spec.tasks { + sessions.push(sess(&task_id(spec, task), true)); + } + } + let (entered_tx, entered_rx) = mpsc::sync_channel(0); + let (release_tx, release_rx) = mpsc::channel(); + let runner = SteadyChainRunner { + sessions: std::sync::Mutex::new(sessions), + block_id: "hetz.worker.later".to_owned(), + entered: entered_tx, + release: std::sync::Mutex::new(release_rx), + }; + let stop = AtomicBool::new(false); + let (first_report_tx, first_report_rx) = mpsc::sync_channel(1); + let observer_catalog = catalog.path().to_path_buf(); + + let evidence = std::thread::scope(|scope| { + let observer_stop = &stop; + let observer = scope.spawn(move || { + first_report_rx.recv().unwrap(); + + std::fs::write(&root_goal, "steady baseline transition\n").unwrap(); + let root_initial = wait_for_resync_event_for_key(&root_dir, "goal"); + let lead_initial = + wait_for_resync_event_for_key(&lead_dir, "goal@hetz.root"); + let worker_initial = + wait_for_resync_event_for_key(&worker_dir, "goal@hetz.root"); + + write_notify_chain_agent( + &observer_catalog, + "worker", + Some("hetz.lead"), + true, + ); + let entered = entered_rx.recv_timeout(Duration::from_secs(5)).is_ok(); + let root_after_reconcile = root_initial.as_deref().and_then(|prior| { + std::fs::write(&root_goal, "transition during steady reconcile\n").unwrap(); + wait_for_resync_event_key_change(&root_dir, "goal", prior) + }); + let lead_after_reconcile = lead_initial.as_deref().and_then(|prior| { + wait_for_resync_event_key_change(&lead_dir, "goal@hetz.root", prior) + }); + let worker_after_reconcile = worker_initial.as_deref().and_then(|prior| { + wait_for_resync_event_key_change(&worker_dir, "goal@hetz.root", prior) + }); + + std::fs::write(&lead_goal, "lead transition during steady reconcile\n").unwrap(); + let lead_own = wait_for_resync_event_for_key(&lead_dir, "goal"); + let worker_from_lead = + wait_for_resync_event_for_key(&worker_dir, "goal@hetz.lead"); + + let _ = release_tx.send(()); + observer_stop.store(true, Ordering::SeqCst); + ( + entered, + root_initial, + lead_initial, + worker_initial, + root_after_reconcile, + lead_after_reconcile, + worker_after_reconcile, + lead_own, + worker_from_lead, + ) + }); + up_loop_until( + catalog.path(), + "hetz", + &runner, + Duration::from_millis(25), + &stop, + |_, _| None, + |_| { + let _ = first_report_tx.try_send(()); + }, + ) + .unwrap(); + observer.join().unwrap() + }); + + assert!(evidence.0, "the steady-state reconcile must reach its later task"); + assert!(evidence.1.is_some(), "root must receive its own transition"); + assert!( + evidence.2.is_some() && evidence.3.is_some(), + "root transition must fan out through lead and worker" + ); + assert!( + evidence.4.is_some() && evidence.5.is_some() && evidence.6.is_some(), + "a steady reconcile must not replace chain sets with self-only sets" + ); + assert!( + evidence.7.is_some() && evidence.8.is_some(), + "lead transition must reach lead and worker" + ); + assert_up_loop_full_refresh_keeps_a_retired_middle_as_live_child_topology(); +} + +#[cfg(all(test, feature = "wasm-resolver"))] +fn assert_up_loop_full_refresh_keeps_a_retired_middle_as_live_child_topology() { + for retirement in [ + "retired #true", + "desired-state \"retired\" reason=\"fixture\"", + ] { + let catalog = tempfile::tempdir().unwrap(); + write_notify_chain_profile(catalog.path()); + let (root_dir, root_goal) = + write_notify_chain_agent(catalog.path(), "root", None, false); + let (middle_dir, _middle_goal) = write_notify_chain_agent_with_state( + catalog.path(), + "middle", + Some("hetz.root"), + false, + Some(retirement), + ); + let (child_dir, _child_goal) = + write_notify_chain_agent(catalog.path(), "child", Some("hetz.middle"), false); + let specs = crate::discover_strict(catalog.path()).specs; + let sessions = specs + .iter() + .filter(|spec| spec.desired_state.is_running()) + .flat_map(|spec| { + spec.tasks.iter().map(|task| { + let id = task + .id + .clone() + .unwrap_or_else(|| format!("{}.{}", spec.bus_id("hetz"), task.name)); + sess(&id, true) + }) + }) + .collect(); + let (entered_tx, _entered_rx) = mpsc::sync_channel(1); + let (_release_tx, release_rx) = mpsc::channel(); + let runner = SteadyChainRunner { + sessions: std::sync::Mutex::new(sessions), + block_id: "never-block".to_owned(), + entered: entered_tx, + release: std::sync::Mutex::new(release_rx), + }; + let stop = AtomicBool::new(false); + let missing_supervisor = AtomicBool::new(false); + let (first_report_tx, first_report_rx) = mpsc::sync_channel(1); + + let evidence = std::thread::scope(|scope| { + let observer_stop = &stop; + let observer = scope.spawn(move || { + first_report_rx.recv().unwrap(); + // Let the asynchronous full refresh replace the synchronous install before + // mutating the root carrier. The child must retain the complete catalog chain. + std::thread::sleep(Duration::from_millis(300)); + std::fs::write(&root_goal, "root transition after full refresh\n").unwrap(); + let root_event = wait_for_resync_event_for_key(&root_dir, "goal"); + let child_event = + wait_for_resync_event_for_key(&child_dir, "goal@hetz.root"); + let middle_event = current_resync_event_for_key(&middle_dir, "goal@hetz.root"); + observer_stop.store(true, Ordering::SeqCst); + (root_event, child_event, middle_event) + }); + up_loop_until( + catalog.path(), + "hetz", + &runner, + Duration::from_millis(25), + &stop, + |_, _| None, + |report| { + if report + .errors + .iter() + .chain(&report.warnings) + .any(|message| message.contains("MissingSupervisor")) + { + missing_supervisor.store(true, Ordering::SeqCst); + } + let _ = first_report_tx.try_send(()); + }, + ) + .unwrap(); + observer.join().unwrap() + }); + + assert!( + evidence.0.is_some(), + "root must receive its own event ({retirement})" + ); + assert!( + evidence.1.is_some(), + "the live child must receive exactly its owner-qualified root event through the \ + retired middle after full refresh ({retirement})" + ); + assert!( + evidence.2.is_none(), + "the retired middle must own no active subscription ({retirement})" + ); + assert!( + !missing_supervisor.load(Ordering::SeqCst), + "the complete catalog graph must prevent MissingSupervisor ({retirement})" + ); + } +} + +#[test] +fn compile_invalid_seat_does_not_block_existing_live_resync_watch() { + let catalog = tempfile::tempdir().unwrap(); + let (live_dir, live_goal) = write_resync_agent(catalog.path(), "live"); + let broken_dir = catalog.path().join("agents/hetz/broken"); + let broken_resources = broken_dir.join("resources"); + std::fs::create_dir_all(&broken_resources).unwrap(); + std::fs::create_dir_all(catalog.path().join("broken-workspace")).unwrap(); + let broken_declaration = broken_dir.join("agent.kdl"); + std::fs::write( + &broken_declaration, + r#"agent "broken" { + host "hetz" + deliver "mcp" + workspace "$CATALOG/broken-workspace" + exec "agent" { command "true" } + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let broken_goal = broken_resources.join("goal.md"); + std::fs::write(&broken_goal, "before\n").unwrap(); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + + let runner = SpawnCountingRunner { + sessions: RefCell::new(vec![ + sess("hetz.live", true), + sess("hetz.broken.agent", true), + ]), + ..SpawnCountingRunner::default() + }; + let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + let mut cap = FlappingCap::default(); + let mut debounce = LivenessDebounce::new(DEBOUNCE_GRACE); + let mut presentation_cursor = PresentationPatchCursor::default(); + + let first = reconcile_pass(catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), None); + assert!( + first.errors.iter().any(|error| { + error.contains("compile generated tasks") + && error.contains("non-PTY canonical task") + }), + "{first:#?}" + ); + assert!(!first.skipped, "the valid subset completed its pass"); + assert!( + runner.spawned.borrow().is_empty(), + "the compile-invalid seat must not launch" + ); + + std::fs::write(&live_goal, "changed while compile failed\n").unwrap(); + let first_event = wait_for_resync_event(&live_dir) + .expect("the already-live valid seat must stay watched across the compile error"); + assert!(first_event.contains(r#""binding":"goal""#), "{first_event}"); + + std::fs::write(&broken_goal, "invalid seat changed\n").unwrap(); + std::thread::sleep(Duration::from_millis(750)); + assert!( + current_resync_event(&broken_dir).is_none(), + "a compile-invalid seat must not be watched even when its canonical task is live" + ); + + std::fs::write(&live_goal, "changed while declaration is corrected\n").unwrap(); + std::fs::write( + &broken_declaration, + r#"agent "broken" { + host "hetz" + deliver "mcp" + workspace "$CATALOG/broken-workspace" + pty "agent" { command "true" } + resource "goal" uri="resources/goal.md" reason="Mission." +}"#, + ) + .unwrap(); + let corrected = reconcile_pass(catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), None); + assert!( + corrected + .errors + .iter() + .all(|error| !error.contains("compile generated tasks")), + "{corrected:#?}" + ); + assert!(corrected.launched.is_empty(), "{corrected:#?}"); + assert!( + corrected.adopted.iter().any(|identity| identity == "broken"), + "the corrected already-live seat should be adopted: {corrected:#?}" + ); + let corrected_event = wait_for_resync_event_change(&live_dir, &first_event) + .expect("correcting another declaration must not reseed and hide the live transition"); + assert!(corrected_event.contains(r#""binding":"goal""#), "{corrected_event}"); +} + +#[test] +fn materialization_failure_retains_only_the_observed_live_resync_watch() { + let catalog = tempfile::tempdir().unwrap(); + let write_broken_agent = |identity: &str| { + let (agent_dir, goal) = write_resync_agent(catalog.path(), identity); + let workspace = catalog.path().join(format!("{identity}-workspace")); + std::fs::create_dir_all(&workspace).unwrap(); + std::fs::write( + agent_dir.join("agent.kdl"), + format!( + r#"agent "{identity}" {{ + host "hetz" + workspace "{}" + command "agent" + resource "goal" uri="resources/goal.md" reason="Mission." + render {{ + copy "_templates/{identity}.md" "AGENTS.md" + }} +}}"#, + workspace.display() + ), + ) + .unwrap(); + (agent_dir, goal) + }; + let (live_dir, live_goal) = write_broken_agent("live"); + let (dormant_dir, dormant_goal) = write_broken_agent("dormant"); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + + let runner = SpawnCountingRunner { + sessions: RefCell::new(vec![sess("hetz.live", true)]), + ..SpawnCountingRunner::default() + }; + let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + let mut cap = FlappingCap::default(); + let mut debounce = LivenessDebounce::new(DEBOUNCE_GRACE); + let mut presentation_cursor = PresentationPatchCursor::default(); + + let failed = reconcile_pass(catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), None); + assert!( + failed + .errors + .iter() + .filter(|error| error.contains("copy source")) + .count() + >= 2, + "{failed:#?}" + ); + assert!( + runner.spawned.borrow().is_empty(), + "materialization-failed seats must not launch" + ); + + std::fs::write(&live_goal, "changed while materialization failed\n").unwrap(); + std::fs::write(&dormant_goal, "unwatched while materialization failed\n").unwrap(); + let first_event = wait_for_resync_event(&live_dir) + .expect("the observed live seat must remain watched through materialization failure"); + assert!(first_event.contains(r#""binding":"goal""#), "{first_event}"); + std::thread::sleep(Duration::from_millis(750)); + assert!( + current_resync_event(&dormant_dir).is_none(), + "a materialization-failed seat without an observed live session must stay unwatched" + ); + + std::fs::write(&live_goal, "changed immediately before recovery\n").unwrap(); + std::fs::create_dir_all(catalog.path().join("_templates")).unwrap(); + std::fs::write(catalog.path().join("_templates/live.md"), "rendered\n").unwrap(); + let recovered = reconcile_pass(catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), None); + assert!( + recovered + .errors + .iter() + .all(|error| !error.contains("_templates/live.md")), + "{recovered:#?}" + ); + assert!(recovered.launched.is_empty(), "{recovered:#?}"); + let recovered_event = wait_for_resync_event_change(&live_dir, &first_event) + .expect("recovery must preserve the pending transition instead of silently reseeding"); + assert!( + recovered_event.contains(r#""binding":"goal""#), + "{recovered_event}" + ); +} + +fn execute_resync_plan( + plan: &ReconcilePlan<'_>, + runner: &dyn Runner, + specs: &[AgentSpec], + resync: &crate::resync::ResyncSupervisor, +) -> UpReport { + let mut report = UpReport::default(); + let mut install_count = 0; + execute_with_presentation_cursor( + plan, + runner, + &mut FlappingCap::default(), + &mut PresentationPatchCursor::default(), + &mut report, + &mut |spec| { + install_count += 1; + assert!(resync.install_live(spec, specs, "hetz").is_empty()); + }, + ); + assert!( + resync + .refresh( + specs, + &live_resync_specs(specs, "hetz", &[], &report), + "hetz", + &[], + &[], + ) + .is_empty() + ); + assert!(install_count > 0 || report.launched.is_empty()); + report +} + +#[cfg(feature = "wasm-resolver")] +#[test] +fn notify_chain_launch_boundary_installs_ancestors_before_a_later_task_finishes() { + let catalog = tempfile::tempdir().unwrap(); + write_notify_chain_profile(catalog.path()); + let (_root_dir, root_goal) = + write_notify_chain_agent(catalog.path(), "root", None, false); + write_notify_chain_agent(catalog.path(), "lead", Some("hetz.root"), false); + let (worker_dir, _worker_goal) = + write_notify_chain_agent(catalog.path(), "worker", Some("hetz.lead"), false); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let specs = crate::discover_strict(catalog.path()).specs; + let worker = specs + .iter() + .find(|spec| spec.identity == "worker") + .unwrap(); + let mut later = target("hetz.worker.later", "later"); + later.name = "later".into(); + later.derived = true; + let plan = ReconcilePlan { + launch: vec![Launch { + spec: worker, + tasks: vec![target("hetz.worker.agent", "agent"), later], + live_derived: Vec::new(), + }], + ..ReconcilePlan::default() + }; + let (entered_tx, entered_rx) = mpsc::sync_channel(0); + let (release_tx, release_rx) = mpsc::channel(); + let runner = BlockingLaunchRunner { + sessions: RefCell::new(Vec::new()), + fail_id: None, + block_id: "hetz.worker.later".to_owned(), + entered: entered_tx, + release: RefCell::new(release_rx), + }; + let resync = crate::resync::ResyncSupervisor::with_profiles( + catalog.path().to_path_buf(), + "hetz".into(), + crate::catalog::declared_profiles(catalog.path()).unwrap(), + ); + + let event = std::thread::scope(|scope| { + let observer = scope.spawn(move || { + entered_rx.recv().unwrap(); + std::fs::write(&root_goal, "changed while later task launches\n").unwrap(); + let event = + wait_for_resync_event_for_key(&worker_dir, "goal@hetz.root"); + release_tx.send(()).unwrap(); + event + }); + let report = execute_resync_plan(&plan, &runner, &specs, &resync); + assert_eq!( + report.launched, + ["hetz.worker.agent", "hetz.worker.later"] + ); + observer.join().unwrap() + }) + .expect("the fresh worker must receive its ancestor transition before full refresh"); + assert!(event.contains("key: goal@hetz.root"), "{event}"); +} + +#[test] +fn resync_launch_boundary_seeds_first_seat_before_later_seat_finishes() { + let catalog = tempfile::tempdir().unwrap(); + let (first_dir, first_goal) = write_resync_agent(catalog.path(), "first"); + write_resync_agent(catalog.path(), "second"); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let specs = crate::discover_strict(catalog.path()).specs; + let first = specs.iter().find(|spec| spec.identity == "first").unwrap(); + let second = specs.iter().find(|spec| spec.identity == "second").unwrap(); + let plan = ReconcilePlan { + launch: vec![ + Launch { + spec: first, + tasks: vec![target("hetz.first.agent", "agent")], + live_derived: Vec::new(), + }, + Launch { + spec: second, + tasks: vec![target("hetz.second.agent", "agent")], + live_derived: Vec::new(), + }, + ], + ..ReconcilePlan::default() + }; + let (entered_tx, entered_rx) = mpsc::sync_channel(0); + let (release_tx, release_rx) = mpsc::channel(); + let runner = BlockingLaunchRunner { + sessions: RefCell::new(Vec::new()), + fail_id: None, + block_id: "hetz.second.agent".to_owned(), + entered: entered_tx, + release: RefCell::new(release_rx), + }; + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + + std::thread::scope(|scope| { + scope.spawn(move || { + entered_rx.recv().unwrap(); + std::fs::write(&first_goal, "changed while second launches\n").unwrap(); + std::thread::sleep(Duration::from_secs(1)); + release_tx.send(()).unwrap(); + }); + let report = execute_resync_plan(&plan, &runner, &specs, &resync); + assert_eq!( + report.launched, + ["hetz.first.agent", "hetz.second.agent"] + ); + }); + + let event = wait_for_resync_event(&first_dir) + .expect("the first seat must observe a carrier transition during the later launch"); + assert!(event.contains(r#""binding":"goal""#), "{event}"); +} + +#[test] +fn resync_launch_boundary_excludes_failed_canonical_seat() { + let catalog = tempfile::tempdir().unwrap(); + let (first_dir, first_goal) = write_resync_agent(catalog.path(), "first"); + write_resync_agent(catalog.path(), "second"); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let specs = crate::discover_strict(catalog.path()).specs; + let first = specs.iter().find(|spec| spec.identity == "first").unwrap(); + let second = specs.iter().find(|spec| spec.identity == "second").unwrap(); + let plan = ReconcilePlan { + launch: vec![ + Launch { + spec: first, + tasks: vec![target("hetz.first.agent", "agent")], + live_derived: Vec::new(), + }, + Launch { + spec: second, + tasks: vec![target("hetz.second.agent", "agent")], + live_derived: Vec::new(), + }, + ], + ..ReconcilePlan::default() + }; + let (entered_tx, entered_rx) = mpsc::sync_channel(0); + let (release_tx, release_rx) = mpsc::channel(); + let runner = BlockingLaunchRunner { + sessions: RefCell::new(Vec::new()), + fail_id: Some("hetz.first.agent".to_owned()), + block_id: "hetz.second.agent".to_owned(), + entered: entered_tx, + release: RefCell::new(release_rx), + }; + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + + std::thread::scope(|scope| { + scope.spawn(move || { + entered_rx.recv().unwrap(); + std::fs::write(&first_goal, "changed after failed launch\n").unwrap(); + std::thread::sleep(Duration::from_secs(1)); + release_tx.send(()).unwrap(); + }); + let report = execute_resync_plan(&plan, &runner, &specs, &resync); + assert_eq!(report.launched, ["hetz.second.agent"]); + assert!(report.errors.iter().any(|error| { + error.contains("hetz.first.agent") && error.contains("simulated launch failure") + })); + }); + + std::thread::sleep(Duration::from_millis(750)); + assert!( + current_resync_event(&first_dir).is_none(), + "desired-but-failed canonical seats must remain unwatched" + ); +} + +#[test] +fn dead_resync_seat_is_deactivated_before_its_relaunch_blocks() { + let catalog = tempfile::tempdir().unwrap(); + let (agent_dir, goal) = write_resync_agent(catalog.path(), "worker"); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let (entered_tx, entered_rx) = mpsc::sync_channel(0); + let (release_tx, release_rx) = mpsc::channel(); + let runner = BlockingLaunchRunner { + sessions: RefCell::new(vec![sess("hetz.worker", true)]), + fail_id: None, + block_id: "hetz.worker".to_owned(), + entered: entered_tx, + release: RefCell::new(release_rx), + }; + let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + let mut cap = FlappingCap::default(); + let mut debounce = LivenessDebounce::new(Duration::ZERO); + let mut presentation_cursor = PresentationPatchCursor::default(); + + let seeded = reconcile_pass(catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), None); + assert!(seeded.adopted.iter().any(|identity| identity == "worker")); + *runner.sessions.borrow_mut() = vec![sess("hetz.worker", false)]; + let blocked_goal = goal.clone(); + + let relaunched = std::thread::scope(|scope| { + scope.spawn(move || { + entered_rx.recv().unwrap(); + std::fs::write(&blocked_goal, "changed while replacement launch blocks\n").unwrap(); + std::thread::sleep(Duration::from_secs(1)); + release_tx.send(()).unwrap(); + }); + reconcile_pass(catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), None) + }); + assert_eq!(relaunched.restarted, ["hetz.worker"]); + std::thread::sleep(Duration::from_millis(750)); + assert!( + current_resync_event(&agent_dir).is_none(), + "a carrier mutation while no canonical seat is live must not emit" + ); + + std::fs::write(&goal, "changed after replacement launch\n").unwrap(); + let event = wait_for_resync_event(&agent_dir) + .expect("the successful replacement must receive a fresh silent baseline"); + assert!(event.contains(r#""binding":"goal""#), "{event}"); +} + +/// A publication the resync worker cannot finish must not hold up a reconcile pass. +/// +/// Every per-seat `install_live` handshake is answered by the same worker thread that runs +/// publications, so a publication in progress serializes the whole pass behind it. That is the +/// coupling which let a terminal-refusal loop keep every pass from completing for two hours +/// (#431): the refusals only had power because they denied the pass that would have ended +/// them. Blocking one real publication on the recipient's stream lock is the sharpest form of +/// the same coupling — a slow publication makes a pass late, a stuck one makes it never +/// finish — and it holds the pass at exactly the point `emit_admitted` serializes. +#[test] +fn reconcile_pass_completes_while_a_resync_publication_is_blocked() { + use std::os::fd::AsRawFd as _; + use std::sync::atomic::{AtomicBool, Ordering as AtomicOrdering}; + + let catalog = tempfile::tempdir().unwrap(); + let (agent_dir, goal) = write_resync_agent(catalog.path(), "worker"); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let runner = SpawnCountingRunner { + sessions: RefCell::new(vec![sess("hetz.worker", true)]), + ..Default::default() + }; + let task_context = TaskCompileContext::current(catalog.path().to_path_buf()).unwrap(); + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + let mut cap = FlappingCap::default(); + let mut debounce = LivenessDebounce::new(DEBOUNCE_GRACE); + let mut presentation_cursor = PresentationPatchCursor::default(); + + let seeded = reconcile_pass( + catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), + None, + ); + assert!( + seeded.adopted.iter().any(|identity| identity == "worker"), + "{seeded:#?}" + ); + + // One completed publication first: it creates the recipient's resync stream state + // directory, whose `.lock` is the gate below, and proves the publication path is live. + std::fs::write(&goal, "changed before the gate closes\n").unwrap(); + let published = wait_for_resync_event(&agent_dir) + .expect("the live seat must observe its first carrier transition"); + + let gate = std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(agent_dir.join("resources/streams/resync/.lock")) + .unwrap(); + assert_eq!( + unsafe { libc::flock(gate.as_raw_fd(), libc::LOCK_EX) }, + 0, + "the test must own the stream lock the publication path takes" + ); + let gate_fd = gate.as_raw_fd(); + + // The worker has no other work and `IMMEDIATE_WINDOW` is 500 ms, so it is inside the + // blocked publication well before this wait ends; the unchanged event is the positive + // evidence that the publication has not completed. + std::fs::write(&goal, "changed while the gate is closed\n").unwrap(); + std::thread::sleep(Duration::from_secs(2)); + assert_eq!( + current_resync_event(&agent_dir).as_deref(), + Some(published.as_str()), + "the gate must hold the second publication open" + ); + + // The watchdog releases the gate only when the pass fails to complete on its own, which + // is what separates a decoupled pass from one that merely finished after the rescue. + let rescued = AtomicBool::new(false); + let rescued_flag = &rescued; + let (finished_tx, finished_rx) = mpsc::channel::<()>(); + let pass = std::thread::scope(|scope| { + scope.spawn(move || { + if finished_rx.recv_timeout(Duration::from_secs(20)).is_err() { + rescued_flag.store(true, AtomicOrdering::SeqCst); + unsafe { libc::flock(gate_fd, libc::LOCK_UN) }; + } + }); + let pass = reconcile_pass( + catalog.path(), + "hetz", + &task_context, + &runner, + &mut cap, + &mut debounce, + &mut presentation_cursor, + Some(&resync), + None, + ); + let _ = finished_tx.send(()); + pass + }); + assert!( + !rescued.load(AtomicOrdering::SeqCst), + "the pass only completed after the blocked publication was released: {pass:#?}" + ); + assert!( + pass.adopted.iter().any(|identity| identity == "worker"), + "{pass:#?}" + ); + drop(gate); +} + +#[test] +fn resync_launch_boundary_preserves_baseline_across_derived_companion() { + let catalog = tempfile::tempdir().unwrap(); + let (agent_dir, goal) = write_resync_agent(catalog.path(), "worker"); + crate::event::publish_owner_binding_for_test(catalog.path(), "hetz").unwrap(); + let specs = crate::discover_strict(catalog.path()).specs; + let spec = &specs[0]; + let mut derived = target("hetz.worker.ding", "ding"); + derived.name = "ding".into(); + derived.derived = true; + let plan = ReconcilePlan { + launch: vec![Launch { + spec, + tasks: vec![target("hetz.worker.agent", "agent"), derived], + live_derived: Vec::new(), + }], + ..ReconcilePlan::default() + }; + let (entered_tx, entered_rx) = mpsc::sync_channel(0); + let (release_tx, release_rx) = mpsc::channel(); + let runner = BlockingLaunchRunner { + sessions: RefCell::new(Vec::new()), + fail_id: None, + block_id: "hetz.worker.ding".to_owned(), + entered: entered_tx, + release: RefCell::new(release_rx), + }; + let resync = + crate::resync::ResyncSupervisor::spawn(catalog.path().to_path_buf(), "hetz".into()); + let installs = AtomicUsize::new(0); + let mut report = UpReport::default(); + + std::thread::scope(|scope| { + scope.spawn(move || { + entered_rx.recv().unwrap(); + std::fs::write(&goal, "changed while companion launches\n").unwrap(); + std::thread::sleep(Duration::from_secs(1)); + release_tx.send(()).unwrap(); + }); + execute_with_presentation_cursor( + &plan, + &runner, + &mut FlappingCap::default(), + &mut PresentationPatchCursor::default(), + &mut report, + &mut |spec| { + installs.fetch_add(1, AtomicOrdering::SeqCst); + assert!(resync.install_live(spec, &specs, "hetz").is_empty()); + }, + ); + }); + assert!( + resync + .refresh( + &specs, + &live_resync_specs(&specs, "hetz", &[], &report), + "hetz", + &[], + &[], + ) + .is_empty() + ); + + assert_eq!( + installs.load(AtomicOrdering::SeqCst), + 1, + "only the canonical task transition may install its watch set" + ); + let event = wait_for_resync_event(&agent_dir) + .expect("the companion launch and final refresh must preserve the canonical baseline"); + assert!(event.contains(r#""binding":"goal""#), "{event}"); +} + +#[test] +fn debounce_absorbs_a_gc_flicker_but_reaps_a_stable_death() { + let t0 = Instant::now(); + let mut db = LivenessDebounce::new(Duration::from_secs(10)); + db.observe(&[sess("hetz.demo.agent", true)], t0); + + // Flicker: reads not-alive 1s later but was alive within the grace → deferred (left running). + let mut plan = ReconcilePlan::default(); + plan.gc.push("hetz.demo.agent".into()); + let deferred = db.defer_flickers(&mut plan, t0 + Duration::from_secs(1)); + assert!( + plan.gc.is_empty(), + "a recently-alive flicker must NOT be GC'd" + ); + assert_eq!(deferred, vec!["hetz.demo.agent".to_string()]); + + // CENTRAL anti-over-correction check: a STABLE death past the grace IS still reaped — the + // debounce must never MASK a real death. + let mut plan = ReconcilePlan::default(); + plan.gc.push("hetz.demo.agent".into()); + let deferred = db.defer_flickers(&mut plan, t0 + Duration::from_secs(11)); + assert_eq!( + plan.gc, + vec!["hetz.demo.agent".to_string()], + "a stable death must still be reaped" + ); + assert!(deferred.is_empty()); +} + +#[test] +fn effective_pty_root_prefers_an_exported_ambient_root_else_catalog_pty() { + let cat = std::path::Path::new("/deep/sandbox/st-root"); + // No ambient PTY_ROOT → the rendered default `/pty`. + assert_eq!(effective_pty_root_from(cat, None), cat.join("pty")); + assert_eq!( + effective_pty_root_from(cat, Some("".into())), + cat.join("pty"), + "empty is treated as unset" + ); + // An exported ambient PTY_ROOT (e.g. an eval's short decoupled root) WINS — so a deep catalog + // path can't blow the unix-socket limit, and spawn agrees with list/kill. + let short = std::ffi::OsString::from("/tmp/stev-abc123"); + assert_eq!( + effective_pty_root_from(cat, Some(short)), + std::path::PathBuf::from("/tmp/stev-abc123") + ); +} + +#[test] +fn a_catalog_declared_root_outranks_the_default_but_never_an_ambient_one() { + let tmp = tempfile::tempdir().unwrap(); + let cat = tmp.path(); + std::fs::write( + cat.join(crate::catalog::CONFIG_FILE), + "catalog { pty-root \"/run/agents/pty\" }\n", + ) + .unwrap(); + + // The declaration replaces the `/pty` default for every st2 pty op — so a reader + // that resolves the catalog finds the sessions without being handed an env var. + assert_eq!( + effective_pty_root_from(cat, None), + std::path::PathBuf::from("/run/agents/pty") + ); + // An explicit ambient root still wins: an eval run's short decoupled partition must be able + // to override a catalog it copied from. + assert_eq!( + effective_pty_root_from(cat, Some("/tmp/stev-abc123".into())), + std::path::PathBuf::from("/tmp/stev-abc123") + ); +} + +#[test] +fn debounce_never_defers_a_never_seen_task() { + let t0 = Instant::now(); + let db = LivenessDebounce::new(Duration::from_secs(10)); + // A genuinely-new target (never observed alive) is handled immediately, not deferred. + let mut plan = ReconcilePlan::default(); + plan.gc.push("hetz.brandnew.agent".into()); + let deferred = db.defer_flickers(&mut plan, t0); + assert_eq!(plan.gc, vec!["hetz.brandnew.agent".to_string()]); + assert!(deferred.is_empty()); +} + +#[test] +fn debounce_defers_a_flickering_launch_target_too() { + let t0 = Instant::now(); + let mut db = LivenessDebounce::new(Duration::from_secs(10)); + db.observe(&[sess("hetz.demo.agent", true)], t0); + + // The same recently-alive id showing up as a launch target (Absent/Dead) is also deferred — + // no noisy "already in use" re-launch of a live session. + let spec = spec_fixture(); + let mut plan = ReconcilePlan::default(); + plan.launch.push(Launch { + spec: &spec, + tasks: vec![target("hetz.demo.agent", "x")], + live_derived: Vec::new(), + }); + let deferred = db.defer_flickers(&mut plan, t0 + Duration::from_secs(2)); + assert!( + plan.launch.is_empty(), + "a recently-alive flicker must NOT be re-launched" + ); + assert_eq!(deferred, vec!["hetz.demo.agent".to_string()]); +} + +#[test] +fn codex_hook_gate_accepts_new_agents_without_mutating_the_launch_plan() { + let mut left = spec_fixture(); + left.identity = "left".into(); + left.path = PathBuf::from("/catalog/node/left/agent.kdl"); + let mut right = spec_fixture(); + right.identity = "right".into(); + right.path = PathBuf::from("/catalog/node/right/agent.kdl"); + let mut left_agent = target("node.left.agent", "exec codex --model gpt-5"); + left_agent.workspace = Some("/workspaces/shared".into()); + let mut right_agent = target("node.right.agent", "/opt/bin/codex --model gpt-5"); + right_agent.workspace = Some("/workspaces/shared".into()); + let mut plan = ReconcilePlan::default(); + plan.launch.push(Launch { + spec: &left, + tasks: vec![left_agent], + live_derived: Vec::new(), + }); + plan.launch.push(Launch { + spec: &right, + tasks: vec![right_agent], + live_derived: Vec::new(), + }); + let expected = plan + .launch + .iter() + .map(|launch| launch.spec.identity.clone()) + .collect::>(); + let mut report = UpReport::default(); + + gate_harness_launches_on_hooks(&mut plan, Path::new("/catalog"), &mut report, |_| Ok(())); + + assert_eq!( + plan.launch + .iter() + .map(|launch| launch.spec.identity.clone()) + .collect::>(), + expected, + "successful hook verification must leave the launch plan unchanged" + ); + assert_eq!(plan.launch.len(), 2); + assert!(report.errors.is_empty()); +} + +#[test] +fn codex_hook_gate_does_not_touch_adopted_agents_or_sidecar_only_repairs() { + let mut spec = spec_fixture(); + spec.identity = "root".into(); + let mut ding = target("node.root.ding", "st2 ding"); + ding.name = "ding".into(); + let mut plan = ReconcilePlan::default(); + plan.adopt.push(&spec); + plan.launch.push(Launch { + spec: &spec, + tasks: vec![ding], + live_derived: Vec::new(), + }); + let mut report = UpReport::default(); + + gate_harness_launches_on_hooks(&mut plan, Path::new("/catalog"), &mut report, |_| { + panic!("an already-live Codex agent must not enter the hook gate") + }); + + assert_eq!(plan.adopt, [&spec]); + assert_eq!(plan.launch.len(), 1); + assert_eq!(plan.launch[0].tasks[0].name, "ding"); + assert!(report.errors.is_empty()); +} + +#[test] +fn hook_verification_failure_suppresses_only_new_codex_agents() { + let mut codex = spec_fixture(); + codex.identity = "codex".into(); + codex.path = PathBuf::from("/catalog/node/codex/agent.kdl"); + let mut claude = spec_fixture(); + claude.identity = "claude".into(); + claude.path = PathBuf::from("/catalog/node/claude/agent.kdl"); + let mut codex_agent = target("node.codex.agent", "exec codex"); + codex_agent.workspace = Some("/workspaces/codex".into()); + let claude_agent = target("node.claude.agent", "exec claude"); + let mut plan = ReconcilePlan::default(); + plan.launch.push(Launch { + spec: &codex, + tasks: vec![codex_agent], + live_derived: Vec::new(), + }); + plan.launch.push(Launch { + spec: &claude, + tasks: vec![claude_agent], + live_derived: Vec::new(), + }); + let mut report = UpReport::default(); + + gate_harness_launches_on_hooks(&mut plan, Path::new("/catalog"), &mut report, |_| { + anyhow::bail!("stale receipt") + }); + + assert_eq!( + plan.launch + .iter() + .map(|launch| launch.spec.identity.as_str()) + .collect::>(), + ["claude"] + ); + assert_eq!(report.errors.len(), 1); + assert!(report.errors[0].contains("stale receipt")); + assert!(report.errors[0].contains("launch suppressed")); +} + +/// The built `pty run` argv runs the command verbatim under `sh -c`, detached, with the pinned id +/// and the established fallback presentation when no Agent Spec name is projected. +#[test] +fn build_run_command_wraps_command_in_sh_c() { + let cli = PtyCli::default(); + let t = target( + "hetz.demo.agent", + "exec claude --permission-mode bypassPermissions 'boot'", + ); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + + assert_eq!(cmd.get_program(), OsStr::new("pty")); + let args: Vec = cmd + .get_args() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + // Stable launch/config arguments precede the persisted environment and command separator. + assert_eq!(&args[0..2], &["run", "-d"]); + assert!(args.contains(&"--force".to_string())); + let id_pos = args.iter().position(|a| a == "--id").unwrap(); + assert_eq!(args[id_pos + 1], "hetz.demo.agent"); + let name_pos = args.iter().position(|a| a == "--name").unwrap(); + assert_eq!(args[name_pos + 1], "hetz.demo"); + let sep = args.iter().position(|a| a == "--").unwrap(); + assert_eq!( + &args[sep + 1..], + &[ + "sh", + "-c", + "exec claude --permission-mode bypassPermissions 'boot'" + ] + ); +} + +#[test] +fn build_run_command_projects_primary_name_and_owned_tags_at_spawn() { + let key = "ST2_TEST_PRESENTATION_LITERAL_71c"; + unsafe { std::env::set_var(key, "expanded") } + + let cli = PtyCli::default(); + let mut t = target("hetz.demo", "codex"); + t.bus_id = "hetz.demo".to_owned(); + t.tags + .insert("unrelated".to_owned(), "preserved".to_owned()); + t.presentation = Some(PtyPresentation { + pty_id: "hetz.demo".to_owned(), + display_name: Some(Some("Build owner".to_owned())), + tags: BTreeMap::from([ + ("agent.presentation.schema".to_owned(), Some("1".to_owned())), + ("agent.actor.path".to_owned(), Some("hetz.demo".to_owned())), + ( + "agent.presentation.description".to_owned(), + Some(format!("${key}")), + ), + ]), + }); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + let args = cmd + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + + let name = args.iter().position(|arg| arg == "--name").unwrap(); + assert_eq!(args[name + 1], "Build owner"); + let tags = args + .windows(2) + .filter(|pair| pair[0] == "--tag") + .map(|pair| pair[1].as_str()) + .collect::>(); + assert!(tags.contains("unrelated=preserved")); + assert!(tags.contains("agent.presentation.schema=1")); + assert!(tags.contains("agent.actor.path=hetz.demo")); + assert!(tags.contains("agent.presentation.description=$ST2_TEST_PRESENTATION_LITERAL_71c")); +} + +#[test] +fn metadata_patch_uses_exact_id_and_one_json_stdin_payload() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("pty-capture"); + std::fs::write( + &executable, + r#"#!/bin/sh +printf '%s\n' "$@" > "$0.args" +printf '' > "$0.ready.tmp" +mv "$0.ready.tmp" "$0.ready" +while [ ! -e "$0.release" ]; do sleep 0.01; done +cat > "$0.stdin" +"#, + ) + .unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + reset_fixture_barrier(&executable); + let observed_executable = executable.clone(); + let cli = PtyCli { + bin: executable.display().to_string(), + catalog_root: temporary.path().to_path_buf(), + on_command_spawn: Some(std::sync::Arc::new(move |pid| { + release_ready_fixture( + pid, + &observed_executable, + "fake PTY metadata command was not ready", + ); + })), + }; + let presentation = PtyPresentation { + pty_id: "stable.agent.id".to_owned(), + display_name: Some(None), + tags: BTreeMap::from([ + ("agent.presentation.schema".to_owned(), Some("1".to_owned())), + ("agent.presentation.description".to_owned(), None), + ]), + }; + + cli.patch_presentation(&presentation).unwrap(); + + assert_eq!( + std::fs::read_to_string(executable.with_extension("args")).unwrap(), + "metadata\npatch\n--id\nstable.agent.id\n" + ); + let payload: serde_json::Value = + serde_json::from_slice(&std::fs::read(executable.with_extension("stdin")).unwrap()) + .unwrap(); + assert_eq!(payload["displayName"], serde_json::Value::Null); + assert_eq!(payload["tags"]["agent.presentation.schema"], "1"); + assert_eq!( + payload["tags"]["agent.presentation.description"], + serde_json::Value::Null + ); +} + +#[test] +fn input_write_failure_terminates_and_reaps_the_child() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("close-stdin"); + let stdin_closed = temporary.path().join("stdin-closed"); + // The script signals only AFTER closing its stdin, so the barrier below returns exactly when + // the read end is gone and the parent's very next write must fail with EPIPE. + std::fs::write( + &executable, + "#!/bin/sh\nexec 0<&-\n: > \"$READY.tmp\"\nmv \"$READY.tmp\" \"$READY\"\nsleep 60\n", + ) + .unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + let input = vec![b'x'; 1024 * 1024]; + // The pid comes from the parent at spawn, and the barrier makes the deadline measure only + // the behaviour under test. Without it the 1s budget also had to cover fork+exec of the + // shell, so a loaded host reported `timed out after 1.0s` instead of `Broken pipe` — the + // fixture's scheduling consumed the deadline the assertion is about. + let mut spawned = None; + let error = output_with_input_timeout_observed( + Command::new(&executable).env("READY", &stdin_closed), + Duration::from_secs(1), + Some(input), + |pid| { + spawned = Some(pid); + await_fixture_ready(pid, &stdin_closed, "the child never closed its stdin"); + }, + ) + .unwrap_err(); + let pid = spawned.expect("the child was spawned before the input write failed"); + + assert!( + format!("{error:#}").contains("Broken pipe"), + "unexpected write error: {error:#}" + ); + assert!( + !crate::host_lock::process_alive(pid), + "failed metadata child {pid} was not terminated and reaped" + ); +} + +/// The process-group kill is the entire stated reason [`terminate_and_reap_before`] exists — its +/// docstring is about an escaped descendant that inherited stdout/stderr and would otherwise +/// block cleanup. Nothing constructed such a descendant, so `kill(-pid, SIGKILL)` was asserted +/// by no test: removing it alone left the suite green, because `child.kill()` already satisfies +/// every assertion that only looks at the direct child. +#[test] +fn the_group_kill_reaps_a_descendant_that_outlives_the_direct_child() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("spawn-descendant"); + let descendant_pidfile = temporary.path().join("descendant.pid"); + // The descendant inherits stdout/stderr and outlives the direct child, which is exactly the + // shape the docstring describes. `child.kill()` cannot reach it; only the group signal can. + // It publishes its pid by atomic rename, so the barrier never reads a truncated file. + std::fs::write( + &executable, + "#!/bin/sh\nsh -c 'printf \"%s\" \"$$\" > \"$DESCENDANT_PIDFILE.tmp\"; mv \"$DESCENDANT_PIDFILE.tmp\" \"$DESCENDANT_PIDFILE\"; sleep 60' &\nsleep 60\n", + ) + .unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + + // This test *requires* the child to have run — a descendant it never forked is nothing to + // reap. Waiting for the pidfile inside `on_spawn` makes that a barrier instead of a race: + // the deadline then only has to outlast a `sleep`, never a fork+exec, so a loaded host can + // no longer end the run before the fixture has built the thing under test. + let error = output_with_input_timeout_observed( + Command::new(&executable).env("DESCENDANT_PIDFILE", &descendant_pidfile), + Duration::from_millis(500), + None, + |pid| { + await_fixture_ready( + pid, + &descendant_pidfile, + "the child never forked a descendant, so this case would test nothing", + ) + }, + ) + .unwrap_err(); + assert!( + format!("{error:#}").contains("timed out"), + "unexpected error: {error:#}" + ); + + let descendant = std::fs::read_to_string(&descendant_pidfile) + .expect("the readiness barrier returned without a pidfile") + .parse::() + .unwrap(); + + // Generous on purpose: the descendant is orphaned by the same group kill, so its exit is + // observable only once the reparenting init reaps it. That latency is not the behaviour + // under test, and waiting longer costs nothing when the kill did reach it. + let deadline = Instant::now() + Duration::from_secs(5); + while process_can_retain_cleanup_resources(descendant) && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + let survived = process_can_retain_cleanup_resources(descendant); + if survived { + // Do not leak a 60s sleeper into the test host when the assertion is about to fail. + unsafe { libc::kill(descendant, libc::SIGKILL) }; + } + assert!( + !survived, + "escaped descendant {descendant} survived cleanup: the process-group kill did not reach it" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn a_zombie_cannot_retain_cleanup_resources() { + let mut child = Command::new("sh").arg("-c").arg("exit 0").spawn().unwrap(); + let pid = child.id() as i32; + let deadline = Instant::now() + Duration::from_secs(1); + let mut state = None; + while Instant::now() < deadline { + state = linux_process_state(pid); + if state == Some('Z') { + break; + } + std::thread::sleep(Duration::from_millis(10)); + } + let kill_probe_considered_alive = crate::host_lock::process_alive(pid); + let retained_cleanup_resources = process_can_retain_cleanup_resources(pid); + let _ = child.wait(); + + assert_eq!(state, Some('Z'), "child did not become a zombie"); + assert!( + kill_probe_considered_alive, + "the fixture must expose kill(pid, 0) treating a zombie as alive" + ); + assert!( + !retained_cleanup_resources, + "a terminated zombie cannot retain cleanup resources" + ); +} + +#[test] +fn input_write_obeys_the_child_deadline() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("ignore-stdin"); + std::fs::write(&executable, "#!/bin/sh\nsleep 60\n").unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + let input = vec![b'x'; 1024 * 1024]; + // The pid comes from the parent at spawn, not from the child, and this case cannot use a + // readiness barrier: it is precisely the one where the child may never be scheduled. The + // write blocks as soon as the pipe buffer fills, which needs no execution by the child at + // all, and the deadline then terminates the whole group. Anything the child was supposed to + // record would never be written, so a test that waits for it fails on exactly the condition + // it exists to cover. The observed spawn instant is therefore also the clock: timing from + // before the call would charge fork+exec to the 1s budget this assertion polices. + let mut spawned = None; + let error = output_with_input_timeout_observed( + &mut Command::new(&executable), + Duration::from_millis(100), + Some(input), + |pid| spawned = Some((pid, Instant::now())), + ) + .unwrap_err(); + let (pid, started) = + spawned.expect("the child was spawned before the input deadline expired"); + + assert!( + format!("{error:#}").contains("timed out"), + "unexpected write error: {error:#}" + ); + assert!( + started.elapsed() < Duration::from_secs(1), + "blocked stdin write ignored the child deadline" + ); + let reap_deadline = Instant::now() + Duration::from_secs(1); + while crate::host_lock::process_alive(pid) && Instant::now() < reap_deadline { + std::thread::sleep(Duration::from_millis(10)); + } + assert!(!crate::host_lock::process_alive(pid)); +} + +/// The two lifecycle tests above block in `on_spawn` until their fixture reached the state under +/// test, which only keeps them load-insensitive because [`run_captured`] starts the child +/// deadline AFTER `on_spawn` returns. Nothing else proves that order: reversing it leaves every +/// other test green on an idle host and silently puts both back on a race with the scheduler. +#[test] +fn the_spawn_observer_runs_before_the_child_deadline_starts() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("close-stdin"); + let stdin_closed = temporary.path().join("stdin-closed"); + std::fs::write( + &executable, + "#!/bin/sh\nexec 0<&-\n: > \"$READY.tmp\"\nmv \"$READY.tmp\" \"$READY\"\nsleep 60\n", + ) + .unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + let timeout = Duration::from_millis(200); + + // The barrier deliberately outlasts `timeout`, so the outcome depends on the order alone and + // on nothing the host's scheduler does. Deadline after `on_spawn`: the write meets a closed + // read end and fails with EPIPE at once. Deadline before `on_spawn`: it has already expired + // when the barrier returns, so the write never runs and the call reports a timeout instead. + let error = output_with_input_timeout_observed( + Command::new(&executable).env("READY", &stdin_closed), + timeout, + Some(vec![b'x'; 1024]), + |pid| { + await_fixture_ready(pid, &stdin_closed, "the child never closed its stdin"); + std::thread::sleep(timeout * 2); + }, + ) + .unwrap_err(); + + assert!( + format!("{error:#}").contains("Broken pipe"), + "the child deadline started before `on_spawn` returned: {error:#}" + ); +} + +#[test] +fn bounded_capture_keeps_the_tail_of_an_oversized_stream() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("flood"); + // Start marker, 1 MiB of filler (4x the cap, so both streams truncate), end marker. + std::fs::write( + &executable, + "#!/bin/sh\nprintf START; head -c 1048576 /dev/zero; printf END\n", + ) + .unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let output = + output_with_timeout(&mut Command::new(&executable), Duration::from_secs(5)).unwrap(); + + assert_eq!(output.stdout.len(), CAPTURE_CAP_BYTES); + assert!( + output.stdout.ends_with(b"END"), + "capped stdout lost the tail" + ); + assert!( + !output.stdout.starts_with(b"START"), + "capped stdout kept the head instead of the tail" + ); + // stderr is empty here, so only the stdout read-back may have been capped. +} + +#[test] +fn full_stdout_variant_returns_complete_output_larger_than_the_cap() { + use std::os::unix::fs::PermissionsExt as _; + + let temporary = tempfile::tempdir().unwrap(); + let executable = temporary.path().join("flood"); + std::fs::write( + &executable, + "#!/bin/sh\nprintf START; head -c 1048576 /dev/zero; printf END\n", + ) + .unwrap(); + std::fs::set_permissions(&executable, std::fs::Permissions::from_mode(0o755)).unwrap(); + + let output = + output_full_stdout_with_timeout(&mut Command::new(&executable), Duration::from_secs(5)) + .unwrap(); + + assert!(output.stdout.len() > CAPTURE_CAP_BYTES); + assert!( + output.stdout.starts_with(b"START") && output.stdout.ends_with(b"END"), + "full-stdout variant truncated structured output: {} bytes", + output.stdout.len() + ); +} + +/// Proves the shared reaper actually waits: the killed child is observed as a zombie BEFORE +/// `reap_detached` runs, so only the reaper's `wait()` can clear that state. +#[cfg(target_os = "linux")] +#[test] +fn the_shared_reaper_reaps_a_killed_child() { + let mut child = Command::new("sh") + .arg("-c") + .arg("sleep 60") + .spawn() + .unwrap(); + let pid = child.id() as i32; + unsafe { + libc::kill(pid, libc::SIGKILL); + } + let _ = child.kill(); + let deadline = Instant::now() + Duration::from_secs(1); + while linux_process_state(pid) != Some('Z') && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + assert_eq!( + linux_process_state(pid), + Some('Z'), + "fixture did not produce a zombie" + ); + + reap_detached(child); + let deadline = Instant::now() + Duration::from_secs(2); + while linux_process_state(pid) == Some('Z') && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(10)); + } + assert_ne!( + linux_process_state(pid), + Some('Z'), + "the shared reaper did not reap the killed child {pid}" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn undrained_reader_does_not_retain_the_nonblocking_writer() { + use std::os::fd::{FromRawFd as _, OwnedFd}; + + let mut pipe_fds = [0; 2]; + assert_eq!( + unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC) }, + 0 + ); + let reader = unsafe { OwnedFd::from_raw_fd(pipe_fds[0]) }; + let writer = unsafe { OwnedFd::from_raw_fd(pipe_fds[1]) }; + let pipe = std::fs::read_link(format!("/proc/self/fd/{}", reader.as_raw_fd())).unwrap(); + let started = Instant::now(); + + assert!( + !write_all_before( + ChildStdin::from(writer), + &vec![b'x'; 1024 * 1024], + Instant::now() + Duration::from_millis(100), + ) + .unwrap() + ); + let retained_writers = std::fs::read_dir("/proc/self/fd") + .unwrap() + .filter_map(Result::ok) + .filter_map(|entry| std::fs::read_link(entry.path()).ok()) + .filter(|target| target == &pipe) + .count(); + + assert!(started.elapsed() < Duration::from_secs(1)); + assert_eq!( + retained_writers, 1, + "the undrained pipe retained a writer after the deadline" + ); +} + +#[cfg(target_os = "linux")] +#[test] +fn expired_write_deadline_prevents_further_progress() { + use std::os::fd::{FromRawFd as _, OwnedFd}; + + let mut pipe_fds = [0; 2]; + assert_eq!( + unsafe { libc::pipe2(pipe_fds.as_mut_ptr(), libc::O_CLOEXEC) }, + 0 + ); + let _reader = unsafe { OwnedFd::from_raw_fd(pipe_fds[0]) }; + let writer = unsafe { OwnedFd::from_raw_fd(pipe_fds[1]) }; + + assert!( + !write_all_before(ChildStdin::from(writer), b"x", Instant::now()).unwrap(), + "an expired child deadline still allowed stdin progress" + ); +} + +#[test] +fn expired_cleanup_deadline_hands_reaping_off_without_blocking() { + let child = Command::new("sleep").arg("60").spawn().unwrap(); + let pid = child.id() as i32; + let started = Instant::now(); + terminate_and_reap_before(child, pid, Instant::now()); + + assert!( + started.elapsed() < Duration::from_secs(1), + "expired cleanup deadline blocked the caller" + ); + let reap_deadline = Instant::now() + Duration::from_secs(1); + while crate::host_lock::process_alive(pid) && Instant::now() < reap_deadline { + std::thread::sleep(Duration::from_millis(10)); + } + assert!( + !crate::host_lock::process_alive(pid), + "background reaper did not collect child {pid}" + ); +} + +#[test] +fn build_run_command_passes_direct_argv_without_a_shell() { + let cli = PtyCli::new(PathBuf::from("/my/catalog")); + let mut t = target("hetz.demo.agent", "unused"); + t.launch = TaskLaunch::Argv(vec![ + "axe".into(), + "agent".into(), + "exec".into(), + "--".into(), + "claude".into(), + "--resume".into(), + "$CATALOG/session id".into(), + ]); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + let args = cmd + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + let sep = args.iter().position(|arg| arg == "--").unwrap(); + + assert_eq!( + &args[sep + 1..], + [ + "axe", + "agent", + "exec", + "--", + "claude", + "--resume", + "/my/catalog/session id" + ] + ); + assert!(!args[sep + 1..].iter().any(|arg| arg == "sh")); +} + +#[test] +fn build_run_command_expands_direct_argv_with_the_managed_agent_environment() { + let cli = PtyCli::new(PathBuf::from("/eval/catalog")); + let mut t = target("local.worker", "unused"); + t.env.insert("ST_AGENT".into(), "local.worker".into()); + t.env.insert("ST_ROOT".into(), "/eval/catalog".into()); + t.launch = TaskLaunch::Argv(vec![ + "claude".into(), + "$ST_AGENT reads $ST_ROOT and $CATALOG".into(), + ]); + + let cmd = cli.build_run_command(&t, Path::new("/eval/catalog/local/worker")); + let args = cmd + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + let separator = args.iter().position(|arg| arg == "--").unwrap(); + + assert_eq!( + &args[separator + 1..], + [ + "claude", + "local.worker reads /eval/catalog and /eval/catalog" + ] + ); +} + +#[test] +fn build_run_command_persists_the_complete_managed_environment_before_the_command() { + let cli = PtyCli::new(PathBuf::from("/my/catalog")); + let mut t = target("hetz.demo.agent", "exec codex 'boot'"); + t.env.insert("CUSTOM".into(), "task-value".into()); + t.env.insert("ST_AGENT".into(), "hetz.demo".into()); + t.env.insert("ST_ROOT".into(), "$CATALOG/custom-bus".into()); + t.env.insert("TERM".into(), "screen-256color".into()); + t.env + .insert("PTY_ROOT".into(), "/declared/root/must-not-win".into()); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + + let args: Vec = cmd + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + let separator = args.iter().position(|arg| arg == "--").unwrap(); + let mut persisted = BTreeMap::new(); + let mut index = 0; + while index < separator { + if args[index] == "--env" { + let (key, value) = args[index + 1].split_once('=').unwrap(); + assert!( + persisted + .insert(key.to_string(), value.to_string()) + .is_none(), + "the final managed overlay needs only one persisted value per key" + ); + index += 2; + } else { + index += 1; + } + } + let inherited = cmd + .get_envs() + .filter_map(|(key, value)| { + value.map(|value| { + ( + key.to_string_lossy().into_owned(), + value.to_string_lossy().into_owned(), + ) + }) + }) + .collect::>(); + assert_eq!( + persisted, inherited, + "initial process env and restart-persisted env must be the same resolved overlay" + ); + assert_eq!( + persisted.get("CATALOG").map(String::as_str), + Some("/my/catalog") + ); + assert_eq!( + persisted.get("ST_ROOT").map(String::as_str), + Some("/my/catalog/custom-bus") + ); + assert_eq!( + persisted.get("PTY_ROOT").map(String::as_str), + Some( + effective_pty_root(&cli.catalog_root) + .to_string_lossy() + .as_ref() + ) + ); + assert_eq!( + persisted.get("TERM").map(String::as_str), + Some("screen-256color") + ); + assert_eq!( + persisted.get("ST_AGENT").map(String::as_str), + Some("hetz.demo") + ); + assert_eq!( + persisted.get("CUSTOM").map(String::as_str), + Some("task-value") + ); + assert!(persisted.contains_key("ST_HOOKS")); +} + +#[test] +fn build_run_command_omits_an_alias_equal_to_the_lifecycle_id() { + let cli = PtyCli::default(); + let mut t = target("hetz.demo", "exec codex 'boot'"); + t.bus_id = t.pty_id.clone(); + t.presentation = Some(PtyPresentation { + pty_id: t.pty_id.clone(), + display_name: Some(Some(t.pty_id.clone())), + tags: BTreeMap::new(), + }); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + let args: Vec = cmd + .get_args() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + + assert_eq!( + args.iter() + .position(|arg| arg == "--id") + .map(|position| args[position + 1].as_str()), + Some("hetz.demo") + ); + assert!( + !args.iter().any(|arg| arg == "--name"), + "pty rejects a display name equal to the stable session id" + ); + assert!( + args.iter().any(|arg| arg == "--no-display-name"), + "without this flag pty would create an unrelated automatic alias" + ); +} + +#[test] +fn build_run_command_defaults_cwd_to_spec_dir_and_passes_tags_and_env() { + let cli = PtyCli::default(); + let mut t = target("hetz.demo.agent", "exec claude 'boot'"); + t.tags.insert("role".into(), "agent".into()); + t.env.insert("ST_AGENT".into(), "hetz.demo-claude".into()); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + + let args: Vec = cmd + .get_args() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + let cwd_pos = args.iter().position(|a| a == "--cwd").unwrap(); + assert_eq!(args[cwd_pos + 1], "/cat/hetz/demo"); // no cwd, no workspace → spec dir + let tag_pos = args.iter().position(|a| a == "--tag").unwrap(); + assert_eq!(args[tag_pos + 1], "role=agent"); + + // env injected onto the child process + let envs: BTreeMap> = cmd + .get_envs() + .map(|(k, v)| { + ( + k.to_string_lossy().into_owned(), + v.map(|v| v.to_string_lossy().into_owned()), + ) + }) + .collect(); + assert_eq!( + envs.get("ST_AGENT"), + Some(&Some("hetz.demo-claude".to_string())) + ); + assert_eq!( + envs.get("TERM"), + Some(&Some("xterm-256color".to_string())), + "headless st2 launches must not pass TERM=dumb into an interactive harness" + ); + assert!( + envs.get("ST_HOOKS") + .and_then(Option::as_deref) + .is_some_and(|path| !path.contains("/sets/sha256-")), + "managed tasks keep ST_HOOKS at the receipt-bearing root; only rendered hook commands use a versioned set" + ); +} + +#[test] +fn managed_agent_scrubs_ambient_no_color_unless_explicitly_declared() { + let cli = PtyCli::default(); + let agent = target("hetz.demo.agent", "exec claude 'boot'"); + let command = cli.build_run_command(&agent, Path::new("/cat/hetz/demo")); + assert_eq!( + command + .get_envs() + .find(|(key, _)| *key == OsStr::new("NO_COLOR")) + .map(|(_, value)| value), + Some(None), + "ambient NO_COLOR must not silently disable an interactive agent's color" + ); + let args = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + assert!( + args.windows(2) + .any(|pair| pair == ["--unset-env", "NO_COLOR"]), + "the removal must be persisted for PTY restart" + ); + + let mut explicit = target("hetz.explicit.agent", "exec claude 'boot'"); + explicit.env.insert("NO_COLOR".into(), "1".into()); + let command = cli.build_run_command(&explicit, Path::new("/cat/hetz/explicit")); + assert_eq!( + command + .get_envs() + .find(|(key, _)| *key == OsStr::new("NO_COLOR")) + .and_then(|(_, value)| value), + Some(OsStr::new("1")), + "an explicit Agent Spec NO_COLOR remains authoritative" + ); + let args = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + assert!( + !args + .windows(2) + .any(|pair| pair == ["--unset-env", "NO_COLOR"]), + "an explicit assignment must not also persist a removal" + ); +} + +#[test] +fn non_agent_task_does_not_claim_no_color_policy() { + let cli = PtyCli::default(); + let mut task = target("hetz.demo.sidecar", "exec sleep 1"); + task.name = "sidecar".into(); + let command = cli.build_run_command(&task, Path::new("/cat/hetz/demo")); + + assert!( + command + .get_envs() + .all(|(key, _)| key != OsStr::new("NO_COLOR")), + "non-agent services keep the caller's ambient NO_COLOR semantics" + ); + let args = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); + assert!( + !args + .windows(2) + .any(|pair| pair == ["--unset-env", "NO_COLOR"]), + "non-agent services must not persist an st2-owned removal" + ); +} + +#[test] +fn isolation_wrapper_preserves_environment_removals() { + let mut inner = Command::new("pty"); + inner.env("TERM", "xterm-256color").env_remove("NO_COLOR"); + let mut outer = Command::new("systemd-run"); + + apply_command_env(&inner, &mut outer); + + let env = outer + .get_envs() + .map(|(key, value)| (key.to_os_string(), value.map(OsStr::to_os_string))) + .collect::>(); + assert_eq!( + env.get(OsStr::new("TERM")).and_then(Option::as_deref), + Some(OsStr::new("xterm-256color")) + ); + assert_eq!(env.get(OsStr::new("NO_COLOR")), Some(&None)); +} + +#[test] +fn build_run_command_allows_a_task_to_override_the_default_term() { + let cli = PtyCli::default(); + let mut t = target("hetz.demo.agent", "exec codex 'boot'"); + t.env.insert("TERM".into(), "screen-256color".into()); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + let term = cmd + .get_envs() + .find(|(key, _)| *key == OsStr::new("TERM")) + .and_then(|(_, value)| value) + .map(|value| value.to_string_lossy().into_owned()); + assert_eq!(term.as_deref(), Some("screen-256color")); +} + +#[test] +fn build_run_command_defaults_cwd_to_workspace_when_task_cwd_absent() { + let cli = PtyCli::default(); + let mut t = target("hetz.demo.agent", "exec claude 'boot'"); + t.workspace = Some("/repos/demo".into()); // no task cwd → workspace (spec.md §2) + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + let args: Vec = cmd + .get_args() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + let cwd_pos = args.iter().position(|a| a == "--cwd").unwrap(); + assert_eq!(args[cwd_pos + 1], "/repos/demo"); +} + +#[test] +fn build_run_command_expands_catalog_var_and_sets_it_in_env() { + let cli = PtyCli::new(PathBuf::from("/my/catalog")); + let mut t = target("hetz.demo.agent", "run"); + t.env.insert("DATA".into(), "$CATALOG/evals/x".into()); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + let envs: BTreeMap> = cmd + .get_envs() + .map(|(k, v)| { + ( + k.to_string_lossy().into_owned(), + v.map(|v| v.to_string_lossy().into_owned()), + ) + }) + .collect(); + assert_eq!( + envs.get("DATA"), + Some(&Some("/my/catalog/evals/x".to_string())) + ); + assert_eq!(envs.get("CATALOG"), Some(&Some("/my/catalog".to_string()))); +} + +#[test] +fn build_run_command_expands_vars_in_env_cwd_and_tags_but_not_command() { + // Unique var name so the process-global set_var can't collide with a parallel test. + let key = "ST2_TEST_EXPAND_NET_9f3"; + unsafe { std::env::set_var(key, "/net/xyz") } + + let cli = PtyCli::default(); + let mut t = target("hetz.demo.agent", "exec claude $ST2_TEST_EXPAND_NET_9f3/go"); + t.cwd = Some(format!("${key}/work")); + t.tags.insert("net".into(), format!("${key}")); + t.env.insert("ST_ROOT".into(), format!("${key}/custom-bus")); + let cmd = cli.build_run_command(&t, Path::new("/cat/hetz/demo")); + + let args: Vec = cmd + .get_args() + .map(|a| a.to_string_lossy().into_owned()) + .collect(); + // cwd expanded (absolute → replaces the spec dir) + let cwd_pos = args.iter().position(|a| a == "--cwd").unwrap(); + assert_eq!(args[cwd_pos + 1], "/net/xyz/work"); + // tag value expanded + let tag_pos = args.iter().position(|a| a == "--tag").unwrap(); + assert_eq!(args[tag_pos + 1], "net=/net/xyz"); + // command left verbatim for sh -c to expand at spawn + assert_eq!( + args.last().unwrap(), + "exec claude $ST2_TEST_EXPAND_NET_9f3/go" + ); + + // env value expanded + let envs: std::collections::BTreeMap> = cmd + .get_envs() + .map(|(k, v)| { + ( + k.to_string_lossy().into_owned(), + v.map(|v| v.to_string_lossy().into_owned()), + ) + }) + .collect(); + assert_eq!( + envs.get("ST_ROOT"), + Some(&Some("/net/xyz/custom-bus".to_string())) + ); + + unsafe { std::env::remove_var(key) } +} + +#[test] +fn resolve_cwd_honors_relative_absolute_workspace_and_default() { + let cli = PtyCli::default(); + let mut t = target("x", "y"); + // relative cwd → joined onto the spec dir + t.cwd = Some("sub".into()); + assert_eq!( + cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), + Path::new("/cat/hetz/demo/sub") + ); + // absolute cwd → replaces + t.cwd = Some("/repos/fabric".into()); + assert_eq!( + cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), + Path::new("/repos/fabric") + ); + // no cwd but a workspace → workspace + t.cwd = None; + t.workspace = Some("/repos/ws".into()); + assert_eq!( + cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), + Path::new("/repos/ws") + ); + // neither → spec dir + t.workspace = None; + assert_eq!( + cli.resolve_cwd(&t, Path::new("/cat/hetz/demo")), + Path::new("/cat/hetz/demo") + ); +} + +#[test] +fn detect_host_returns_a_nonempty_short_name() { + let h = detect_host(); + assert!(!h.is_empty()); + assert!(!h.contains('.'), "short name only, got {h}"); +} + +#[test] +fn task_observation_of_missing_pty_root_is_complete_and_does_not_create_it() { + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + std::fs::create_dir(&catalog).unwrap(); + std::fs::write( + catalog.join("catalog.kdl"), + format!( + "catalog {{ pty-root {:?} }}\n", + tmp.path().join("missing-pty").display().to_string() + ), + ) + .unwrap(); + let root = effective_pty_root_from(&catalog, None); + assert!(!root.exists()); + let batch = PtyCli::new(catalog).task_observations(&HashSet::from(["h.worker"])); + assert!(batch.complete, "{:?}", batch.errors); + assert!(batch.observations.is_empty()); + assert!(!root.exists(), "read-only observation created the PTY root"); +} + +#[test] +fn unreadable_pty_root_evidence_is_indeterminate_not_absent() { + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + let loop_path = tmp.path().join("pty-loop"); + std::fs::create_dir(&catalog).unwrap(); + std::os::unix::fs::symlink(&loop_path, &loop_path).unwrap(); + std::fs::write( + catalog.join("catalog.kdl"), + format!( + "catalog {{ pty-root {:?} }}\n", + loop_path.display().to_string() + ), + ) + .unwrap(); + let batch = PtyCli::new(catalog) + .task_observations_at_root(&HashSet::from(["h.worker"]), &loop_path); + assert!(!batch.complete); + assert!(batch.observations.is_empty()); + assert!( + batch.errors[0].contains("cannot inspect PTY root"), + "{:?}", + batch.errors + ); +} + +#[test] +fn removed_and_recreated_pty_root_is_indeterminate_not_absent() { + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path().join("pty"); + std::fs::create_dir(&root).unwrap(); + let fake = tmp.path().join("pty-bin"); + std::fs::write( + &fake, + r#"#!/bin/sh +rmdir "$PTY_ROOT" +mkdir "$PTY_ROOT" +printf '%s\n' '[]' +printf '' > "$0.ready.tmp" +mv "$0.ready.tmp" "$0.ready" +while [ ! -e "$0.release" ]; do sleep 0.01; done +"#, + ) + .unwrap(); + let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); + permissions.set_mode(0o755); + std::fs::set_permissions(&fake, permissions).unwrap(); + + reset_fixture_barrier(&fake); + let observed_fake = fake.clone(); + let batch = PtyCli { + bin: fake.display().to_string(), + catalog_root: tmp.path().join("catalog"), + on_command_spawn: Some(std::sync::Arc::new(move |pid| { + release_ready_fixture( + pid, + &observed_fake, + "fake PTY inventory was not published", + ); + })), + } + .task_observations_at_root(&HashSet::from(["h.worker"]), &root); + assert!(!batch.complete); + assert!(batch.observations.is_empty()); + assert!( + batch.errors[0].contains("changed identity during observation"), + "{:?}", + batch.errors + ); +} + +#[test] +fn pty_stats_rejects_pid_reuse_between_registry_snapshot_and_start_token_capture() { + let initial = PtyListEntry { + name: "h.worker".into(), + status: "running".into(), + exit_code: None, + pid: Some(41), + created_at: Some("2026-09-05T10:00:00.000Z".into()), + display_name: None, + tags: BTreeMap::new(), + }; + let stats = |alive: bool, created_at: &str| PtyStatsEntry { + name: "h.worker".into(), + process: Some(PtyStatsProcess { alive }), + daemon: Some(PtyStatsDaemon { pid: 41 }), + created_at: Some(created_at.into()), + }; + + // PID 41 has been reused. A token captured after the registry snapshot + // would describe the replacement, but its live socket reports a new + // creation generation and prevents that token from being admitted. + let replacement = stats(true, "2026-09-05T10:00:01.000Z"); + assert_eq!( + confirm_pty_generation(&initial, &[replacement]), + Err(ResourceTargetUnavailableReason::GenerationChanged) + ); + + let exited = stats(false, "2026-09-05T10:00:00.000Z"); + assert_eq!( + confirm_pty_generation(&initial, &[exited]), + Err(ResourceTargetUnavailableReason::ProcessUnavailable) + ); + let stable = stats(true, "2026-09-05T10:00:00.000Z"); + assert_eq!(confirm_pty_generation(&initial, &[stable]), Ok(())); +} + +#[test] +fn pty_resource_observation_uses_one_stats_snapshot_and_preserves_generation_id() { + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path().join("pty"); + std::fs::create_dir(&root).unwrap(); + let invocations = tmp.path().join("invocations"); + let failed_once = tmp.path().join("stats-failed-once"); + let pid = std::process::id(); + let fake = tmp.path().join("pty-bin"); + std::fs::write( + &fake, + format!( + r#"#!/bin/sh +printf '%s\n' "$*" >> {invocations:?} +case "$1" in + list) + printf '%s\n' '[{{"name":"h.a","status":"running","pid":{pid},"createdAt":"2026-09-05T10:00:00.000Z"}},{{"name":"h.b","status":"running","pid":{pid},"createdAt":"2026-09-05T10:00:01.000Z"}}]' + ;; + stats) + if [ ! -e {failed_once:?} ]; then + : > {failed_once:?} + exit 1 + fi + printf '%s\n' '[{{"name":"h.a","process":{{"alive":true}},"daemon":{{"pid":{pid}}},"createdAt":"2026-09-05T10:00:00.000Z"}},{{"name":"h.b","process":{{"alive":true}},"daemon":{{"pid":{pid}}},"createdAt":"2026-09-05T10:00:01.000Z"}}]' + ;; +esac +"#, + invocations = invocations, + failed_once = failed_once, + ), + ) + .unwrap(); + let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); + permissions.set_mode(0o755); + std::fs::set_permissions(&fake, permissions).unwrap(); + + let cli = PtyCli { + bin: fake.display().to_string(), + catalog_root: tmp.path().join("catalog"), + on_command_spawn: None, + }; + let desired_ids = HashSet::from(["h.a", "h.b"]); + let unavailable = cli.task_observations_at_root(&desired_ids, &root); + let available = cli.task_observations_at_root(&desired_ids, &root); + assert!(unavailable.complete, "{:?}", unavailable.errors); + assert!(available.complete, "{:?}", available.errors); + assert_eq!(available.observations.len(), 2); + assert_eq!( + std::fs::read_to_string(invocations).unwrap(), + "list --json\nstats --json\nlist --json\nstats --json\n" + ); + for (before, after) in unavailable + .observations + .iter() + .zip(&available.observations) + { + let (ObservedState::Running(before), ObservedState::Running(after)) = + (&before.state, &after.state) + else { + panic!("live PTY lost generation"); + }; + assert!(matches!( + before.resource_target(), + ResourceTarget::Unavailable { .. } + )); + assert!(!matches!( + after.resource_target(), + ResourceTarget::Unavailable { .. } + )); + assert_eq!( + before.generation_id(), + after.generation_id(), + "transient target proof changed stable PTY generation identity" + ); + } +} + +#[test] +fn pty_task_observation_preserves_exact_generation_and_closed_states() { + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + let pty_root = tmp.path().join("pty"); + std::fs::create_dir_all(&catalog).unwrap(); + std::fs::create_dir(&pty_root).unwrap(); + std::fs::write( + catalog.join("catalog.kdl"), + format!( + "catalog {{ pty-root {:?} }}\n", + pty_root.display().to_string() + ), + ) + .unwrap(); + let fake = tmp.path().join("pty-bin"); + std::fs::write( + &fake, + r#"#!/bin/sh +printf '%s\n' '[{"name":"h.live","status":"running","pid":41,"createdAt":"2026-07-31T10:00:00.000Z","displayName":"Build owner","tags":{"agent.presentation.schema":"1","unrelated":"preserved"}},{"name":"h.exit","status":"exited","exitCode":0,"pid":42,"createdAt":"2026-07-31T09:00:00.000Z"},{"name":"h.gone","status":"vanished","pid":43,"createdAt":"2026-07-31T08:00:00.000Z"}]' +printf '' > "$0.ready.tmp" +mv "$0.ready.tmp" "$0.ready" +while [ ! -e "$0.release" ]; do sleep 0.01; done +"#, + ) + .unwrap(); + let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); + permissions.set_mode(0o755); + std::fs::set_permissions(&fake, permissions).unwrap(); + + let observed_fake = fake.clone(); + let cli = PtyCli { + bin: fake.display().to_string(), + catalog_root: catalog, + on_command_spawn: Some(std::sync::Arc::new(move |pid| { + release_ready_fixture( + pid, + &observed_fake, + "fake PTY inventory was not published", + ); + })), + }; + let desired = HashSet::from(["h.live", "h.exit", "h.gone"]); + reset_fixture_barrier(&fake); + let first = cli.task_observations(&desired); + reset_fixture_barrier(&fake); + let second = cli.task_observations(&desired); + assert!(first.complete, "{:?}", first.errors); + assert_eq!(first, second, "same PTY evidence changed generation"); + let ObservedState::Running(generation) = &first.observations[0].state else { + panic!("running PTY lost generation: {:?}", first.observations[0]); + }; + assert_eq!(generation.pid(), 41); + assert_eq!(generation.created_at(), "2026-07-31T10:00:00.000Z"); + assert!(generation.generation_id().starts_with("sha256:")); + assert_eq!(first.observations[1].state, ObservedState::Exited); + assert_eq!(first.observations[2].state, ObservedState::Vanished); + + reset_fixture_barrier(&fake); + let sessions = cli.list_sessions().unwrap(); + let presentation = sessions[0].presentation.as_ref().unwrap(); + assert_eq!(presentation.display_name.as_deref(), Some("Build owner")); + assert_eq!( + presentation + .tags + .get("agent.presentation.schema") + .map(String::as_str), + Some("1") + ); + assert_eq!( + presentation.tags.get("unrelated").map(String::as_str), + Some("preserved") + ); +} + +#[test] +fn running_pty_without_complete_generation_is_indeterminate() { + use std::os::unix::fs::PermissionsExt; + + let tmp = tempfile::tempdir().unwrap(); + let catalog = tmp.path().join("catalog"); + let pty_root = tmp.path().join("pty"); + std::fs::create_dir_all(&catalog).unwrap(); + std::fs::create_dir(&pty_root).unwrap(); + std::fs::write( + catalog.join("catalog.kdl"), + format!( + "catalog {{ pty-root {:?} }}\n", + pty_root.display().to_string() + ), + ) + .unwrap(); + let fake = tmp.path().join("pty-bin"); + std::fs::write( + &fake, + r#"#!/bin/sh +printf '%s\n' '[{"name":"h.live","status":"running"}]' +printf '' > "$0.ready.tmp" +mv "$0.ready.tmp" "$0.ready" +while [ ! -e "$0.release" ]; do sleep 0.01; done +"#, + ) + .unwrap(); + let mut permissions = std::fs::metadata(&fake).unwrap().permissions(); + permissions.set_mode(0o755); + std::fs::set_permissions(&fake, permissions).unwrap(); + reset_fixture_barrier(&fake); + let observed_fake = fake.clone(); + let batch = PtyCli { + bin: fake.display().to_string(), + catalog_root: catalog, + on_command_spawn: Some(std::sync::Arc::new(move |pid| { + release_ready_fixture( + pid, + &observed_fake, + "fake PTY inventory was not published", + ); + })), + } + .task_observations(&HashSet::from(["h.live"])); + assert!(!batch.complete); + assert!(matches!( + batch.observations[0].state, + ObservedState::Indeterminate(_) + )); +}