diff --git a/INVARIANTS.md b/INVARIANTS.md index bc5c6fe7..77401bac 100644 --- a/INVARIANTS.md +++ b/INVARIANTS.md @@ -13,6 +13,7 @@ materialization, messaging, DING, or presence must preserve them. | **Bounded restart diagnostics** | Relaunching an exec task preserves the just-finished log as one prior generation while bounding retained diagnostics to current plus prior. Final retirement removes the PID and both logs. | `tests/exec_backend.rs::exec_restart_reap_keeps_bounded_diagnostics_and_final_remove_cleans_them`; `tests/run.rs::up_once_finally_removes_dead_retired_tasks_without_restarting_them` | | **Derived companion lifecycle** | A generated DING or launched stream starts only with an eligible canonical agent, is suppressed or stopped while that target is held, suspended, unavailable, retired, or terminally parked, and remains coupled without changing explicit sibling task behavior. A stream lowers its authored adapter launch directly, parks and surfaces independently, and neither makes an otherwise empty agent runnable nor claims a delivery transport. Stream authoring is serialized, authority-scoped, source-preserving, and fail-closed for Nix ownership and invalid declarations. Healthy compact startup launches companions in one pass, and targeted reconciliation holds a missing generated companion rather than broadening to its agent. | `crates/agent-spec/tests/discovery.rs::streams_are_typed_and_only_launched_streams_lower_to_derived_exec_tasks`; `crates/agent-spec/tests/discovery.rs::stream_names_launches_and_task_collisions_fail_closed`; `src/agent_author.rs::stream_add_supports_external_command_and_argv_and_external_remove_is_idempotent`; `src/agent_author.rs::stream_authoring_enforces_authority_nix_ownership_and_canonical_validation`; `tests/run.rs::fresh_compact_agent_launches_with_its_derived_ding`; `tests/run.rs::held_adopt_only_compact_agent_stops_its_live_derived_ding`; `tests/run.rs::parked_compact_agent_stops_its_live_derived_ding`; `tests/run.rs::retired_compact_agent_stops_agent_and_derived_ding`; `tests/run.rs::selected_missing_derived_ding_is_held_without_broadening_to_its_agent`; `tests/run.rs::fresh_compact_agent_launches_with_its_derived_stream`; `tests/run.rs::retired_compact_agent_stops_agent_and_derived_stream`; `tests/run.rs::suspended_compact_agent_stops_its_derived_stream_without_touching_a_sibling`; `tests/run.rs::held_adopt_only_compact_agent_stops_its_live_derived_stream`; `tests/run.rs::a_crash_looping_stream_parks_and_surfaces_without_disturbing_its_agent`; `tests/run.rs::parked_compact_agent_stops_its_live_derived_stream`; `tests/run.rs::selected_missing_derived_stream_is_held_without_broadening_to_its_agent`; `tests/run.rs::a_launched_stream_alone_is_rejected_before_reconciliation`; `tests/run.rs::a_stream_does_not_claim_a_delivery_transport` | | **Exactly-once-safe native bus** | Messages use stable `-.md` files. An archive filename is a durable receipt that shadows and cleans restored inbox replicas and makes repeated archive cleanup idempotent. | `src/message.rs::filename_grammar`; `src/message.rs::archive_receipt_suppresses_and_idempotently_cleans_a_restored_inbox_copy`; `tests/message.rs` | +| **Evidence-graded delivery ledger** | Native delivery keeps one host-local `st2.delivery-ledger.v1` record per harness, keyed per message filename under the durable correlation it was transported with, so a bounded multi-message prefix is N monotone entries rather than one record naming one file. Phases are monotone (`attempted < transportAccepted < persisted < admitted < consumed`) and a lowering write is refused, so a restart at any persistence boundary reads a true lower bound and holds instead of re-transporting. Each adapter may only record evidence its harness can honestly produce: Codex's typed completed user message is consumption and its true ceiling, OpenCode's durable read-back is storage and never releases ownership while prompt admission is merely unread. A v1 `delivery-state.json` is carried forward exactly once, label-downgrading (Codex `Accepted` → consumed, OpenCode `Accepted` → persisted, either `Attempted` → attempted), and adoption alone authorizes no transport: only a negative acknowledgement or a durable idempotent correlation does. An unreadable, foreign, or tampered record quarantines the pump — no transport, reason retained, evidence destroyed — instead of refusing to start. Before every first transport a never-advanced v1-shaped `Attempted` floor is written at the old path and re-asserted while the entry is outstanding, cleared only on release, so a rolled-back binary reconciles instead of duplicating. Retention is a decision, never a mutation: the ledger moves no inbox file, and the archive receipt remains the recipient agent's act and the sole settlement authority. | `src/delivery_ledger.rs::phase_is_monotonic_and_durable_across_every_write_boundary`; `src/delivery_ledger.rs::v1_accepted_adopts_at_the_evidence_it_actually_proved`; `src/delivery_ledger.rs::adoption_alone_never_authorizes_a_transport`; `src/delivery_ledger.rs::a_negative_receipt_is_retained_and_re_offered`; `src/delivery_ledger.rs::release_requires_the_harness_scheduler_or_a_true_ceiling`; `src/delivery_ledger.rs::unknown_and_dishonest_adapter_evidence_fails_closed`; `src/delivery_ledger.rs::an_unreadable_ledger_quarantines_the_pump_instead_of_refusing_to_start`; `src/delivery_ledger.rs::a_v1_readable_floor_precedes_the_first_transport_and_clears_only_on_release`; `src/delivery_ledger.rs::a_crash_at_any_persistence_boundary_never_duplicates_a_held_delivery`; `src/delivery_ledger.rs::a_bounded_multi_message_prefix_is_n_entries_under_one_correlation`; `src/opencode_session.rs::a_durable_read_back_is_persistence_that_never_releases_the_inbox_entry`; `src/opencode_session.rs::a_post_migration_delivery_leaves_a_v1_readable_floor_so_a_rollback_cannot_re_post`; `src/codex_app_server.rs::an_ambiguous_attempt_reconciles_resume_history_before_retry`; `tests/delivery_ledger_migration.rs::in_place_schema_bump_is_rejected_by_construction`; `tests/delivery_ledger_migration.rs::old_binary_rollback_neither_duplicates_nor_refuses_to_start` | | **Idempotent service requests** | A declared non-agent service principal publishes one exact JSON request per caller-supplied idempotency key to a canonical Agent Spec inbox. Concurrent or crash-replayed publication reuses the reserved filename; conflicting key reuse fails. The typed reply routes to the principal's canonical inbox without an Agent Spec identity or orphan mailbox. | `tests/request_cli.rs::stable_request_key_atomically_deduplicates_one_canonical_agent_message`; `tests/request_cli.rs::concurrent_replays_publish_exactly_one_request`; `tests/request_cli.rs::typed_reply_routes_to_the_principal_and_status_is_a_tagged_json_union`; `tests/request_cli.rs::request_api_rejects_agent_impersonation_and_unknown_flat_principals` | | **Bounded idempotent stream ingress** | A running agent accepts events only for a declared stream. Within the retained 128-receipt ring, concurrent or crash-replayed `(stream, event-id)` publication reuses one canonical filename, conflicting content fails, and supersession archives only the matching keyed predecessor or the stream-wide head through ordinary archive semantics. State remains bounded and honestly treats an identity evicted from the ring as new without searching inbox or archive history. Events do not write the Sent ledger and DING marks them as stream work. | `tests/event_e2e.rs::stable_event_identity_publishes_exactly_one_canonical_message`; `tests/event_e2e.rs::concurrent_replays_publish_exactly_one_event`; `tests/event_e2e.rs::conflicting_reuse_and_undeclared_or_suspended_ingress_fail_closed`; `tests/event_e2e.rs::supersede_collapses_only_the_matching_key_and_preserves_archive_receipts`; `tests/event_e2e.rs::keyless_supersede_replaces_the_stream_wide_head`; `tests/event_e2e.rs::crash_replay_honors_an_archive_receipt_and_never_restores_the_inbox_copy`; `tests/event_e2e.rs::subject_frontmatter_injection_is_refused_before_any_write`; `tests/event_e2e.rs::stream_state_is_bounded_and_forgets_only_beyond_its_honest_horizon`; `tests/event_e2e.rs::event_emit_cli_returns_a_stable_json_receipt_and_ding_marks_the_record`; `tests/stream_authoring_cli.rs::a_direct_adapter_launch_executes_the_exact_event_cli_contract`; `tests/run.rs::suspend_and_resume_relaunch_the_agent_and_stream_together` | | **Fail-closed observed native DING** | Each unread message becomes one normalized `[DING]` frame. Fresh delivery records ownership, then preserves the one combined bracketed-paste, 0.5 second delay, and Return transaction. PTY and Return success are transport only: `Delivered` additionally requires adapter classification of the expected notice text in a submitted-prompt or queued-message pattern while the lowest live composer is empty or an accepted idle placeholder. Retry never re-pastes and may send one bare Return only after two adjacent `RetainedSafe` observations. A maintained adapter's positive `NotRetained` observation releases only an already archived staged head; unread, blocked, timed-out, errored, unknown, and unrecognized states retain ownership and later FIFO work remains blocked. Ownership prevents duplicate paste across command failures, receipt ambiguity, archive races, and restart adoption without letting a vanished archived head block FIFO indefinitely. Startup backlog otherwise becomes one generic recovery DING; new arrivals remain FIFO; `busy` delivers immediately; only fresh `dnd` defers. | `src/ding/mod.rs::poke_text_normalizes_and_bounds_untrusted_fields`; `src/ding/mod.rs::malicious_controls_cannot_escape_the_single_paste_frame`; `src/ding/mod.rs::pty_delivery_uses_face607_delay_order_and_seconds`; `src/ding/mod.rs::maintained_composer_classifiers_require_exact_idle_state`; `src/ding/mod.rs::successful_transport_with_retained_or_unproven_pixels_is_not_delivered`; `src/ding/mod.rs::ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership`; `src/ding/mod.rs::adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt`; `src/ding/mod.rs::staged_retry_submits_only_retained_safe_and_requires_a_receipt`; `src/ding/mod.rs::staged_retry_keeps_unproven_and_retained_blocked_owned`; `src/ding/mod.rs::staged_ownership_survives_archive_and_never_repastes`; `src/ding/mod.rs::archived_not_retained_releases_fifo_without_repasting_owned_notice`; `src/ding/mod.rs::unread_not_retained_keeps_fifo_ownership_without_repasting`; `src/ding/mod.rs::pty_commands_have_a_real_outer_timeout`; `src/ding/mod.rs::session_watch_has_startup_grace_debounce_and_live_reset`; `src/ding/mod.rs::new_arrivals_is_fifo_and_archive_receipts_prevent_reding`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry`; `src/ding/mod.rs::startup_recovery_notice_retries_in_memory`; `src/ding/mod.rs::startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke` | @@ -23,11 +24,11 @@ materialization, messaging, DING, or presence must preserve them. | **Stable roster JSON** | `st2 agents --json [--enrich]` appends immutable `id`, effective `address`, and nullable `busAddress` (null for a proved non-routable retired subject) after its existing fields, keeps `identity` as the positional `.` declaration key, projects each supervisor as the parent's migrated ID, and otherwise preserves field names, order, null handling, presence, typed desired state and rationale, the retirement compatibility projection, opaque declared Resource descriptors, origin-timed activity, inbox counts, and the appended `observedState`, `driverDiagnostic`, and `context` objects. Declared presence, desired lifecycle, observed harness state, native-driver degradation, and harness context are independent axes: none is derived from another; a missing observed or context record is `null`, while missing diagnostic evidence is explicitly `absent`, never healthy. The version 3 axes and the shared disposition are appended, never substituted: `observedState` keeps its shipped fields and gains the exact declared record `schema` — which is what makes a migration's drain gate positive — plus typed `indeterminacy`, the tagged `condition`, the tagged `humanAsk`, and a `conversationRef` that is `null` when the record claims no capability, and each row gains a `disposition` of exactly three closed axes. That disposition is a row-level sibling because it folds both `observedState` and `driverDiagnostic`, it comes from the one shared derivation the catalog graph and Doctor also read, and it rewrites no raw axis. The catalog graph's runtime value keeps every key it carried — presence, activity, inbox, `observedState`, `driverDiagnostic`, and `context` — with the shared `disposition` appended beside them, never displacing one. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `src/agents.rs::observed_state_joins_declared_presence_without_touching_either`; `src/agents.rs::driver_diagnostic_wire_exposes_failure_and_evidence_age_without_identity_payloads`; `src/agents.rs::context_is_a_fourth_axis_that_survives_an_indeterminate_observed_state`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence`; `tests/status_agents.rs::roster_keeps_presence_separate_from_suspended_desired_state`; `tests/status_agents.rs::roster_uses_version_1_origin_time_for_last_activity`; `tests/status_agents.rs::roster_joins_a_real_context_record_independently_of_observed_state`; `src/agents.rs::the_fault_axis_and_the_shared_disposition_ride_the_roster_wire` | | **Agent-declared presence** | Refresh preserves non-DND declared status and advances the version 1 heartbeat. A missing status starts as `available`. Legacy DND migrates without renewing its hold. Version 1 DND is not refreshed. Stale, malformed, or implausibly future heartbeats read as `unknown`. The outer Codex, Claude, and pi session wrappers own a five-minute heartbeat while their provider remains alive. | `src/status.rs::refresh_preserves_value_and_changes_heartbeat_bytes`; `src/status.rs::refresh_upgrades_legacy_dnd_without_renewing_the_hold`; `src/status.rs::refresh_missing_writes_available_default`; `src/status.rs::version_1_staleness_and_future_skew_are_bounded`; `src/status.rs::malformed_versioned_record_is_unknown_without_mtime_fallback`; `src/claude_session.rs::idle_provider_refreshes_presence_without_mcp_input`; `src/pi_session.rs::idle_pi_provider_refreshes_presence_without_channel_input`; `src/codex_app_server.rs::inbox_fallback_does_not_write_a_fifteen_second_presence_heartbeat` | | **Scoped delivery-input wakeups** | Native delivery pumps watch only their inputs: the agent's `resources/inbox` subtree and its `status` file. Runtime records written beside them by the pump's own process group — presence temp siblings, the `harness-state` and `harness-context` records with their locks and staged siblings, stream state — never wake delivery, so a producer that writes on every turn boundary cannot pump its own delivery loop. | `src/watch.rs::delivery_watcher_ignores_runtime_records_but_wakes_on_inbox_and_status` | -| **Observed harness state discipline** | The `harness-state` record is written only by the owning session's driver processes — wrapper, channel, or hooks — serialized by a cross-process lock and coalesced against the on-disk record, atomically and byte-distinct on every write that lands, with freshness from its embedded timestamp and never file mtime; restating an unchanged state touches the record only when the refresh cadence is due. `unknown` is derived, never written: staleness, future skew, malformation, an unsupported schema, and a provably dead pty session each read as `unknown` with a distinct reason; an indeterminate liveness probe downgrades nothing; a missing record is no observation rather than `unknown`; no absence derives a definite state. A writer that loses sight of its harness stops heartbeating instead of refreshing a state it cannot see, and a predecessor session's record is never re-stamped. A reaped provider yields a terminal `ended` record carrying its real exit — written before the stop path's SIGKILL escalation and rewritten from the escalation cover when a grace-window reap observes the real status, proven against the real wrapper binaries of both stop implementations — and never a live state. Faults are their own axis, in their own record version: `st2.harness-state.v3` carries a tagged condition whose fault names a closed category, an open provider-namespaced code, a recovery class, and its own SEMANTIC observation time, beside a tagged ask axis that speaks only about actual human prompts — a fault is not an ask, and where both hold remediation is primary while the ask stays visible. Versions 1 and 2 project their condition as explicitly absent: never `clear`, and no fault is inferred from their legacy words. Every contradiction is rejected with its own reason word and reported as typed indeterminacy carrying the age of the evidence, while an unrecognized category leaves a fault untyped and still routed by its recovery, and an unrecognized recovery is treated like one needing a human. The heartbeat moves transport freshness only: it never moves a fault's observation time or its deadline, attention is derived at read time, and an automatic recovery past its own deadline becomes an untyped unknown-recovery fault that pages until an explicit paired clear, a terminal record, a new claim, or a new incarnation replaces it. One pure function folds those axes and the sibling driver diagnostic into exactly three closed disposition axes that the roster, the catalog graph, and Doctor all read rather than re-derive, where ended and record-level indeterminate never page and no delivery path reads the result. A conversation reference is identity and capability only — tagged, with an explicit history-mutability claim, its evidence, and a positive finite verification bound — and carries no conversation content. Reader-first holds for the new version too: this build reads, strictly validates, and projects version 3 while one writer-selection point keeps the writer on version 2, and the version-independent ownership envelope makes a version 2 claim refuse a version 3 record rather than overwrite a meaning it cannot read, while still carrying an undecodable record's ownership sequence forward. Strictness is scoped to what a contradiction damages, because it must not destroy evidence: a recovery deadline is refused only on a recovery this version recognizes as non-automatic, an unrecognized recovery word keeps its fault paging, an inverted deadline is refused whatever the recovery says, and a badly stated conversation reference degrades only that axis to `unavailable` carrying st2's own rejection word while the activity, condition, and ask axes stand. | `src/harness_state.rs::unknown_state_is_derived_and_cannot_be_written`; `src/harness_state.rs::malformed_record_is_unknown_without_mtime_fallback`; `src/harness_state.rs::staleness_and_future_skew_derive_unknown_with_distinct_reasons`; `src/harness_state.rs::a_dead_session_reads_unknown_even_while_fresh_but_ended_survives`; `src/harness_state.rs::every_landed_write_is_byte_distinct_and_fresh_restatements_do_not_write`; `src/harness_state.rs::a_chatty_producer_restating_its_state_causes_zero_writes`; `src/harness_state.rs::concurrent_writers_defer_to_the_on_disk_record_not_their_cache`; `src/harness_state.rs::a_predecessor_sessions_record_is_never_heartbeat_eligible`; `src/harness_state.rs::missing_record_reads_as_none_not_unknown`; `src/codex_app_server.rs::pump_publishes_observations_and_stops_heartbeating_on_evidence_loss`; `src/claude_session.rs::a_provider_killed_mid_turn_reads_ended_rather_than_active`; `src/claude_session.rs::a_clean_provider_exit_writes_the_terminal_record`; `tests/harness_state_teardown.rs::stop_escalation_writes_the_terminal_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_stop_escalation_writes_the_cover_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_graceful_stop_records_the_real_reaped_exit`; `src/harness_state.rs::golden_fixtures_cover_every_shape_a_consumer_must_handle`; `src/harness_state.rs::strict_version_three_edges_are_rejected_with_distinct_reasons`; `src/harness_state.rs::the_conversation_bridge_requires_a_complete_finite_link`; `src/harness_state.rs::the_heartbeat_moves_transport_freshness_only_and_never_the_semantic_fault_clock`; `src/harness_state.rs::a_driver_diagnostic_failure_contributes_without_touching_raw_axes_or_paging_the_dead`; `src/harness_state.rs::a_version_three_record_is_read_but_never_written_over_by_this_builds_writer`; `src/harness_state.rs::an_undecodable_records_envelope_still_carries_its_ownership_forward` | -| **Harness context discipline** | The numeric `harness-context` record is a sibling of `harness-state` and shares none of its ownership machinery: `incarnation` is provenance and never a fence, and a straggler's write lands rather than being refused. A reading is written only when it enters a different bucket of `usedPercent`, when a compaction edge arrives, when proven Claude account-window exhaustion changes, or when the record is older than the heartbeat, so a chatty producer cannot inflate the write rate; Codex account-window occupancy does not classify availability without its separate credit metadata. Every landed write is byte-distinct, and freshness comes from the record's own `observedAtMs`, never file mtime. A withheld value is carried as `null` and never fabricated from zero, the previous reading, or a division st2 could have done itself, and a reading above the window is carried unclamped. There is no `unknown` on this axis: past the horizon the reading is returned marked `stale` with its age, surviving every derivation the categorical record makes, while an unparseable record, a foreign schema, an uninterpretable harness, and an untrustworthy clock each read as absent. The relaunch claim removes the record, and Doctor's high-reading and stale-record lines are advisory in both directions and never change its exit status. Where a producer ships, its numerator is the harness's own and is pinned to the build it was measured on: pi's `tokens` is the last assistant message's `totalTokens` and omp's is its prompt figure, so neither harness's arithmetic can be published under the other's tag, and a harness bump that changes a number's meaning without changing its shape fails a fixture rather than shipping. A harness that positively reports it does not know its own occupancy is forwarded as withheld in the same write as the compaction edge that emptied it, a compaction edge whose harness names no reason yields `unknown` rather than an invented word, and a durable count the producer cannot read narrows to st2 counting edges rather than losing them. | `src/harness_context.rs::a_reading_inside_the_written_bucket_does_not_write_and_a_crossing_does`; `src/harness_context.rs::claude_rate_limit_exhaustion_and_reset_crossings_land_inside_one_usage_bucket`; `src/harness_context.rs::codex_account_window_exhaustion_does_not_prove_the_runtime_is_blocked`; `src/harness_context.rs::a_withheld_percent_has_no_bucket_and_only_a_compaction_or_heartbeat_writes`; `src/harness_context.rs::a_record_older_than_the_heartbeat_is_rewritten_and_every_landed_write_is_byte_distinct`; `src/harness_context.rs::a_compaction_always_lands_with_its_trigger_and_may_carry_a_durable_count`; `src/harness_context.rs::withheld_values_are_null_and_are_never_fabricated`; `src/harness_context.rs::a_reading_above_the_window_is_carried_unclamped`; `src/harness_context.rs::a_stale_reading_is_returned_with_its_age_rather_than_derived_away`; `src/harness_context.rs::freshness_comes_from_the_record_bytes_and_never_from_file_mtime`; `src/harness_context.rs::additive_fields_decode_but_foreign_schema_and_harness_read_as_nothing`; `src/harness_context.rs::a_record_beyond_the_future_skew_bound_reads_as_nothing`; `src/harness_context.rs::a_straggler_lands_and_is_visible_as_provenance_rather_than_being_refused`; `src/harness_state.rs::the_relaunch_claim_removes_the_harness_context_record`; `src/agents.rs::context_is_a_fourth_axis_that_survives_an_indeterminate_observed_state`; `tests/doctor.rs::harness_context_doctor_lines_are_advisory_and_never_change_the_exit_status`; `src/pi_channel.rs::the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/pi_channel.rs::a_pi_compaction_withholds_the_reading_it_emptied_in_the_same_write`; `src/pi_channel.rs::an_omp_compaction_yields_unknown_because_the_event_names_no_reason`; `src/pi_channel.rs::an_unreadable_durable_count_degrades_to_counting_edges_not_to_losing_them`; `src/pi_channel.rs::context_frames_decode_conservatively_or_not_at_all`; `src/pi_channel.rs::the_measured_pi_release_is_the_one_the_extension_gate_pins`; `src/omp_session.rs::the_measured_context_builds_are_admitted_by_this_gate` | +| **Observed harness state discipline** | The `harness-state` record is written only by the owning session's driver processes — wrapper, channel, or hooks — serialized by a cross-process lock and coalesced against the on-disk record, atomically and byte-distinct on every write that lands, with freshness from its embedded timestamp and never file mtime; restating an unchanged state touches the record only when the refresh cadence is due. `unknown` is derived, never written: staleness, future skew, malformation, an unsupported schema, and a provably dead pty session each read as `unknown` with a distinct reason; an indeterminate liveness probe downgrades nothing; a missing record is no observation rather than `unknown`; no absence derives a definite state. A writer that loses sight of its harness stops heartbeating instead of refreshing a state it cannot see, and a predecessor session's record is never re-stamped. A reaped provider yields a terminal `ended` record carrying its real exit — written before the stop path's SIGKILL escalation and rewritten from the escalation cover when a grace-window reap observes the real status, proven against the real wrapper binaries of both stop implementations — and never a live state. Faults are their own axis, in their own record version: `st2.harness-state.v3` carries a tagged condition whose fault names a closed category, an open provider-namespaced code, a recovery class, and its own SEMANTIC observation time, beside a tagged ask axis that speaks only about actual human prompts — a fault is not an ask, and where both hold remediation is primary while the ask stays visible. Versions 1 and 2 project their condition as explicitly absent: never `clear`, and no fault is inferred from their legacy words. Every contradiction is rejected with its own reason word and reported as typed indeterminacy carrying the age of the evidence, while an unrecognized category leaves a fault untyped and still routed by its recovery, and an unrecognized recovery is treated like one needing a human. The heartbeat moves transport freshness only: it never moves a fault's observation time or its deadline, attention is derived at read time, and an automatic recovery past its own deadline becomes an untyped unknown-recovery fault that pages until an explicit paired clear, a terminal record, a new claim, or a new incarnation replaces it. One pure function folds those axes and the sibling driver diagnostic into exactly three closed disposition axes that the roster, the catalog graph, and Doctor all read rather than re-derive, where ended and record-level indeterminate never page and no delivery path reads the result. A conversation reference is identity and capability only — tagged, with an explicit history-mutability claim, its evidence, and a positive finite verification bound — and carries no conversation content. Version 3 is ACTIVE: one writer-selection point puts the writer on version 3, versions 1 and 2 remain readable and project their condition as explicitly absent rather than as health, and the version-independent ownership envelope keeps the claim fence one-way from both sides — a version 3 claim supersedes every supported version and no older claim supersedes a version 3 record rather than overwriting a meaning it cannot read, while still carrying an undecodable record's ownership sequence forward. Because a claim fence states nothing and is never carried forward, every terminal owner states the condition axis itself when nobody else did, so a wrapper-only `ended` still lands. Strictness is scoped to what a contradiction damages, because it must not destroy evidence: a recovery deadline is refused only on a recovery this version recognizes as non-automatic, an unrecognized recovery word keeps its fault paging, an inverted deadline is refused whatever the recovery says, and a badly stated conversation reference degrades only that axis to `unavailable` carrying st2's own rejection word while the activity, condition, and ask axes stand. | `src/harness_state.rs::unknown_state_is_derived_and_cannot_be_written`; `src/harness_state.rs::malformed_record_is_unknown_without_mtime_fallback`; `src/harness_state.rs::staleness_and_future_skew_derive_unknown_with_distinct_reasons`; `src/harness_state.rs::a_dead_session_reads_unknown_even_while_fresh_but_ended_survives`; `src/harness_state.rs::every_landed_write_is_byte_distinct_and_fresh_restatements_do_not_write`; `src/harness_state.rs::a_chatty_producer_restating_its_state_causes_zero_writes`; `src/harness_state.rs::concurrent_writers_defer_to_the_on_disk_record_not_their_cache`; `src/harness_state.rs::a_predecessor_sessions_record_is_never_heartbeat_eligible`; `src/harness_state.rs::missing_record_reads_as_none_not_unknown`; `src/codex_app_server.rs::pump_publishes_observations_and_stops_heartbeating_on_evidence_loss`; `src/claude_session.rs::a_provider_killed_mid_turn_reads_ended_rather_than_active`; `src/claude_session.rs::a_clean_provider_exit_writes_the_terminal_record`; `tests/harness_state_teardown.rs::stop_escalation_writes_the_terminal_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_stop_escalation_writes_the_cover_record_before_sigkill`; `tests/harness_state_teardown.rs::opencode_graceful_stop_records_the_real_reaped_exit`; `src/harness_state.rs::golden_fixtures_cover_every_shape_a_consumer_must_handle`; `src/harness_state.rs::strict_version_three_edges_are_rejected_with_distinct_reasons`; `src/harness_state.rs::the_conversation_bridge_requires_a_complete_finite_link`; `src/harness_state.rs::the_heartbeat_moves_transport_freshness_only_and_never_the_semantic_fault_clock`; `src/harness_state.rs::a_driver_diagnostic_failure_contributes_without_touching_raw_axes_or_paging_the_dead`; `src/harness_state.rs::a_version_two_writer_never_takes_over_a_version_three_record`; `src/harness_state.rs::the_writer_declares_version_three_and_still_means_an_immutable_id`; `src/harness_state.rs::a_version_three_claim_supersedes_every_supported_version_and_is_never_superseded`; `src/harness_state.rs::a_legacy_observation_projects_into_the_version_three_wire`; `src/harness_state.rs::a_legacy_record_is_still_read_and_projected_after_the_writer_cutover`; `src/harness_state.rs::a_terminal_record_lands_even_when_no_producer_ever_stated_the_condition_axis`; `src/harness_state.rs::the_version_two_projection_is_byte_identical_and_carries_no_condition_axis`; `src/pi_channel.rs::a_pi_failure_lands_a_version_three_fault_beside_an_honest_idle`; `src/pi_channel.rs::an_omp_approval_lands_a_version_three_ask_and_links_its_conversation`; `src/harness_state.rs::an_undecodable_records_envelope_still_carries_its_ownership_forward` | +| **Harness context discipline** | The numeric `harness-context` record is a sibling of `harness-state` and shares none of its ownership machinery: `incarnation` is provenance and never a fence, and a straggler's write lands rather than being refused. A reading is written only when it enters a different bucket of `usedPercent`, when a compaction edge arrives, when proven Claude account-window exhaustion changes, or when the record is older than the heartbeat, so a chatty producer cannot inflate the write rate; Codex account-window occupancy does not classify availability without its separate credit metadata. Every landed write is byte-distinct, and freshness comes from the record's own `observedAtMs`, never file mtime. A withheld value is carried as `null` and never fabricated from zero, the previous reading, or a division st2 could have done itself, and a reading above the window is carried unclamped. There is no `unknown` on this axis: past the horizon the reading is returned marked `stale` with its age, surviving every derivation the categorical record makes, while an unparseable record, a foreign schema, an uninterpretable harness, and an untrustworthy clock each read as absent. The relaunch claim removes the record, and Doctor's high-reading and stale-record lines are advisory in both directions and never change its exit status. Where a producer ships, its numerator is the harness's own and is pinned to the build it was measured on: pi's `tokens` is the last assistant message's `totalTokens` and omp's is its prompt figure, so neither harness's arithmetic can be published under the other's tag, and a harness bump that changes a number's meaning without changing its shape fails a fixture rather than shipping. A harness that positively reports it does not know its own occupancy is forwarded as withheld in the same write as the compaction edge that emptied it, a compaction edge whose harness names no reason yields `unknown` rather than an invented word, and a durable count the producer cannot read narrows to st2 counting edges rather than losing them. | `src/harness_context.rs::a_reading_inside_the_written_bucket_does_not_write_and_a_crossing_does`; `src/harness_context.rs::claude_rate_limit_exhaustion_and_reset_crossings_land_inside_one_usage_bucket`; `src/harness_context.rs::codex_account_window_exhaustion_does_not_prove_the_runtime_is_blocked`; `src/harness_context.rs::a_withheld_percent_has_no_bucket_and_only_a_compaction_or_heartbeat_writes`; `src/harness_context.rs::a_record_older_than_the_heartbeat_is_rewritten_and_every_landed_write_is_byte_distinct`; `src/harness_context.rs::a_compaction_always_lands_with_its_trigger_and_may_carry_a_durable_count`; `src/harness_context.rs::withheld_values_are_null_and_are_never_fabricated`; `src/harness_context.rs::a_reading_above_the_window_is_carried_unclamped`; `src/harness_context.rs::a_stale_reading_is_returned_with_its_age_rather_than_derived_away`; `src/harness_context.rs::freshness_comes_from_the_record_bytes_and_never_from_file_mtime`; `src/harness_context.rs::additive_fields_decode_but_foreign_schema_and_harness_read_as_nothing`; `src/harness_context.rs::a_record_beyond_the_future_skew_bound_reads_as_nothing`; `src/harness_context.rs::a_straggler_lands_and_is_visible_as_provenance_rather_than_being_refused`; `src/harness_state.rs::the_relaunch_claim_removes_the_harness_context_record`; `src/agents.rs::context_is_a_fourth_axis_that_survives_an_indeterminate_observed_state`; `tests/doctor.rs::harness_context_doctor_lines_are_advisory_and_never_change_the_exit_status`; `src/pi_channel.rs::the_pinned_pi_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/pi_channel.rs::a_pi_compaction_withholds_the_reading_it_emptied_in_the_same_write`; `src/pi_channel.rs::an_omp_compaction_yields_unknown_because_the_event_names_no_reason`; `src/pi_channel.rs::an_unreadable_durable_count_degrades_to_counting_edges_not_to_losing_them`; `src/pi_channel.rs::context_frames_decode_conservatively_or_not_at_all`; `src/pi_channel.rs::the_measured_pi_release_is_the_one_the_extension_gate_pins`; `src/omp_session.rs::the_measured_context_builds_are_admitted_by_this_gate` | | **Replicated-path discipline** | st2 pins the exact driver-record names it expects the replication transport's include list to carry — `harness-state` and `harness-context` — and both readers derive their paths from that list, so a rename cannot silently stop replication in a repository that does not own the list. A harness-context write stages in the catalog control plane, outside the complete `agents` identity namespace, after validating canonical ancestry and same-filesystem atomic rename. Current-catalog identity walkers overlook only an exact legacy `.harness-context.tmp--` regular non-symlink host child and leave it untouched; directories, symlinks, special files, generic dotfiles, near misses, and prepared topology remain strict. `harness-state` still stages beside itself, pending its own change; the shared write helper takes the staging directory as an argument precisely because the two answer this differently. | `src/harness_context.rs::the_replicated_driver_record_names_are_pinned`; `src/harness_context.rs::writes_stage_in_catalog_control_and_clean_up_after_success_or_failure`; `src/catalog_transaction.rs::current_projection_ignores_and_preserves_an_exact_legacy_staging_file`; `src/catalog_transaction.rs::current_projection_rejects_legacy_shape_type_confusion_and_near_misses`; `src/message.rs::transition_addressability_ignores_and_preserves_exact_legacy_staging_files`; `src/message.rs::transition_addressability_rejects_legacy_type_confusion_and_near_misses`; `tests/catalog_apply.rs::snapshot_ignores_and_preserves_an_exact_legacy_harness_context_staging_file` | | **Status-line slot chaining** | Claude's `statusLine` is a single slot whose winning declaration replaces the others outright, and `.claude/settings.local.json` — the file st2 materializes — wins over every other settings file. So st2's entry there is a tee: it records the reading and then invokes the operator's own renderer, resolved from `$ST_CLAUDE_STATUSLINE_RENDERER` and then `~/.claude/statusline-renderer.json`, first hit wins and never both. Where no renderer resolves, and wherever a resolved renderer fails, the tee writes NOTHING to stdout and puts its diagnostic on stderr: the payload is machine JSON, so echoing it into the slot is strictly worse for the operator than a blank row. Recording is unaffected by which arm runs. | `tests/claude_statusline.rs::the_tee_records_the_reading_and_hands_the_same_payload_to_the_env_renderer`; `tests/claude_statusline.rs::the_operator_file_supplies_the_renderer_when_no_variable_does`; `tests/claude_statusline.rs::the_variable_wins_over_the_file_and_the_file_is_never_also_run`; `tests/claude_statusline.rs::with_no_renderer_the_tee_renders_nothing_rather_than_the_raw_payload`; `tests/claude_statusline.rs::a_recording_failure_still_renders_the_status_line`; `tests/claude_statusline.rs::a_renderer_that_exits_non_zero_leaves_stdout_empty`; `tests/claude_statusline.rs::a_renderer_file_that_is_not_executable_leaves_stdout_empty`; `tests/claude_statusline.rs::without_st2_on_path_the_script_drains_stdin_and_renders_nothing`; `tests/claude_statusline.rs::the_rendered_registration_names_the_chaining_tee_and_carries_the_whole_slot` | -| **Version-pinned producer arithmetic** | Every harness-context producer publishes the number its own harness means by that number, and each one is pinned to the exact build the meaning was measured on: Claude's integer percent over `total_input_tokens` and `context_window_size` (2.1.250), Codex's window with the 12,000 baseline subtracted from both sides (codex-cli 0.151.0), pi's last assistant `totalTokens` and omp's prompt-only `input` (0.84.2 and 18.0.9), and OpenCode's last non-summary assistant total over the providers window (1.18.25). The failure this bounds is a harness bump that changes what a key means without changing its shape, which no type gate and no round-trip assertion can see, so each fixture decodes a verbatim capture, asserts the version literally, and asserts the wrong numerator the same capture would yield. | `src/claude_session.rs::a_mid_session_statusline_payload_yields_claudes_own_triple`; `src/claude_session.rs::a_pre_turn_statusline_payload_withholds_rather_than_reporting_zero`; `src/codex_app_server.rs::codex_context_recomputes_the_captured_reading_and_pins_its_verified_version`; `src/pi_channel.rs::the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/opencode_session.rs::captured_opencode_turns_publish_the_assistant_total_over_the_providers_window` | +| **Version-pinned producer arithmetic** | Every harness-context producer publishes the number its own harness means by that number, and each one is pinned to the exact build the meaning was measured on: Claude's integer percent over `total_input_tokens` and `context_window_size` (2.1.250), Codex's window with the 12,000 baseline subtracted from both sides (codex-cli 0.151.0), pi's last assistant `totalTokens` and omp's prompt-only `input` (0.84.4 and 18.0.9), and OpenCode's last non-summary assistant total over the providers window (1.18.25). The failure this bounds is a harness bump that changes what a key means without changing its shape, which no type gate and no round-trip assertion can see, so each fixture decodes a verbatim capture, asserts the version literally, and asserts the wrong numerator the same capture would yield. | `src/claude_session.rs::a_mid_session_statusline_payload_yields_claudes_own_triple`; `src/claude_session.rs::a_pre_turn_statusline_payload_withholds_rather_than_reporting_zero`; `src/codex_app_server.rs::codex_context_recomputes_the_captured_reading_and_pins_its_verified_version`; `src/pi_channel.rs::the_pinned_pi_fixture_pins_total_tokens_as_the_numerator`; `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`; `src/opencode_session.rs::captured_opencode_turns_publish_the_assistant_total_over_the_providers_window` | | **Native-driver diagnostics are bounded and recoverable** | One additive-tolerant `st2.driver-diagnostic.v1` snapshot exposes the earliest failing native-driver boundary through closed driver/stage/reason/source/support vocabulary and origin-timed evidence age. Absent, malformed, foreign, unknown, and wrongly paired records never read healthy. Stage recovery clears only its own failure and removes the record after final recovery without changing prompt submission, retry, read-back, or archive semantics. A rejected provider credential is published on its own `providerAuth` stage from the harness's own typed turn result — never from provider prose, and never from an auth-class flag a capacity, policy, or retry class also sets — outranking the delivery and read-back symptoms it causes while the four producer gates outrank it; only a turn that reached its ordinary end clears it, and no credential knowledge or remedy text enters st2. Roster and Doctor share the typed reader/repair policy — Doctor advising on absence only for a driver that publishes a boundary result on every launch; telemetry labels contain only bounded driver/stage/reason/source/support/outcome values, never versions, identities, prompts, messages, or paths. | `src/driver_diagnostic.rs::every_stage_reason_and_source_has_bounded_wire_vocabulary`; `src/driver_diagnostic.rs::additive_fields_decode_but_malformed_foreign_and_unknown_records_are_indeterminate`; `src/driver_diagnostic.rs::recovery_clears_only_its_stage_and_reveals_the_next_failure`; `src/driver_diagnostic.rs::a_credential_rejection_is_evidence_only_from_a_typed_turn_result`; `src/driver_diagnostic.rs::a_rejected_credential_outranks_its_symptoms_but_not_the_producer_gates`; `src/claude_session.rs::stop_failure_classifies_only_the_credential_class_as_provider_auth`; `src/claude_session.rs::a_rejected_claude_credential_stands_until_a_turn_reaches_its_ordinary_end`; `src/codex_app_server.rs::a_rejected_codex_credential_reads_provider_auth_while_a_quota_failure_does_not`; `src/codex_app_server.rs::protocol_schema_gate_requires_the_distinct_credential_and_quota_error_words`; `src/opencode_session.rs::delivery_and_read_back_boundaries_publish_and_clear_diagnostics_without_changing_retry`; `src/metrics.rs::driver_diagnostic_metric_attributes_are_exactly_the_bounded_axes`; `tests/doctor.rs::native_driver_diagnostic_roster_and_doctor_agree_and_recovery_clears`; `src/pi_channel.rs::only_omps_own_credential_class_becomes_provider_auth`; `src/pi_channel.rs::a_rejected_omp_credential_stands_until_a_turn_reaches_its_ordinary_end`; `src/pi_channel.rs::the_pi_channel_publishes_no_credential_verdict`; `tests/doctor.rs::a_rejected_provider_credential_advises_while_absence_stays_silent` | | **Retirement health** | A retired declaration is healthy only after every declared task ID is absent. Any live or dead declared task record reports incomplete retirement; retired declarations do not require presence. Live declarations retain their existing task and presence checks. | `tests/doctor.rs::retired_declaration_is_healthy_when_tasks_and_presence_are_absent`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_declared_task_is_alive`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_dead_task_record_remains` | | **Suspension health** | A suspended declaration is healthy when no declared task is live and every retained dead record is explicitly keep-pinned. It requires no presence, but this weaker result never proves retirement. Resume preserves ordinary keep and adopt-only policy. | `tests/doctor.rs::suspended_declaration_is_healthy_when_tasks_are_absent_without_presence`; `tests/doctor.rs::suspended_declaration_distinguishes_live_dead_keep_and_dead_nonkeep`; `tests/reconcile.rs::resuming_uses_ordinary_reconcile_and_does_not_override_keep` | diff --git a/docs/vrs/08-harness-context/spec.md b/docs/vrs/08-harness-context/spec.md index 6b9d4d40..764f687e 100644 --- a/docs/vrs/08-harness-context/spec.md +++ b/docs/vrs/08-harness-context/spec.md @@ -415,7 +415,7 @@ number (HC-R02); where st2 computes it, the row says so. | --- | --- | --- | --- | --- | --- | | claude | 2.1.250 | `statusLine` command stdin JSON | `context_window.total_input_tokens` = `input + cache_creation + cache_read` of the last response | `context_window.context_window_size` | `context_window.used_percentage` — Claude's own integer, clamped 0..100 | | codex | codex-cli 0.151.0 | app-server `thread/tokenUsage/updated` | `tokenUsage.last.totalTokens` | `tokenUsage.modelContextWindow` | st2 computes with the baseline rule below; equals `100 −` Codex's displayed "% context left" | -| pi | 0.84.2 | injected extension `ctx.getContextUsage()` | `.tokens` = last assistant `totalTokens` (input + output + cacheRead + cacheWrite) | `.contextWindow` | `.percent` (float) | +| pi | 0.84.4 | injected extension `ctx.getContextUsage()` | `.tokens` = last assistant `totalTokens` (input + output + cacheRead + cacheWrite) | `.contextWindow` | `.percent` (float) | | omp | 18.0.9 (and 18.0.3) | injected extension `ctx.getContextUsage()` | `.tokens` = last assistant **`input`** only | `.contextWindow` | `.percent` (float) | | opencode | 1.18.25 | SSE `message.updated` joined with `GET /config/providers` | last **non-summary** assistant `tokens.total` | `providers[].models[].limit.context` | st2 computes `usedTokens / windowTokens`; the server displays none | @@ -727,7 +727,7 @@ turn the measurement into a semantic-version promise. Every later Codex release still requires its own source comparison and live delivery proof before the exact launch gate moves. -### pi (0.84.2) +### pi (0.84.4) One call answers everything, present on the `ctx` of every lifecycle event: `getContextUsage()` returns `{ tokens, contextWindow, percent }`. @@ -1126,7 +1126,7 @@ each only once a real test proves it (per `CLAUDE.md`): baseline-free `last.totalTokens / window` reads 36, the truth is 33). **Shipped for pi and omp**: - `src/pi_channel.rs::the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator`, + `src/pi_channel.rs::the_pinned_pi_fixture_pins_total_tokens_as_the_numerator`, `src/pi_channel.rs::the_omp_18_0_9_fixture_pins_prompt_input_as_the_numerator`, `src/pi_channel.rs::a_pi_compaction_withholds_the_reading_it_emptied_in_the_same_write`, `src/pi_channel.rs::an_omp_compaction_yields_unknown_because_the_event_names_no_reason`, diff --git a/flake.nix b/flake.nix index 7a30cdaf..5801a1f2 100644 --- a/flake.nix +++ b/flake.nix @@ -492,10 +492,10 @@ # without integrity hashes, which `fetchNpmDeps` cannot express. checks.pi-extension-types = let - piVersion = "0.84.2"; + piVersion = "0.84.4"; piTarball = pkgs.fetchurl { url = "https://registry.npmjs.org/@earendil-works/pi-coding-agent/-/pi-coding-agent-${piVersion}.tgz"; - hash = "sha256-lbiZzXsaDB8BdMe/M6tCdDXjVTp9H0dWZhqpx/Gmj/o="; + hash = "sha256-W852bRnDzroY8/uq2RxEnJ+dc5gfnjQA7O+TIAbwaWg="; }; nodeTypesTarball = pkgs.fetchurl { url = "https://registry.npmjs.org/@types/node/-/node-26.2.0.tgz"; diff --git a/hooks/omp-channel.ts b/hooks/omp-channel.ts index 601d65ff..ee239c2f 100644 --- a/hooks/omp-channel.ts +++ b/hooks/omp-channel.ts @@ -17,8 +17,19 @@ import type { ExtensionContext, } from "@earendil-works/pi-coding-agent"; +// The wire version this asset ACCEPTS on the hello. It never rises on its own: the hello is +// st2 → asset and is written before any read, so a control plane that unilaterally raised this +// number would be refused by every already-loaded predecessor asset, and a refusal costs that +// seat its mail. const PROTOCOL = 1; +// The protocol this asset can additionally SPEAK, answered with a `client_hello` when st2's hello +// offers it. It adds no new observation to the frames below; it is this asset's positive statement +// that it retires a never-answered ask on a turn boundary and forwards omp's own `sessionId` — +// which is what lets st2 state the ask axis positively and link the conversation. An st2 that +// does not offer it never sees the answer, and an older asset never sends one. +const PROTOCOL_CONDITION_AXIS = 2; + const BIN = "ST2_OMP_CHANNEL_BIN"; const CATALOG = "ST2_OMP_CHANNEL_CATALOG"; const IDENTITY = "ST2_OMP_CHANNEL_IDENTITY"; @@ -30,9 +41,14 @@ const SEQ = "ST2_OMP_CHANNEL_SEQ"; // and never worth a hung agent. const HELLO_TIMEOUT_MS = 5000; +/** Diagnostic prose for a refused approval. It is not a condition and no reader branches on it. */ +const APPROVAL_DENIED = "approvalDenied"; + type Frame = { type?: string; protocol?: number; + /** Every protocol st2 would accept on this connection; absent on a version-1 control plane. */ + protocols?: unknown; sessionContext?: string; content?: string; deliverAs?: "steer" | "followUp"; @@ -68,6 +84,15 @@ type Stash = { pendingAskToolCallId?: string; /** Generation fencing every bounded settle poll against newer activity. */ settleGeneration?: number; + /** + * The protocol st2 and this asset agreed on for the LIVE channel: undefined until a hello + * offers version 2 and this asset answers. It is per-connection, never per-process, because a + * session replacement may re-spawn a DIFFERENT st2 binary; the stash outlives the channel, so + * a stale agreement would let this asset speak a wire its current peer never offered. + */ + negotiated?: number; + /** omp's own session id, once an event has exposed one. Forwarded exactly once per channel. */ + conversationSessionId?: string; }; /** @@ -137,7 +162,7 @@ const terminalProviderError = (event: AgentEndFrame): ProviderError | undefined * build whose telemetry surface moved still loads and still delivers mail, and a widened cast alone * would make that tolerance absolute and silent. Erased at runtime. * - * Note what this can and cannot prove for omp. It pins the SHAPE against pi 0.84.2's typings, which + * Note what this can and cannot prove for omp. It pins the SHAPE against pi 0.84.4's typings, which * is all this asset compiles against — omp ships no typings of its own. It cannot prove omp's * `tokens` still means prompt-only input, because that is a meaning and not a shape; the * version-pinned fixture in `src/pi_channel.rs` is what bounds that (HC-R13, HC-T03). @@ -253,6 +278,11 @@ export default function (pi: ExtensionAPI) { // session's cost as their own. state.lastCostUsd = undefined; state.pendingAskToolCallId = undefined; + // Both are properties of the CONNECTION, not of the process: the successor negotiates for + // itself, and it must re-state omp's session id rather than assume its predecessor's peer + // already recorded it. + state.negotiated = undefined; + state.conversationSessionId = undefined; cancelSettle(); const channelEnv: NodeJS.ProcessEnv = { ...process.env }; @@ -311,6 +341,15 @@ export default function (pi: ExtensionAPI) { return; } clearTimeout(timer); + // Answer the offer — if there is one — as this asset's FIRST write. st2 keys the whole + // connection off this frame: a state frame that reached it first would be folded on the + // legacy path, and the axes below would then be read by a peer that never agreed to + // them. An st2 that offers nothing gets no answer and keeps version 1. + const offered = Array.isArray(frame.protocols) ? frame.protocols : []; + if (offered.includes(PROTOCOL_CONDITION_AXIS)) { + state.negotiated = PROTOCOL_CONDITION_AXIS; + send({ type: "client_hello", protocol: PROTOCOL_CONDITION_AXIS }); + } settle(typeof frame.sessionContext === "string" ? frame.sessionContext : ""); return; } @@ -483,11 +522,21 @@ export default function (pi: ExtensionAPI) { // Registered only now that every helper above is initialized: a use-before-declaration in this // file is the defect class that once shipped green through the type gate. + // A turn boundary retires a never-answered ask. A DENIED ask emits no `tool_result` at all + // (DQ-OMP-1), so without this one denial would hold `pendingAskToolCallId` for the whole + // process lifetime — muting the approval surface and, worse, making st2's positive `none` on + // the ask axis unreachable for every later turn. + const retireAsk = () => { + state.pendingAskToolCallId = undefined; + }; + pi.on("agent_start", async () => { cancelSettle(); + retireAsk(); sendFrame({ type: "state", state: "active" }); }); pi.on("agent_end", async (event, ctx) => { + retireAsk(); captureCost(event); sendContext(ctx); const end = event as AgentEndFrame; @@ -598,10 +647,30 @@ export default function (pi: ExtensionAPI) { // Approval events are an independent human-blocking surface. omp's pinned pi typings do not // declare them, so register through the same widened `on` view. - type ApprovalFrame = { toolName?: unknown }; + type ApprovalFrame = { toolName?: unknown; sessionId?: unknown; approved?: unknown }; + /** + * omp's own `sessionId`, measured on BOTH halves of the approval pair and identical across it + * (18.0.9 and 18.1.2 captures). It is the only typed conversation identity omp exposes to an + * extension — `getContextUsage`, `sessionManager.getEntries`, and `model.id` carry none — so + * the approval events are the only place it can be read without inventing a probe. + * + * Forwarded once per channel and only to a negotiated peer. Until one is observed this asset + * says NOTHING about the axis, which is a different claim from "omp has no conversations". + * Captured before the ask-suppression check below: the identity is a separate axis from the + * ask, and an approval arriving under a pending ask is still evidence of the session. + */ + const noteConversation = (event: { sessionId?: unknown }) => { + if (state.negotiated !== PROTOCOL_CONDITION_AXIS) return; + const id = typeof event.sessionId === "string" ? event.sessionId.trim() : ""; + if (!id || id === state.conversationSessionId) return; + state.conversationSessionId = id; + sendFrame({ type: "conversation", sessionId: id }); + }; + onWidened("tool_approval_requested", async (rawEvent) => { - if (state.pendingAskToolCallId) return; const event = rawEvent as ApprovalFrame; + noteConversation(event); + if (state.pendingAskToolCallId) return; const tool = typeof event.toolName === "string" ? event.toolName : "unknown"; cancelSettle(); sendFrame({ @@ -612,13 +681,23 @@ export default function (pi: ExtensionAPI) { reason: tool, }); }); - onWidened("tool_approval_resolved", async (_event, ctx) => { + onWidened("tool_approval_resolved", async (rawEvent, ctx) => { + const event = rawEvent as ApprovalFrame; + noteConversation(event); if (state.pendingAskToolCallId) return; + // A denial is an INTERRUPTION, not a fault: nothing in the closed condition vocabulary names + // it, so the ask is simply over and the word rides as diagnostic prose. Stated only to a + // negotiated peer, which keeps the version-1 wire byte-identical. + const refused = typeof event.approved === "boolean" && !event.approved; + const denied = + state.negotiated === PROTOCOL_CONDITION_AXIS && refused + ? { reason: APPROVAL_DENIED } + : {}; if (idleProof(ctx)) { - sendFrame({ type: "state", state: "idle" }); + sendFrame({ type: "state", state: "idle", ...denied }); return; } - sendFrame({ type: "state", state: "active" }); + sendFrame({ type: "state", state: "active", ...denied }); watchSettle(ctx); }); diff --git a/hooks/pi-channel.ts b/hooks/pi-channel.ts index f95b7696..385f199f 100644 --- a/hooks/pi-channel.ts +++ b/hooks/pi-channel.ts @@ -20,7 +20,19 @@ import type { SessionShutdownEvent, } from "@earendil-works/pi-coding-agent"; -const PROTOCOL = 1; +// The wire versions this asset speaks, highest first in preference. Version 1 is the frame set +// this file shipped with; version 2 adds the condition frame and nothing else. Negotiation is +// `max(SUPPORTED ∩ hello.protocols)`, with the hello's scalar `protocol` as the only offer when an +// older control plane sends no list — so this asset keeps working against both, and a control +// plane that offers nothing this asset speaks is refused rather than guessed at. +const SUPPORTED = [1, 2] as const; +const CONDITION_PROTOCOL = 2; + +// The only two faults pi can prove, provider-namespaced so a code cannot collide with another +// harness's. Named here because the same strings appear on a raise and on its paired clear, and a +// clear whose code drifted from its raise silently stops clearing anything. +const ASSISTANT_ERROR = "pi/assistantError"; +const COMPACT_FAILED = "pi/session_compact_failed"; const BIN = "ST2_PI_CHANNEL_BIN"; const CATALOG = "ST2_PI_CHANNEL_CATALOG"; @@ -36,6 +48,7 @@ const HELLO_TIMEOUT_MS = 5000; type Frame = { type?: string; protocol?: number; + protocols?: unknown; sessionContext?: string; content?: string; deliverAs?: "steer" | "followUp"; @@ -58,6 +71,13 @@ type Stash = { session?: string; seq?: string; child?: childProcess.ChildProcess; + /** + * The wire version agreed with the channel this stash currently holds, unset until its hello + * arrives. It lives beside `child` because it describes that channel and nothing else: a + * session replacement spawns a new channel and must re-negotiate rather than inherit, or a + * downgraded control plane would keep receiving frames its wire cannot carry. + */ + protocol?: number; /** * The last assistant message's `usage.cost.total`. * @@ -169,7 +189,11 @@ export default function (pi: ExtensionAPI) { const closeChild = (child: childProcess.ChildProcess | undefined) => { if (!child) return; - if (state.child === child) state.child = undefined; + if (state.child === child) { + state.child = undefined; + // The agreement belonged to that channel. A successor re-negotiates from its own hello. + state.protocol = undefined; + } // The channel treats EOF on its stdin as the session boundary, so ending this pipe is what // reaps it. That happens on its own whenever pi exits, however it exits. if (!child.stdin?.destroyed) child.stdin?.end(); @@ -211,6 +235,9 @@ export default function (pi: ExtensionAPI) { { stdio: ["pipe", "pipe", "inherit"], env: channelEnv }, ); state.child = child; + // A fresh channel has agreed nothing yet, and until its hello lands this asset sends only + // the frames every version carries. + state.protocol = undefined; return new Promise((resolve) => { let settled = false; @@ -221,16 +248,20 @@ export default function (pi: ExtensionAPI) { }; const timer = setTimeout(() => settle(""), HELLO_TIMEOUT_MS); timer.unref?.(); + // Retiring the channel retires its agreement with it. + const retire = () => { + if (state.child !== child) return; + state.child = undefined; + state.protocol = undefined; + }; child.on("error", () => { - if (state.child === child) state.child = undefined; + retire(); settle(""); }); // An observability pipe must never take pi down: a channel that closed its stdin mid-write // surfaces EPIPE on the stream, which without a listener is an uncaught exception in the // host process. Retire the channel instead — frames simply stop, fail-open. - child.stdin.on("error", () => { - if (state.child === child) state.child = undefined; - }); + child.stdin.on("error", retire); child.on("exit", () => settle("")); const send = (frame: Record) => { @@ -246,17 +277,34 @@ export default function (pi: ExtensionAPI) { return; } if (frame.type === "hello") { - // A newer control plane may speak a wire this asset was not written against. Refusing is - // the honest outcome: presence still decays, so the agent reads as unreachable rather - // than silently never receiving mail. - if (frame.protocol !== PROTOCOL) { + // Negotiate rather than compare. The hello's scalar `protocol` is a FLOOR that never + // moves — an already-published asset compares it for equality — so a control plane that + // speaks a newer wire advertises the set in `protocols` and this asset picks the highest + // it also speaks. An empty intersection is refused, which is the honest outcome: + // presence still decays, so the agent reads as unreachable rather than silently never + // receiving mail. + const offered = Array.isArray(frame.protocols) + ? frame.protocols.filter((value): value is number => typeof value === "number") + : typeof frame.protocol === "number" + ? [frame.protocol] + : []; + const agreed = offered + .filter((value) => (SUPPORTED as readonly number[]).includes(value)) + .reduce( + (best, value) => (best === undefined || value > best ? value : best), + undefined, + ); + if (agreed === undefined) { closeChild(child); ctx.ui?.notify?.( - `st2: pi channel protocol ${frame.protocol} is not understood by this extension (expected ${PROTOCOL}); reinstall st2's hook set`, + `st2: pi channel offers protocol ${offered.join(", ") || "none"}, which this extension does not speak (it speaks ${SUPPORTED.join(", ")}); reinstall st2's hook set`, ); settle(""); return; } + // Recorded against the channel this hello came from, never "whatever is current": a + // predecessor draining its queued hello must not re-version the successor's channel. + if (state.child === child) state.protocol = agreed; clearTimeout(timer); settle(typeof frame.sessionContext === "string" ? frame.sessionContext : ""); return; @@ -302,7 +350,7 @@ export default function (pi: ExtensionAPI) { // starts exactly at that boundary — an `agent_end` emit would blip a spurious idle before it. // `agent_settled` is the first point pi is provably idle. The frame is observational — st2 // decides what becomes of it — and a closed channel drops it silently, matching the fail-open - // rule this file already follows. pi 0.84.2 exposes no typed waiting-on-a-human event, so no + // rule this file already follows. pi 0.84.4 exposes no typed waiting-on-a-human event, so no // frame here ever claims one. const sendFrame = (frame: Record) => { const child = state.child; @@ -311,6 +359,71 @@ export default function (pi: ExtensionAPI) { }; const sendState = (word: "active" | "idle") => sendFrame({ type: "state", state: word }); + /** + * The condition axis, extension side (protocol 2 only). + * + * A condition frame states the fault axis and NOTHING ELSE: pi's fault evidence (`agent_end`) + * carries no activity claim, and folding it onto a state frame would fabricate one and refresh + * a stale activity from an event that observed none. The converse holds too — a state frame + * never touches the condition axis — so a standing fault survives every activity edge and the + * record settles as `idle` beside it: activity honest, wedged seat visible. That pairing is the + * whole fix for the measured false idle, where `agent_settled` fires from a `finally` after a + * failed turn and published a clean idle for a wedged seat. + * + * Gated on the negotiated version, not on the code being installed: an st2 that speaks only + * protocol 1 has nowhere to put this frame, and sending it anyway would put an unreadable line + * on a wire that is otherwise exactly the one it shipped with. + */ + const sendCondition = (op: Record) => { + if ((state.protocol ?? 0) < CONDITION_PROTOCOL) return; + sendFrame({ type: "condition", ...op }); + }; + + /** + * pi's own verdict on the run that just ended, read off the LAST assistant message in + * `agent_end`'s `messages` array — measured against the published tarball's own declarations + * (`AgentEndEvent { type: "agent_end"; messages: AgentMessage[] }`, 0.84.2 + * dist/core/extensions/types.d.ts:542-544 and 0.84.4 :555-558). There is no `message` singular + * on this event; reading one returned `undefined` for every real end, which is silently the + * worst possible outcome — no raise, no clear, and the false idle unfixed. + * + * The classification compares pi's typed `stopReason` against its own closed vocabulary + * (`pending | stop | length | toolUse | error | aborted | deferred`, pi-ai + * 0.84.4 dist/types.d.ts:287, 0.84.2 :277) and never reads prose. `errorMessage` rides along + * as DIAGNOSTIC detail only: nothing branches on it and no category is inferred from it. + * + * Three answers, because two would have to lie. `failed` is an error-ended run. `completed` is + * an ordinary end — `stop` or `length`, the two words that prove the provider answered and the + * run finished — and it is the only positive success edge. `undefined` covers everything else: + * no assistant message at all, an `aborted` run (a person interrupted it, which proves neither + * health nor fault), and `pending`/`deferred`/`toolUse`/any future word. Those emit NO frame: + * a raise would invent a failure and a clear would silence a fault nobody saw resolve. + */ + const runOutcome = ( + messages: readonly unknown[], + ): { state: "failed" | "completed"; detail?: string } | undefined => { + for (let index = messages.length - 1; index >= 0; index -= 1) { + const message = messages[index]; + if (!message || typeof message !== "object") continue; + if (!("role" in message) || message.role !== "assistant") continue; + if (!("stopReason" in message) || typeof message.stopReason !== "string") return undefined; + if (message.stopReason === "error") { + const detail = + "errorMessage" in message && + typeof message.errorMessage === "string" && + message.errorMessage.trim() !== "" + ? message.errorMessage + : undefined; + return { state: "failed", detail }; + } + if (message.stopReason === "stop" || message.stopReason === "length") { + return { state: "completed" }; + } + return undefined; + } + return undefined; + }; + // Harness context, extension side (HC-R02, HC-R03, HC-R11, HC-R12). // // `ctx.getContextUsage()` answers the whole fill triple in one call and rides the ctx of every @@ -417,31 +530,107 @@ export default function (pi: ExtensionAPI) { sendContext(ctx); }); - const onContextEvent = (name: "message_end" | "turn_end" | "agent_end") => + // A helper for the events pi types loosely; the cast is on the REGISTRATION function, not on an + // event payload, and every payload read below narrows. + const onEvent = ( + name: string, + handler: (event: unknown, ctx: ExtensionContext) => void | Promise, + ) => (pi.on as unknown as ( event: string, handler: (event: unknown, ctx: ExtensionContext) => void | Promise, - ) => void)(name, async (event, ctx) => { + ) => void)(name, handler); + + for (const name of ["message_end", "turn_end"]) { + onEvent(name, async (event, ctx) => { captureCost(event); sendContext(ctx); }); - onContextEvent("message_end"); - onContextEvent("turn_end"); - onContextEvent("agent_end"); + } + + // `agent_end` is where a pi run's OUTCOME becomes visible, and until now this handler read only + // the numbers off it. That outcome is the entire cause of the measured false idle: + // `_emitAgentSettled` runs from the `finally` of `_runAgentPrompt` and fires unconditionally + // after a failed run, so a wedged seat published a clean `idle` with no reason at all. + // Classifying here — before `agent_settled`, which is the measured order — is what makes the + // fault land first and the idle land honestly beside it. + // + // Registered through pi's TYPED overload, deliberately: `event.messages` is then checked + // against the pinned tarball's own `AgentEndEvent`, which is exactly the check the untyped + // registration cast defeated while this handler read a `message` field that does not exist. + pi.on("agent_end", async (event, ctx) => { + captureCost(event); + sendContext(ctx); + const outcome = runOutcome(event.messages ?? []); + if (!outcome) return; + if (outcome.state === "failed") { + // `harness`, not `authentication`/`quota`/`provider`: the typed evidence is "a pi run + // failed", nothing more. pi ships no error-classification field (omp's `errorId` bitfield + // is exactly what pi lacks), so any narrower category would be inferred from prose. The + // prose rides `detail`, diagnostic-only. `unknown` recovery because pi says nothing about + // who clears this — never optimistic, so it pages. + sendCondition({ + op: "raise", + category: "harness", + code: ASSISTANT_ERROR, + recovery: "unknown", + ...(outcome.detail === undefined ? {} : { detail: outcome.detail }), + }); + return; + } + // pi's ONLY positive success edge: a run that reached its ordinary end proves the provider + // accepted the credential and the work ran. `agent_settled` is not this edge — it fires from + // a `finally` after a failure too — which is why the unkeyed clear hangs here alone. + sendCondition({ op: "clearAll", proof: "turnCompleted" }); + }); // pi's `session_compact` carries `reason ∈ manual | threshold | overflow` — the only v1 producer // that names its trigger at all. Measured in the handler itself: `getContextUsage()` already // reports `{tokens: null, percent: null}` there, and `getEntries()` already counts the new // entry, so this one frame carries both the honest withheld reading and the durable count. - (pi.on as unknown as ( - event: string, - handler: (event: unknown, ctx: ExtensionContext) => void | Promise, - ) => void)("session_compact", async (event, ctx) => { - const reason = (event as { reason?: unknown })?.reason; + onEvent("session_compact", async (event, ctx) => { + const reason = event && typeof event === "object" && "reason" in event ? event.reason : null; sendContext(ctx, { trigger: typeof reason === "string" ? reason : null, count: durableCompactions(ctx), }); + // A compaction that succeeded is the paired clear for its own failure, keyed on the EXACT + // category and code. A category-only clear would also wipe any other `context` fault, and + // there is no fallback to an unkeyed clear: on a healthy seat this ordinarily matches nothing + // and st2 logs that at debug. + sendCondition({ op: "clear", category: "context", code: COMPACT_FAILED }); + }); + + // pi-only, and typed: `session_compact_failed` is the failure sibling of the event above, so + // this is a real signal rather than an inference from prose. `context`, because the harness has + // no usable window left and could not reclaim any; `human`, because nothing in pi retries it. + // No paired clear is derived from anything else — only a later successful compaction retires + // it. + // + // Two version facts ride on this handler. The event exists from pi 0.84.3 (zero occurrences in + // the 0.84.2 tarball, declared in 0.84.4 dist/core/extensions/types.d.ts:464-476), and + // registering a name a running build never emits is harmless — pi's `on` is a plain map insert + // (dist/core/extensions/loader.js:209-213) — so on an older build this is simply inert rather + // than a fault this seat cannot report. And the event's own `aborted` flag is load-bearing: a + // cancelled `/compact` is a person changing their mind, not a harness that ran out of context, + // so only `aborted === false` is a fault. The flag is read POSITIVELY — a build that does not + // carry it states nothing, and an unreadable flag must not become a raise. + onEvent("session_compact_failed", async (event) => { + if (!event || typeof event !== "object") return; + if (!("aborted" in event) || event.aborted !== false) return; + const detail = + "errorMessage" in event && + typeof event.errorMessage === "string" && + event.errorMessage.trim() !== "" + ? event.errorMessage + : undefined; + sendCondition({ + op: "raise", + category: "context", + code: COMPACT_FAILED, + recovery: "human", + ...(detail === undefined ? {} : { detail }), + }); }); pi.on("session_start", async (_event, ctx) => { diff --git a/hooks/typecheck/omp-smoke.mjs b/hooks/typecheck/omp-smoke.mjs index 24ac9837..476a59c4 100644 --- a/hooks/typecheck/omp-smoke.mjs +++ b/hooks/typecheck/omp-smoke.mjs @@ -16,7 +16,17 @@ fs.writeFileSync( recorder, `#!${process.execPath} import fs from "node:fs"; -process.stdout.write(JSON.stringify({ type: "hello", protocol: 1, sessionContext: "" }) + "\\n"); +const offer = process.env.ST2_SMOKE_PROTOCOLS; +process.stdout.write( + JSON.stringify({ + type: "hello", + protocol: 1, + sessionContext: "", + // st2's hello keeps \`protocol: 1\` forever — the asset refuses anything else, and a refusal + // costs the seat its mail — and offers the versions it would also accept beside it. + ...(offer ? { protocols: JSON.parse(offer) } : {}), + }) + "\\n", +); process.stdin.setEncoding("utf8"); process.stdin.on("data", (chunk) => fs.appendFileSync(${JSON.stringify(framesPath)}, chunk)); `, @@ -290,6 +300,171 @@ assert.strictEqual(durable.compaction.count, 1, "the count is getEntries() filte // Unlike pi, omp still answers inside its own compact handler, so a real reading rides the edge. assert.strictEqual(durable.reading.usedTokens, 22500, "omp does not null its reading at the edge"); +// ─── Negotiation ──────────────────────────────────────────────────────────────────────────────── +// Everything above ran against a peer whose hello offered nothing, and that is the load-bearing +// half: the version-1 wire must be byte-identical to the one that shipped. No answer, no +// conversation statement, and no prose on an unblocked state frame reached it. +assert.deepStrictEqual( + readFrames().filter((frame) => frame.type === "client_hello"), + [], + "a hello that offers nothing is never answered", +); +assert.deepStrictEqual( + readFrames().filter((frame) => frame.type === "conversation"), + [], + "the conversation axis is stated only to a negotiated peer", +); +assert.ok( + readFrames().every( + (frame) => frame.type !== "state" || !("reason" in frame) || frame.blockedOn === "human", + ), + "only a blocked frame carries prose on the version-1 wire", +); +// And nothing on either wire is a `condition` frame. omp has no asset-side condition signal at +// all: every fault it can prove rides the typed `turn` frame st2 already decodes, so a condition +// frame from this asset would be a claim no capture supports. +assert.deepStrictEqual( + readFrames().filter((frame) => frame.type === "condition"), + [], + "this asset states no conditions in any protocol", +); + +// An offer this asset is not in is not an agreement. +process.env.ST2_SMOKE_PROTOCOLS = "[1]"; +let before = readFrames().length; +await handlers.get("session_start")({}, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 100)); +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: "sess-1" }, activeCtx); +await handlers.get("tool_approval_resolved")({ approved: false, sessionId: "sess-1" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +assert.deepStrictEqual( + readFrames().slice(before).filter((frame) => frame.type !== "context"), + [ + { type: "state", state: "active" }, + { type: "state", state: "active", blockedOn: "human", ask: "permission", reason: "bash" }, + { type: "state", state: "active" }, + ], + "an offer without version 2 keeps the legacy wire: no answer, no session id, no denial prose", +); + +// The negotiated peer. +process.env.ST2_SMOKE_PROTOCOLS = "[1,2]"; +before = readFrames().length; +await handlers.get("session_start")({}, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 100)); +assert.deepStrictEqual( + readFrames().slice(before)[0], + { type: "client_hello", protocol: 2 }, + "the answer is this asset's FIRST write on the connection, before any observation", +); + +// omp's own `sessionId` rides both halves of the approval pair (measured 18.0.9 and 18.1.2). It +// is stated once per channel, and a denied approval is an interruption whose word is prose. +before = readFrames().length; +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: " sess-7 " }, activeCtx); +await handlers.get("tool_approval_resolved")({ approved: false, sessionId: "sess-7" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +assert.deepStrictEqual( + readFrames().slice(before), + [ + { type: "conversation", sessionId: "sess-7" }, + { type: "state", state: "active", blockedOn: "human", ask: "permission", reason: "bash" }, + { type: "state", state: "active", reason: "approvalDenied" }, + ], + "a negotiated peer receives the session id once and the denial as prose", +); + +before = readFrames().length; +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: "sess-7" }, activeCtx); +await handlers.get("tool_approval_resolved")({ approved: true, sessionId: "sess-7" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +assert.deepStrictEqual( + readFrames().slice(before).filter((frame) => frame.type === "conversation"), + [], + "the same session id is stated once per channel, not once per event", +); +assert.deepStrictEqual( + readFrames().slice(before).at(-1), + { type: "state", state: "active" }, + "a granted approval carries no prose", +); + +// The ask outranks the approval surface, and a DENIED ask emits no `tool_result` at all +// (DQ-OMP-1) — so only a turn boundary can retire it. Without that rule one denial mutes the +// approval surface, and st2's positive `none` on the ask axis, for the whole process lifetime. +before = readFrames().length; +await handlers.get("tool_call")( + { + toolName: "ask", + toolCallId: "ask-2", + input: { questions: [{ id: "go", question: "Proceed?" }] }, + }, + activeCtx, +); +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: "sess-7" }, activeCtx); +await handlers.get("tool_approval_resolved")({ approved: true, sessionId: "sess-7" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +assert.deepStrictEqual( + readFrames().slice(before), + [ + { + type: "state", + state: "active", + blockedOn: "human", + ask: "question", + reason: "Proceed?", + }, + ], + "a pending ask suppresses both approval halves", +); + +before = readFrames().length; +await handlers.get("agent_start")({}, activeCtx); +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: "sess-7" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +assert.deepStrictEqual( + readFrames().slice(before).filter((frame) => frame.type === "state"), + [ + { type: "state", state: "active" }, + { type: "state", state: "active", blockedOn: "human", ask: "permission", reason: "bash" }, + ], + "agent_start retires a never-answered ask", +); + +before = readFrames().length; +await handlers.get("tool_call")( + { + toolName: "ask", + toolCallId: "ask-3", + input: { questions: [{ id: "go", question: "Again?" }] }, + }, + activeCtx, +); +await handlers.get("agent_end")({ messages: [{ role: "assistant", stopReason: "stop" }] }, activeCtx); +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: "sess-7" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +assert.deepStrictEqual( + readFrames().slice(before).filter((frame) => frame.type === "state"), + [ + { type: "state", state: "active", blockedOn: "human", ask: "question", reason: "Again?" }, + { type: "state", state: "active", blockedOn: "human", ask: "permission", reason: "bash" }, + ], + "agent_end retires a never-answered ask too", +); +// A replacement channel negotiates for itself and re-states the session id: the stash outlives +// the connection, the agreement must not. +before = readFrames().length; +await handlers.get("session_start")({}, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 100)); +await handlers.get("tool_approval_requested")({ toolName: "bash", sessionId: "sess-7" }, activeCtx); +await new Promise((resolve) => setTimeout(resolve, 50)); +const reopened = readFrames().slice(before); +assert.deepStrictEqual(reopened[0], { type: "client_hello", protocol: 2 }); +assert.ok( + reopened.some((frame) => frame.type === "conversation" && frame.sessionId === "sess-7"), + "a fresh connection re-states the conversation identity", +); + fs.rmSync(dir, { recursive: true, force: true }); console.log("omp extension smoke: ok"); process.exit(0); diff --git a/hooks/typecheck/smoke.mjs b/hooks/typecheck/smoke.mjs index 80533534..8d429e83 100644 --- a/hooks/typecheck/smoke.mjs +++ b/hooks/typecheck/smoke.mjs @@ -13,8 +13,13 @@ import path from "node:path"; // producer that silently writes nothing is indistinguishable from the pre-producer state, where // every declaration's context reads null. That is the failure this file has to be able to see. // -// It speaks the two things the extension needs: a protocol-1 hello so `open()` settles without -// waiting out its timeout, and an append of every frame line to a file this smoke reads back. +// It speaks the two things the extension needs: a hello it can negotiate against — the offer is +// selected by `ST2_SMOKE_HELLO`, so one recorder can stand in for this build's control plane, for +// one that never learned protocol 2, and for one that speaks nothing this asset does — and an +// append of every frame line to the file named by `ST2_SMOKE_FRAMES`. Both ride the environment +// rather than being baked in, because the channel is spawned fresh on every `session_start` and +// inherits this process's environment at that moment, which is what lets one loaded extension be +// driven against several control planes. const dir = fs.mkdtempSync(path.join(os.tmpdir(), "st2-pi-smoke-")); const framesPath = path.join(dir, "frames.jsonl"); const recorder = path.join(dir, "recorder"); @@ -22,14 +27,24 @@ fs.writeFileSync( recorder, `#!${process.execPath} import fs from "node:fs"; -process.stdout.write(JSON.stringify({ type: "hello", protocol: 1, sessionContext: "" }) + "\\n"); +const offers = { + // What this build's hello carries: the floor a published asset compares for strict equality, + // beside the set a newer asset negotiates over. + negotiated: { protocol: 1, protocols: [1, 2] }, + // A control plane that never learned protocol 2 — the bytes this channel shipped with. + legacy: { protocol: 1 }, + // A wire this asset does not speak at all. + foreign: { protocol: 9, protocols: [9] }, +}; +const offer = offers[process.env.ST2_SMOKE_HELLO ?? "negotiated"]; +process.stdout.write(JSON.stringify({ type: "hello", ...offer, sessionContext: "" }) + "\\n"); process.stdin.setEncoding("utf8"); -process.stdin.on("data", (chunk) => fs.appendFileSync(${JSON.stringify(framesPath)}, chunk)); +process.stdin.on("data", (chunk) => fs.appendFileSync(process.env.ST2_SMOKE_FRAMES, chunk)); `, { mode: 0o755 }, ); -const readFrames = () => - (fs.existsSync(framesPath) ? fs.readFileSync(framesPath, "utf8") : "") +const readFrames = (file = framesPath) => + (fs.existsSync(file) ? fs.readFileSync(file, "utf8") : "") .split("\n") .filter((line) => line.trim()) .map((line) => JSON.parse(line)); @@ -40,6 +55,9 @@ process.env.ST2_PI_CHANNEL_IDENTITY = "smoke.worker"; process.env.ST2_PI_CHANNEL_RUNTIME_ID = "smoke.worker"; process.env.ST2_PI_CHANNEL_SESSION = "smoke-session"; process.env.ST2_PI_CHANNEL_SEQ = "1"; +// Read by the recorder, not by the extension: the channel inherits this environment when the +// extension spawns it, so a phase below can redirect the sink and the hello together. +process.env.ST2_SMOKE_FRAMES = framesPath; const mod = await import("./smoke-out/pi-channel.mjs"); assert.strictEqual(typeof mod.default, "function", "extension exports its entry point"); @@ -53,8 +71,15 @@ for (const name of ["session_start", "session_shutdown", "agent_start", "agent_s assert.ok(handlers.has(name), `extension registers ${name}`); } -// The harness-context producer must be registered on the events it reads, or it observes nothing. -for (const name of ["message_end", "turn_end", "agent_end", "session_compact"]) { +// The harness-context producer must be registered on the events it reads, or it observes nothing; +// `session_compact_failed` is the fault axis's only pi-specific source. +for (const name of [ + "message_end", + "turn_end", + "agent_end", + "session_compact", + "session_compact_failed", +]) { assert.ok(handlers.has(name), `extension registers ${name}`); } @@ -109,6 +134,44 @@ const messageEvent = { usage: { input: 23300, output: 25, totalTokens: 23425, cost: { total: 0.070305 } }, }, }; +// The `agent_end` payloads pi ACTUALLY emits: `{type:"agent_end", messages: AgentMessage[]}` +// (0.84.2 dist/core/extensions/types.d.ts:542-544, 0.84.4 :555-558), with `stopReason` drawn from +// pi's own closed vocabulary `pending | stop | length | toolUse | error | aborted | deferred` +// (pi-ai 0.84.4 dist/types.d.ts:287, 0.84.2 :277). These fixtures are the reason this smoke can +// see the defect class it exists for: a producer reading a `message` singular, or comparing +// against a stop word pi never emits, classifies nothing at all — no raise, no clear, false idle. +// +// Each run's tail is the LAST assistant message, so every fixture carries a tool result after a +// tool-use assistant message: a producer scanning forwards, or reading `messages[0]`, gets the +// wrong verdict here rather than passing by luck. +const toolLeg = [ + { role: "assistant", stopReason: "toolUse", usage: { totalTokens: 100 } }, + { role: "toolResult", toolName: "bash", content: [] }, +]; +const failedRun = { + type: "agent_end", + messages: [ + ...toolLeg, + { role: "assistant", stopReason: "error", errorMessage: "401 Unauthorized from fakelab" }, + ], +}; +const completedRun = { + type: "agent_end", + messages: [...toolLeg, { role: "assistant", stopReason: "stop" }], +}; +// A person pressed escape. Neither health nor fault, so neither a raise nor a clear. +const abortedRun = { + type: "agent_end", + messages: [...toolLeg, { role: "assistant", stopReason: "aborted" }], +}; +// A run whose last assistant message never resolved: also no verdict. +const pendingRun = { + type: "agent_end", + messages: [{ role: "assistant", stopReason: "pending" }], +}; +// How many times the loop below drives one full turn sequence — the exact count of raises, clears, +// and clearAlls the negotiated phase must produce. +const drives = 3; for (const ctx of [bareCtx, fullCtx, throwingCtx]) { // Two session starts in a row: the second exercises the predecessor close-and-await path — the @@ -118,7 +181,24 @@ for (const ctx of [bareCtx, fullCtx, throwingCtx]) { await handlers.get("agent_start")({}, ctx); await handlers.get("message_end")(messageEvent, ctx); await handlers.get("turn_end")(messageEvent, ctx); - await handlers.get("agent_end")({}, ctx); + // Every `agent_end` shape pi produces. Only the failed and the completed run say anything + // about the condition axis; a run with no messages at all, an aborted one, and one whose tail + // never resolved each emit nothing. + await handlers.get("agent_end")({ type: "agent_end", messages: [] }, ctx); + await handlers.get("agent_end")(failedRun, ctx); + await handlers.get("agent_end")(completedRun, ctx); + await handlers.get("agent_end")(abortedRun, ctx); + await handlers.get("agent_end")(pendingRun, ctx); + // A real compaction failure, and a cancelled one: `aborted` is what separates a harness out of + // context from a person changing their mind, and only the former is a fault. + await handlers.get("session_compact_failed")( + { reason: "overflow", aborted: false, errorMessage: "summarizer returned no content", willRetry: false, fromExtension: false }, + ctx, + ); + await handlers.get("session_compact_failed")( + { reason: "manual", aborted: true, willRetry: false, fromExtension: false }, + ctx, + ); // pi withholds tokens and percent here for real; the producer must forward that, not fill it in. await handlers.get("session_compact")( { reason: "overflow", willRetry: false }, @@ -183,6 +263,182 @@ assert.strictEqual(durable.reading.usedTokens, null, "the withheld reading rides assert.strictEqual(durable.reading.usedPercent, null); assert.strictEqual(durable.reading.windowTokens, 4000, "pi still knows its denominator"); +// The condition axis, on the wire. Everything below is the only place these emitters actually +// execute, so nothing else couples the words this asset writes to the words +// `src/pi_channel.rs::condition_frame` decodes. +const conditions = frames.filter((frame) => frame.type === "condition"); +// Told apart from a mapping defect deliberately. Every condition assertion below is an exact +// count, and there are two ways to reach zero: the classification is broken, or the recorder's +// hello never arrived inside the asset's 5 s window on a loaded machine and no channel ever +// negotiated protocol 2. The second is this harness's problem, not the asset's, and it must not +// masquerade as the first. +assert.ok( + conditions.length > 0 || frames.length === 0, + "frames arrived but no condition frame did: either the classification is broken or no channel " + + "negotiated protocol 2 (a hello that timed out leaves every condition frame gated off)", +); +// A condition frame states the fault axis and NOTHING else. An activity claim here would +// fabricate one from an event that observed none, and it is what made a wedged seat publish a +// clean idle. +for (const condition of conditions) { + assert.ok(!("state" in condition), "a condition frame never restates activity"); + assert.ok(!("ask" in condition), "a condition frame never claims an ask"); + assert.notStrictEqual(condition.op, "ended", "no frame from this asset can end a session"); +} + +const raises = conditions.filter((frame) => frame.op === "raise"); +const assistantErrors = raises.filter((frame) => frame.code === "pi/assistantError"); +assert.strictEqual( + assistantErrors.length, + drives, + "an error-ended assistant tail raises exactly one fault", +); +for (const raise of assistantErrors) { + // The teeth: prose that says `401` still yields `harness`. pi ships no error-classification + // field, so `authentication` here would be inferred from the very string carried as detail. + assert.strictEqual(raise.category, "harness", "pi's untyped turn failure is a harness fault"); + assert.strictEqual(raise.recovery, "unknown", "pi says nothing about who clears it"); + assert.strictEqual( + raise.detail, + "401 Unauthorized from fakelab", + "the prose is diagnostic only", + ); +} + +const clearAlls = conditions.filter((frame) => frame.op === "clearAll"); +assert.strictEqual(clearAlls.length, drives, "a clean agent_end is pi's only unkeyed clear"); +for (const clear of clearAlls) { + assert.strictEqual(clear.proof, "turnCompleted", "an unkeyed clear names the progress it saw"); +} + +const compactFailures = raises.filter((frame) => frame.code === "pi/session_compact_failed"); +assert.strictEqual( + compactFailures.length, + drives, + "only the non-aborted compaction failure raises; a cancelled /compact is not a fault", +); +for (const raise of compactFailures) { + assert.strictEqual(raise.category, "context"); + assert.strictEqual(raise.recovery, "human", "nothing in pi retries a failed compaction"); + assert.strictEqual( + raise.detail, + "summarizer returned no content", + "the typed event's own errorMessage rides as detail", + ); +} + +const paired = conditions.filter((frame) => frame.op === "clear"); +assert.strictEqual(paired.length, drives, "a successful compaction clears its own failure once"); +for (const clear of paired) { + assert.strictEqual(clear.category, "context"); + // Keyed on the FULL code, never the category alone: a category-only clear would also wipe any + // other context fault standing on the seat. + assert.strictEqual(clear.code, "pi/session_compact_failed"); +} + +// Four condition frames per drive and not a fifth. Everything else driven above is deliberately +// silent: an `agent_end` with no messages, an aborted run, a run whose tail is still pending, and +// a cancelled compaction each emit nothing in either direction. +assert.strictEqual( + conditions.length, + 4 * drives, + "only a failed run, a completed run, a real compaction failure, and a compaction success speak", +); + +// And the idle edge is untouched: `agent_settled` still sends its plain state frame, which is the +// honest activity beside a standing fault rather than a suppressed one. +const idles = frames.filter((frame) => frame.type === "state" && frame.state === "idle"); +assert.ok(idles.length >= drives, "agent_settled still publishes a plain idle"); +for (const idle of idles) { + assert.deepStrictEqual( + Object.keys(idle).sort(), + ["state", "type"], + "pi's state frame carries no ask and no condition", + ); +} + +// The whole point of raising from `agent_end`: it is measured to fire BEFORE `agent_settled`, so +// the fault lands first and the idle that follows carries it forward instead of publishing a +// clean yield for a wedged seat. +const firstRaise = frames.findIndex( + (frame) => frame.type === "condition" && frame.code === "pi/assistantError", +); +const idleAfterRaise = frames.findIndex( + (frame, index) => index > firstRaise && frame.type === "state" && frame.state === "idle", +); +assert.ok(firstRaise >= 0, "the error-tailed turn must raise"); +assert.ok( + idleAfterRaise > firstRaise, + "the standing fault must precede the idle it is published beside", +); + +// The state vocabulary stays closed to what pi's own turn boundaries prove. `ended` is the outer +// session wrapper's word — it alone sees the provider die — and no frame from this asset may +// claim it on any axis. +for (const frame of frames.filter((frame) => frame.type === "state")) { + assert.ok( + frame.state === "active" || frame.state === "idle", + `a state frame states only active or idle: ${JSON.stringify(frame)}`, + ); +} +assert.strictEqual( + frames.filter((frame) => frame.state === "ended" || "exit" in frame).length, + 0, + "the channel's asset never writes a terminal record", +); + +// Phase 2: a control plane that never learned protocol 2. Every frame this asset sends must be +// one that wire can carry — the condition frames are gated on the negotiated version, not on the +// code being installed. +const legacyFrames = path.join(dir, "frames-legacy.jsonl"); +process.env.ST2_SMOKE_HELLO = "legacy"; +process.env.ST2_SMOKE_FRAMES = legacyFrames; +await handlers.get("session_start")({}, fullCtx); +await handlers.get("agent_start")({}, fullCtx); +await handlers.get("agent_end")(failedRun, fullCtx); +await handlers.get("agent_end")(completedRun, fullCtx); +await handlers.get("session_compact_failed")( + { reason: "overflow", aborted: false, willRetry: false, fromExtension: false }, + fullCtx, +); +await handlers.get("agent_settled")({}, fullCtx); +await new Promise((resolve) => setTimeout(resolve, 500)); +const legacy = readFrames(legacyFrames); +assert.ok(legacy.length > 0, "a protocol-1 control plane still receives the frames it understands"); +assert.strictEqual( + legacy.filter((frame) => frame.type === "condition").length, + 0, + "no condition frame may reach a wire that cannot carry one", +); +assert.ok( + legacy.some((frame) => frame.type === "state" && frame.state === "idle"), + "protocol 1 keeps its exact existing state frames", +); +assert.ok( + legacy.some((frame) => frame.type === "context"), + "protocol 1 keeps its exact existing context frames", +); + +// Phase 3: a control plane offering nothing this asset speaks. Refusing is the honest outcome — +// presence decays and the seat reads as unreachable — and it must be a NOTIFIED refusal, not a +// silent one. +const notices = []; +const foreignCtx = Object.create(fullCtx, { + ui: { value: { notify: (message) => notices.push(message) } }, +}); +const foreignFrames = path.join(dir, "frames-foreign.jsonl"); +process.env.ST2_SMOKE_HELLO = "foreign"; +process.env.ST2_SMOKE_FRAMES = foreignFrames; +await handlers.get("session_start")({}, foreignCtx); +await handlers.get("agent_start")({}, foreignCtx); +await handlers.get("agent_settled")({}, foreignCtx); +await new Promise((resolve) => setTimeout(resolve, 500)); +assert.strictEqual(readFrames(foreignFrames).length, 0, "a refused channel receives no frames"); +assert.ok( + notices.some((notice) => notice.includes("protocol")), + "a refused negotiation tells the operator why", +); + fs.rmSync(dir, { recursive: true, force: true }); console.log("pi extension smoke: ok"); process.exit(0); diff --git a/src/claude_session.rs b/src/claude_session.rs index fa198997..2d6b5784 100644 --- a/src/claude_session.rs +++ b/src/claude_session.rs @@ -16,7 +16,11 @@ use std::path::Path; use anyhow::{Context as _, Result}; use crate::harness_context::{self, Compaction, CompactionTrigger, Harness, RateLimits, Reading}; -use crate::harness_state::{Activity, Ask, BlockedOn, InputBuffer, Observation}; +use crate::harness_state::{ + Activity, Ask, AskKind, BlockedOn, CapabilityEvidence, ConditionReport, ConversationClaim, + ConversationState, FaultCategory, FaultReport, Frame, HistoryMutability, HumanAsk, InputBuffer, + Observation, Recovery, WriteOutcome, +}; use crate::provider_session::{ PROVIDER_POLL, STOP, SessionObserver, install_signal_handler, run_provider, }; @@ -180,7 +184,7 @@ pub fn run_observe( if let Some(edge) = provider_auth_edge(event, &payload) { publish_provider_auth(&agent_dir, edge); } - let Some(observation) = observe_hook_event(event, &payload) else { + let Some(observed) = observe_hook_event(event, &payload) else { return Ok(()); }; let mut writer = observe_writer( @@ -201,17 +205,44 @@ pub fn run_observe( } // A late hook finishing after the wrapper reaped Claude must not replace the terminal record // with a live state: the wrapper's `ended` carries this same token and is the session's last - // word. (`false` = suppressed; the hook has nothing else to do with it.) - writer.observe_unless_ended(observation).map(|_wrote| ()) + // word — on both surfaces below. + if !writer.writes_condition_axis() { + // The version this build emits carries no condition, tagged-ask, or conversation axis, + // and this record has exactly ONE source of truth: state the legacy triple, exactly as + // this adapter always has. Keeping a stated fault in a sidecar the session's sibling + // processes cannot see would be that second source. Activation is one flip of the + // writer's emitted version, after which the branch below takes over unchanged. + // (`false` = suppressed; the hook has nothing else to do with it.) + return writer + .observe_unless_ended(observed.observation()) + .map(|_wrote| ()); + } + match writer.publish_unless_ended(observed.frame())? { + // Written or already-said are both "the record now says this". + WriteOutcome::Landed | WriteOutcome::Coalesced => Ok(()), + // A refusal is a VALUE here — a successor session's claim, a foreign schema, a condition + // axis nobody has stated yet, this session's own terminal record — and a hook process has + // no second record to report it through, so it is diagnosed rather than swallowed. Still + // fail-open: the harness is waiting on this hook, and no observation is worth wedging it. + WriteOutcome::Refused(refusal) => { + tracing::warn!( + "st2 claude-observe: observed-state write refused for {event}: {refusal:?}" + ); + Ok(()) + } + } } /// Select the ownership a hook write acts under. The wrapper's exported token makes hook writes /// this session's records (adopted ownership when the claimed sequence travels beside it). A /// wrapperless seat falls back to Claude's own session_id — and because token-only writers never -/// claim, the SessionStart arm IS that path's session boundary and performs the WRITTEN claim +/// claim, the session-boundary arm IS that path's boundary and performs the WRITTEN claim /// (degrading to token-only with a warning if the claim cannot be written); later hooks of the /// same session adopt its records by token. What such a seat still lacks is a heartbeat and /// terminal owner — the documented hooks-only limitation. +/// +/// Which `SessionStart` is that boundary depends on what the emitted version can carry: see +/// [`claims_wrapperless_session`]. #[allow(clippy::too_many_arguments)] fn observe_writer( agent_dir: &Path, @@ -231,7 +262,7 @@ fn observe_writer( }; } if let Some(token) = wrapperless_token(payload) { - if event == "SessionStart" { + if claims_wrapperless_session(writer.writes_condition_axis(), event, payload) { // Eligibility and the written takeover are ONE act under the record lock: a // hooks-only SessionStart racing a wrapper's startup can no longer steal the // sequence between the wrapper's read and its write. Ineligible (a live wrapper or @@ -253,6 +284,28 @@ fn observe_writer( writer } +/// Whether this hook invocation is the WRAPPERLESS seat's session boundary — the one act that +/// performs the written claim. +/// +/// While the emitted version carries no condition axis, this is every `SessionStart`, exactly as +/// this adapter has always behaved: same claim, same sequence, same bytes. +/// +/// Once the condition axis is on the wire a claim is no longer neutral. A claim writes this +/// session's FENCE, and a fence is deliberately excluded from what an unstated axis inherits, so +/// claiming on a MID-PROCESS `SessionStart` — `compact`, `clear` — would drop a standing fault by +/// ownership: exactly the laundering the mapping refuses to do with `clear`, arriving through the +/// other door. Those events keep the token-only path, where the record already carries this +/// session's token, its sequence is adopted, and the standing condition carries. Only a genuinely +/// fresh incarnation claims, which is the one case where superseding the axis is the truth. +fn claims_wrapperless_session( + writes_condition_axis: bool, + event: &str, + payload: &serde_json::Value, +) -> bool { + event == "SessionStart" + && (!writes_condition_axis || session_start_is_fresh_incarnation(payload)) +} + /// Claude's own session id under the wrapperless prefix, for a seat that runs `claude` directly /// with no session wrapper to export a token. /// @@ -529,7 +582,119 @@ fn chain_statusline(raw: &[u8]) -> Result<()> { Ok(()) } -/// Map one Claude hook event to an observation, or `None` when the event says nothing about +/// The ask surface Claude can state, closed at three words. `PermissionRequest`'s `tool_name` is +/// the ONLY ask signal any registered hook carries and it distinguishes exactly two kinds, so +/// both projections below are total functions of one decision. Storing a [`HumanAsk`] here +/// instead would force this adapter to invent a legacy word for `Pending(Review)` and for +/// `Unknown` — kinds Claude has no signal for at all — and an invented projection is how the +/// legacy and tagged axes start disagreeing about the same prompt. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum HookAsk { + None, + Permission, + Question, +} + +impl HookAsk { + fn tagged(self) -> HumanAsk { + match self { + Self::None => HumanAsk::None, + Self::Permission => HumanAsk::Pending(AskKind::Permission), + Self::Question => HumanAsk::Pending(AskKind::Question), + } + } + + /// The legacy pair, verbatim as this adapter has always written it: an ask is meaningful only + /// while blocked on a human, which is exactly what the two pending kinds mean here. + fn legacy(self) -> (BlockedOn, Ask) { + match self { + Self::None => (BlockedOn::None, Ask::None), + Self::Permission => (BlockedOn::Human, Ask::Permission), + Self::Question => (BlockedOn::Human, Ask::Question), + } + } +} + +/// What one Claude hook event states about top-level harness state: every axis at once, decided +/// from one look at the payload, so no reader can observe the activity, the condition, the ask +/// and the conversation link half-applied. +/// +/// `condition: Unchanged` is the load-bearing default. Almost every Claude edge is an activity or +/// ask edge that has observed NOTHING about whether the provider is faulted, and a condition may +/// be cleared only by a positive success edge (`Stop`) or a new incarnation — never by a turn +/// starting, a tool running, or a compaction restart. Stating the carry here rather than reading +/// the standing condition first also keeps the decision inside the single lock the write holds: +/// check-then-act across two acquisitions is the race `claim_wrapperless` exists to avoid. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct HookState { + pub state: Activity, + pub ask: HookAsk, + pub condition: ConditionReport, + /// Stated only by the incarnation boundary and the turn-end edges — the ones that neither + /// repeat nor fire several times a turn — because the link's `verifiedThroughMs` is minted + /// from the hook's clock: restating it on a repeatable edge would make every repeat a + /// different tuple, landing a write and restarting `sinceMs`. `None` leaves the axis exactly + /// as the record has it, which is how the standing link carries. + pub conversation: Option, + /// Diagnostic only, and one of a closed set of static words. No consumer branches on it. + pub reason: Option<&'static str>, +} + +impl HookState { + fn new(state: Activity, ask: HookAsk, condition: ConditionReport) -> Self { + Self { + state, + ask, + condition, + conversation: None, + reason: None, + } + } + + fn with_reason(mut self, reason: &'static str) -> Self { + self.reason = Some(reason); + self + } + + fn with_conversation(mut self, conversation: Option) -> Self { + self.conversation = conversation; + self + } + + /// The legacy observation this event has always written — same activity, same legacy + /// blocked/ask pair, same reason — for the version this build still emits. The condition and + /// conversation axes are dropped rather than stored beside the record: the version 2 wire has + /// nowhere to carry them, and a sidecar would give this record a second source of truth. + fn observation(&self) -> Observation { + let (blocked_on, ask) = self.ask.legacy(); + let observation = + Observation::new(self.state, blocked_on, InputBuffer::Unknown).with_ask(ask); + match self.reason { + Some(reason) => observation.with_reason(reason), + None => observation, + } + } + + /// The version 3 tuple: one statement of every axis this event resolved. + fn frame(&self) -> Frame { + let frame = Frame::new( + self.state, + InputBuffer::Unknown, + self.condition.clone(), + self.ask.tagged(), + ); + let frame = match self.conversation.clone() { + Some(conversation) => frame.with_conversation(conversation), + None => frame, + }; + match self.reason { + Some(reason) => frame.with_reason(reason), + None => frame, + } + } +} + +/// Map one Claude hook event to the state it proves, or `None` when the event says nothing about /// top-level harness state. /// /// Claude gives no call identity on the event that enters `blocked` (`PermissionRequest` carries @@ -540,11 +705,30 @@ fn chain_statusline(raw: &[u8]) -> Result<()> { /// resolution and the batched false-clear #268 §C predicted cannot occur. The residual limit is /// denial: "No" ends the turn with zero further events (no Stop, and no PermissionDenied even when /// registered), so `blocked` stands until the next `UserPromptSubmit`/`SessionStart`. -pub fn observe_hook_event(event: &str, payload: &serde_json::Value) -> Option { +/// +/// DECLARED UNSUPPORTED, stated rather than guessed. A user interrupt (Esc/Ctrl-C) emits no hook +/// event at all, so activity stays `active` with no available edge and no idle is invented. +/// Crash-versus-clean-exit is not knowable here — `SessionEnd` is not registered and its reason +/// vocabulary cannot tell SIGKILL from logout — so the wrapper's exit status remains the only +/// source of `ended`. `Notification`, the only `auth_success` signal, is not registered either, +/// which is why Claude has no paired clear and no `AskKind::Review` mapping exists. +pub fn observe_hook_event(event: &str, payload: &serde_json::Value) -> Option { + observe_hook_event_at(event, payload, message::now_ms()) +} + +/// [`observe_hook_event`] with the hook's own clock passed in, which is what makes the whole +/// mapping a pure function of `(event, payload, now)`: a fault's SEMANTIC observation instant and +/// a conversation link's finite verification bound are both stamped from it. +fn observe_hook_event_at( + event: &str, + payload: &serde_json::Value, + now_ms: u64, +) -> Option { // Any event carrying an agent identity is a subagent's and must never move top-level state: // a phantom `SubagentStop` trails every completed turn, 1.5-2.9s after `Stop`. That phantom // populating `agent_id` is an undocumented emergent property — if a future Claude build omits // it, subagent completions read as top-level activity again and nothing here would catch it. + // Evaluated before every axis: a subagent event must not raise, clear, or move anything. if payload .get("agent_id") .and_then(serde_json::Value::as_str) @@ -552,34 +736,67 @@ pub fn observe_hook_event(event: &str, payload: &serde_json::Value) -> Option Some( - Observation::new(Activity::Idle, BlockedOn::None, InputBuffer::Unknown) - .with_reason("sessionStart"), - ), - "UserPromptSubmit" | "PreToolUse" | "PostToolUse" => Some(Observation::new( - Activity::Active, - BlockedOn::None, - InputBuffer::Unknown, - )), - "Stop" => Some(Observation::new( + let observed = match event { + "SessionStart" => HookState::new( Activity::Idle, - BlockedOn::None, - InputBuffer::Unknown, - )), + HookAsk::None, + // A fresh incarnation is one of the two edges that may clear the axis: a fault stands + // until an explicit paired clear, a terminal record, a new claim, or a NEW + // INCARNATION, and `run_observe`'s `interrupt()` already makes this event that + // boundary. The mid-process sources are NOT boundaries and must not clear. + if session_start_is_fresh_incarnation(payload) { + ConditionReport::Clear + } else { + ConditionReport::Unchanged + }, + ) + .with_reason("sessionStart") + // The boundary edge establishes the link for the incarnation; every later frame inherits + // it from the record. + .with_conversation(conversation_claim(payload, now_ms)), + // Activity edges. Each releases a residual ask — including the DQ-H1 denial residual, for + // which `UserPromptSubmit` is the only recovery edge — and states nothing whatsoever about + // the condition: a turn starting is not evidence that a rejected credential was repaired. + // + // They state nothing about the conversation either, and that is deliberate. These are the + // REPEATABLE edges — several per turn, measured — and the link they would restate is + // identical except for a freshly minted `verifiedThroughMs`. That one moving field would + // make every repeat a different tuple: the write would land instead of coalescing, and + // `sinceMs` would restart on a state the seat never left. The standing link carries. + "UserPromptSubmit" | "PreToolUse" | "PostToolUse" => { + HookState::new(Activity::Active, HookAsk::None, ConditionReport::Unchanged) + } + // THE positive success edge, and Claude's only one: a turn that reached its ordinary end + // is progress a standing fault would have prevented — `ProgressProof::TurnCompleted` — + // so it clears the whole axis. Stated inside the tuple rather than through + // `Writer::clear_all` because the activity and the clear are ONE observation here: two + // writes would publish an intermediate record showing `idle` with the fault still + // standing, and the sibling clear on a fresh incarnation cannot use the condition-only + // path at all, since that record belongs to the predecessor session. + // + // A turn boundary is once-per-turn and never repeats without an intervening prompt, so + // re-verifying the conversation link here refreshes its bound without the churn the + // repeatable edges above would cause. + "Stop" => HookState::new(Activity::Idle, HookAsk::None, ConditionReport::Clear) + .with_conversation(conversation_claim(payload, now_ms)), // `StopFailure` fires INSTEAD of `Stop` when an API error ended the turn (Claude Code's - // own words, 2.1.259), at the same lifecycle point — so the categorical truth is the one - // `Stop` writes and only the reason differs. Deliberately not `ended`: the TUI is still - // live, a human can re-login and carry on, and this seat's terminal record belongs to the - // wrapper (OHS-T04). A hook claiming `ended` here would be a false terminal. - "StopFailure" => Some( - Observation::new(Activity::Idle, BlockedOn::None, InputBuffer::Unknown).with_reason( - match stop_failure_error(payload) { - Some(CLAUDE_AUTH_REJECTED_ERROR) => "providerAuth", - _ => "apiError", - }, - ), - ), + // own words, 2.1.259), at the same lifecycle point — so the activity is the one `Stop` + // writes and the failure rides the condition axis beside it. Deliberately not `ended`: + // the TUI is still live, a human can re-login and carry on, and this seat's terminal + // record belongs to the wrapper (OHS-T04). A hook claiming `ended` here would be a false + // terminal. And deliberately not an ask: the remediation a human owes is visible on the + // condition axis, while a synthesized `pending` would fabricate a prompt nobody can + // answer and would outrank nothing. + "StopFailure" => HookState::new( + Activity::Idle, + HookAsk::None, + stop_failure_condition(payload, now_ms), + ) + .with_reason(match stop_failure_error(payload) { + Some(CLAUDE_AUTH_REJECTED_ERROR) => "providerAuth", + _ => "apiError", + }) + .with_conversation(conversation_claim(payload, now_ms)), "PermissionRequest" => { // Driver-side classification (#162): the payload's tool_name distinguishes Claude's // question form from an ordinary permission prompt — the DQ-H1 captures show @@ -587,18 +804,140 @@ pub fn observe_hook_event(event: &str, payload: &serde_json::Value) -> Option None, + // `PreCompact`/`PostCompact` are harness-CONTEXT edges (see `observe_compaction`) and + // every other event is unregistered or unmapped: silence rather than a guess. + _ => return None, + }; + Some(observed) +} + +/// Whether one `SessionStart` names a genuinely NEW incarnation — the only kind that may clear a +/// standing condition. +/// +/// Two of Claude's source words are mid-process and must not: `compact` is the same session's +/// third sighting of one compaction (see `observe_compaction`), and `clear` empties the +/// conversation inside the running process — same pty, same wrapper, same st2 incarnation. A +/// fault the provider still holds survives both, so clearing on either would silence it on every +/// automatic compaction or every `/clear`. `startup` and `resume` are real process boundaries, +/// and an absent word is treated as one because that is what the event otherwise means. +fn session_start_is_fresh_incarnation(payload: &serde_json::Value) -> bool { + !matches!( + payload.get("source").and_then(serde_json::Value::as_str), + Some(COMPACT_SOURCE | CLEAR_SOURCE) + ) +} + +/// The `SessionStart` source word for a compaction restart: the same incarnation, mid-session. +const COMPACT_SOURCE: &str = "compact"; +/// The `SessionStart` source word for an in-process `/clear`: same incarnation, emptied history. +const CLEAR_SOURCE: &str = "clear"; + +/// The condition one `StopFailure` states, from the closed `error` word and nothing else. +/// +/// PRESENCE-TESTED, not defaulted: every word Claude publishes has its own arm, so +/// `max_output_tokens` can mean "no mapping" — a truncated response is not a seat fault, and +/// collapsing it into the unrecognized-word arm would page an operator once per long answer. +/// Codes are provider-namespaced with a slash, category-routable underneath, and the word rides +/// verbatim so the diagnostic granularity survives the categories collapsing several words into +/// one. +/// +/// No row is [`Recovery::Automatic`] and no row sets a next-observation deadline, because Claude +/// declares neither: it names no reset instant for `rate_limit`, and its internal retries are +/// gated inside the bundle and never surfaced. An `automatic` recovery is projected as +/// recovering/soon and NEVER pages while its deadline holds, so an automatic Claude fault would +/// launder a wedged seat indefinitely. Unclearable classes therefore take [`Recovery::Unknown`], +/// which pages, and Claude's next `Stop` clears them. +/// +/// The credential class comes from this typed word alone. `error_details` and +/// `last_assistant_message` ride the same payload and are prose: splitting `invalid_request` into +/// a context fault by matching English on them is exactly the prose classification INVARIANTS +/// forbids, so the too-large case stays declared-unsupported instead of guessed. +fn stop_failure_condition(payload: &serde_json::Value, observed_at_ms: u64) -> ConditionReport { + let Some(word) = stop_failure_error(payload).filter(|word| !word.is_empty()) else { + // A `StopFailure` with no word at all: the turn provably failed and st2's own driver + // plumbing is the most conservative truthful owner of an unclassified harness-reported + // failure. Codeless rather than code-guessed — a minted word would be indistinguishable + // from one Claude actually published. + return ConditionReport::Fault( + FaultReport::new(FaultCategory::Harness, Recovery::Unknown, observed_at_ms) + .with_detail("stopFailure carried no error word"), + ); + }; + let (category, recovery) = match word { + // A rejected credential: one re-login repairs it. + CLAUDE_AUTH_REJECTED_ERROR => (FaultCategory::Authentication, Recovery::Human), + // An org entitlement no re-login satisfies — a policy refusal, not a credential. + "oauth_org_not_allowed" => (FaultCategory::Policy, Recovery::Human), + // The account itself is the obstacle. Not `quota`: no allowance window is exhausted, and + // an operator's repair is billing rather than waiting. + "account_on_hold" | "billing_error" => (FaultCategory::Account, Recovery::Human), + // Throttled while the allowance itself is intact, with no reset instant published. + "rate_limit" => (FaultCategory::RateLimit, Recovery::Unknown), + // The provider failed on its own side. + "overloaded" | "server_error" => (FaultCategory::Provider, Recovery::Unknown), + // The request this seat's own declaration produced is wrong: a model it may not use, or + // arguments the provider rejects. Both need an operator to change the seat. + "invalid_request" | "model_not_found" => (FaultCategory::Configuration, Recovery::Human), + // DELIBERATE NO MAPPING. The response hit its own output cap; the turn ended, the seat is + // healthy, and nothing about the provider is faulted. An explicit arm rather than a + // default so it can never drift into the unrecognized-word fault below. + "max_output_tokens" => return ConditionReport::Unchanged, + // Claude's own `unknown`, and every future word this build has never seen. Still a fault + // — a turn failed — routed by its recovery and labelled with the most conservative + // truthful category for a harness-reported failure nobody can classify. Guessing a + // neighbouring category would invent a claim the harness never made; the verbatim word + // stays visible in the code. + _ => (FaultCategory::Harness, Recovery::Unknown), + }; + ConditionReport::Fault( + FaultReport::new(category, recovery, observed_at_ms).with_code(format!("claude/{word}")), + ) +} + +/// Claude's conversation identity, from the one typed field every hook payload carries. +/// +/// `session_id` is the provider's own conversation identity — already load-bearing here, since +/// [`wrapperless_token`] derives a whole ownership namespace from it — and it rides VERBATIM, +/// never the prefixed token, which is st2's ownership namespace rather than Claude's identity. +/// The incarnation is not stated: the writing session stamps it, so a reader can always falsify a +/// mismatch against the record's own. +/// +/// `Rewritable`, because Claude compacts: `PreCompact`/`PostCompact` are registered and counted in +/// this same file, so a prefix read once may be gone. `Declared`, because that is pinned knowledge +/// of Claude 2.1.x and st2 never probed the transcript. `transcript_path` rides the same payload +/// and is deliberately not carried: this link is identity and capability only, and a host-local +/// transcript path is content-bearing. +/// +/// A payload with no `session_id` states NOTHING — `None` leaves the axis as the record has it. +/// `Unsupported` would be false (Claude plainly has conversations) and a half-stated link is what +/// the reader degrades with its own rejection word. +fn conversation_claim(payload: &serde_json::Value, now_ms: u64) -> Option { + let conversation = payload + .get("session_id") + .and_then(serde_json::Value::as_str) + .filter(|id| !id.is_empty())?; + // The verification bound must be finite and positive; a zero clock is not a bound, so it + // withholds the claim rather than writing one this build's own reader would degrade. + if now_ms == 0 { + return None; } + Some(ConversationState::Linked(ConversationClaim { + driver: "claude".to_string(), + conversation: conversation.to_string(), + history_mutability: HistoryMutability::Rewritable, + capability_evidence: CapabilityEvidence::Declared, + verified_through_ms: now_ms, + })) } /// The `StopFailure` error word that names a rejected provider credential. @@ -675,6 +1014,20 @@ mod tests { use super::*; use crate::harness_state::harness_state_path; + /// One hook write on the surface `run_observe` uses after activation: the version 3 frame, + /// through the real locked writer. The outcome is returned rather than unwrapped into a bool + /// because a refusal is a VALUE here — a successor's claim, a terminal record, or an axis + /// nobody has stated — and several tests below assert exactly which one. + fn hook_publish( + writer: &mut harness_state::Writer, + event: &str, + payload: &serde_json::Value, + ) -> harness_state::WriteOutcome { + writer + .publish_unless_ended(observe_hook_event(event, payload).unwrap().frame()) + .unwrap() + } + #[test] fn only_the_packaged_channel_requests_the_installation_preflight() { assert!(requires_st2_channel(&[ @@ -760,23 +1113,25 @@ mod tests { } #[test] - fn hook_events_map_to_observations_with_the_blocked_edges() { + fn hook_events_map_to_the_typed_tuple_with_the_blocked_edges() { let none = serde_json::Value::Null; let blocked = observe_hook_event("PermissionRequest", &none).unwrap(); assert_eq!(blocked.state, Activity::Active); - assert_eq!(blocked.blocked_on, BlockedOn::Human); - assert_eq!(blocked.reason.as_deref(), Some("permissionRequest")); + assert_eq!(blocked.ask, HookAsk::Permission); + assert_eq!(blocked.ask.legacy(), (BlockedOn::Human, Ask::Permission)); + assert_eq!(blocked.reason, Some("permissionRequest")); // The exit edges: tool progress or a turn boundary clears the human hold. for event in ["PreToolUse", "PostToolUse"] { let cleared = observe_hook_event(event, &none).unwrap(); assert_eq!(cleared.state, Activity::Active); - assert_eq!(cleared.blocked_on, BlockedOn::None); + assert_eq!(cleared.ask, HookAsk::None); + assert_eq!(cleared.ask.legacy(), (BlockedOn::None, Ask::None)); } let stop = observe_hook_event("Stop", &none).unwrap(); assert_eq!(stop.state, Activity::Idle); - assert_eq!(stop.blocked_on, BlockedOn::None); + assert_eq!(stop.ask, HookAsk::None); assert_eq!( observe_hook_event("UserPromptSubmit", &none).unwrap().state, @@ -821,8 +1176,8 @@ mod tests { ] { let observed = observe_hook_event("StopFailure", payload).unwrap(); assert_eq!(observed.state, Activity::Idle, "the turn ended: {reason}"); - assert_eq!(observed.blocked_on, BlockedOn::None); - assert_eq!(observed.reason.as_deref(), Some(reason)); + assert_eq!(observed.ask, HookAsk::None, "a fault is never an ask"); + assert_eq!(observed.reason, Some(reason)); } assert_eq!( @@ -957,15 +1312,15 @@ mod tests { let entered = observe_hook_event("PermissionRequest", &permission_request).unwrap(); assert_eq!( - (entered.state, entered.blocked_on), - (Activity::Active, BlockedOn::Human) + (entered.state, entered.ask.legacy()), + (Activity::Active, (BlockedOn::Human, Ask::Permission)) ); // 33 s of open prompt produced no intervening event in the capture; the very next event // is the granted call's own PostToolUse, which correctly releases the block. let released = observe_hook_event("PostToolUse", &post_touch).unwrap(); assert_eq!( - (released.state, released.blocked_on), - (Activity::Active, BlockedOn::None) + (released.state, released.ask.legacy()), + (Activity::Active, (BlockedOn::None, Ask::None)) ); let _ = observe_hook_event("PreToolUse", &pre_touch); assert_eq!( @@ -983,16 +1338,18 @@ mod tests { SessionObserver::new(tmp.path(), "hetz.worker", "claude", "hetz.worker").unwrap(); // A hook process wrote a blocked observation between wrapper ticks — carrying the - // wrapper's exported token, exactly as the env plumbing arranges in a real seat. - harness_state::Writer::new( + // wrapper's exported token, exactly as the env plumbing arranges in a real seat, and on + // the surface `run_observe` uses: the boundary event states the condition axis over the + // claim fence, and later hooks ride on top of it. + let mut hook = harness_state::Writer::new( tmp.path(), "hetz.worker", "claude", Some("hetz.worker".to_string()), ) - .with_session(observer.session()) - .observe(observe_hook_event("PermissionRequest", &serde_json::Value::Null).unwrap()) - .unwrap(); + .with_session(observer.session()); + hook_publish(&mut hook, "SessionStart", &serde_json::Value::Null); + hook_publish(&mut hook, "PermissionRequest", &serde_json::Value::Null); let before = fs::read(&record).unwrap(); std::thread::sleep(Duration::from_millis(2)); @@ -1021,7 +1378,11 @@ mod tests { "claude", Some("hetz.worker".to_string()), ) - .observe(observe_hook_event("UserPromptSubmit", &serde_json::Value::Null).unwrap()) + .observe( + observe_hook_event("UserPromptSubmit", &serde_json::Value::Null) + .unwrap() + .observation(), + ) .unwrap(); let result = run_provider( @@ -1068,24 +1429,551 @@ mod tests { #[test] fn permission_requests_classify_their_ask_kind_from_the_tool_name() { - use crate::harness_state::Ask; let permission = observe_hook_event( "PermissionRequest", &serde_json::json!({ "tool_name": "Bash", "tool_input": {} }), ) .unwrap(); - assert_eq!(permission.ask, Ask::Permission); + assert_eq!(permission.ask, HookAsk::Permission); + assert_eq!( + permission.ask.tagged(), + HumanAsk::Pending(AskKind::Permission) + ); + assert_eq!(permission.ask.legacy(), (BlockedOn::Human, Ask::Permission)); let question = observe_hook_event( "PermissionRequest", &serde_json::json!({ "tool_name": "AskUserQuestion", "tool_input": {} }), ) .unwrap(); - assert_eq!(question.ask, Ask::Question); + assert_eq!(question.ask, HookAsk::Question); + assert_eq!(question.ask.tagged(), HumanAsk::Pending(AskKind::Question)); + assert_eq!(question.ask.legacy(), (BlockedOn::Human, Ask::Question)); + + // Claude has no review signal at all, so no mapping may produce one. + assert_ne!(question.ask.tagged(), HumanAsk::Pending(AskKind::Review)); // Non-blocking events carry no ask. let idle = observe_hook_event("Stop", &serde_json::json!({})).unwrap(); - assert_eq!(idle.ask, Ask::None); + assert_eq!(idle.ask, HookAsk::None); + assert_eq!(idle.ask.tagged(), HumanAsk::None); + assert_eq!(idle.ask.legacy(), (BlockedOn::None, Ask::None)); + } + + /// A deterministic clock, so the mapping is asserted as the pure function it is. + const OBSERVED_AT_MS: u64 = 1_764_000_000_000; + + /// The closed `StopFailure` vocabulary (Claude Code 2.1.259), pinned word by word: ten words + /// that state a condition and one — `max_output_tokens` — that deliberately states none. A + /// `_ =>` default over this table would publish a page per truncated response, which is why + /// every word has its own arm and why this test enumerates all eleven rather than sampling. + #[test] + fn every_stop_failure_word_maps_to_its_own_fault_or_to_no_mapping_at_all() { + let table: [(&str, Option<(FaultCategory, &str, Recovery)>); 11] = [ + ( + "authentication_failed", + Some(( + FaultCategory::Authentication, + "claude/authentication_failed", + Recovery::Human, + )), + ), + ( + "oauth_org_not_allowed", + Some(( + FaultCategory::Policy, + "claude/oauth_org_not_allowed", + Recovery::Human, + )), + ), + ( + "account_on_hold", + Some(( + FaultCategory::Account, + "claude/account_on_hold", + Recovery::Human, + )), + ), + ( + "billing_error", + Some(( + FaultCategory::Account, + "claude/billing_error", + Recovery::Human, + )), + ), + ( + "rate_limit", + Some(( + FaultCategory::RateLimit, + "claude/rate_limit", + Recovery::Unknown, + )), + ), + ( + "overloaded", + Some(( + FaultCategory::Provider, + "claude/overloaded", + Recovery::Unknown, + )), + ), + ( + "server_error", + Some(( + FaultCategory::Provider, + "claude/server_error", + Recovery::Unknown, + )), + ), + ( + "invalid_request", + Some(( + FaultCategory::Configuration, + "claude/invalid_request", + Recovery::Human, + )), + ), + ( + "model_not_found", + Some(( + FaultCategory::Configuration, + "claude/model_not_found", + Recovery::Human, + )), + ), + // The deliberate null: a response that hit its own output cap says nothing about the + // seat's health, so the condition axis is carried untouched. + ("max_output_tokens", None), + ( + "unknown", + Some((FaultCategory::Harness, "claude/unknown", Recovery::Unknown)), + ), + ]; + + for (word, want) in table { + let payload = serde_json::json!({"hook_event_name": "StopFailure", "error": word}); + let condition = stop_failure_condition(&payload, OBSERVED_AT_MS); + let Some((category, code, recovery)) = want else { + assert_eq!( + condition, + ConditionReport::Unchanged, + "{word} maps to nothing at all" + ); + continue; + }; + let ConditionReport::Fault(fault) = condition else { + panic!("{word} states a fault") + }; + assert_eq!(fault.category, category, "{word}"); + assert_eq!(fault.code.as_deref(), Some(code), "{word}"); + assert_eq!(fault.recovery, recovery, "{word}"); + // The SEMANTIC instant, from the hook's own clock. + assert_eq!(fault.observed_at_ms, OBSERVED_AT_MS, "{word}"); + assert_eq!( + fault.detail, None, + "`error_details` is prose and never rides the fault: {word}" + ); + } + + // A word this build has never seen is still a fault — visible, routed by its recovery, + // and never guessed into a neighbouring category — carrying the verbatim word. + let ConditionReport::Fault(future) = stop_failure_condition( + &serde_json::json!({"error": "quantum_flux"}), + OBSERVED_AT_MS, + ) else { + panic!("an unrecognized word is still a failed turn") + }; + assert_eq!(future.category, FaultCategory::Harness); + assert_eq!(future.code.as_deref(), Some("claude/quantum_flux")); + assert_eq!(future.recovery, Recovery::Unknown); + + // Presence-tested, not defaulted: a `StopFailure` carrying no word at all is codeless + // rather than code-guessed, because a minted word would be indistinguishable from one + // Claude published. + for payload in [ + serde_json::json!({"hook_event_name": "StopFailure"}), + serde_json::json!({"error": ""}), + ] { + let ConditionReport::Fault(wordless) = stop_failure_condition(&payload, OBSERVED_AT_MS) + else { + panic!("a wordless StopFailure still failed the turn: {payload}") + }; + assert_eq!(wordless.category, FaultCategory::Harness); + assert_eq!(wordless.code, None); + assert_eq!(wordless.recovery, Recovery::Unknown); + } + } + + /// The anti-laundering guard. `Recovery::Automatic` is projected as recovering/soon and NEVER + /// pages while its deadline holds, so an automatic Claude fault would keep a wedged seat + /// quiet indefinitely — and Claude has no retry visibility and publishes no reset instant to + /// time one with. No row may declare it, and no row may set a deadline. + #[test] + fn claude_never_declares_a_recovery_it_cannot_time() { + for word in [ + "authentication_failed", + "oauth_org_not_allowed", + "account_on_hold", + "billing_error", + "rate_limit", + "overloaded", + "server_error", + "invalid_request", + "model_not_found", + "max_output_tokens", + "unknown", + "a_word_from_a_later_build", + "", + ] { + let payload = serde_json::json!({"error": word}); + let ConditionReport::Fault(fault) = stop_failure_condition(&payload, OBSERVED_AT_MS) + else { + continue; + }; + assert_ne!(fault.recovery, Recovery::Automatic, "{word}"); + assert_ne!( + fault.recovery, + Recovery::Terminal, + "a re-login or a wait can still clear it: {word}" + ); + assert_eq!(fault.next_observation_due_ms, None, "{word}"); + } + } + + /// `StopFailure` is idle PLUS a fault: the TUI is live and only the wrapper's process exit + /// writes the terminal word. It synthesizes no ask either — the remediation is on the + /// condition axis, and a pending ask would fabricate a prompt nobody can answer. + #[test] + fn a_stop_failure_is_idle_with_a_fault_and_never_ended_or_an_ask() { + let payload = serde_json::json!({ + "hook_event_name": "StopFailure", + "error": "rate_limit", + "session_id": "s-1", + "error_details": "please try again later", + }); + let observed = observe_hook_event_at("StopFailure", &payload, OBSERVED_AT_MS).unwrap(); + assert_eq!(observed.state, Activity::Idle); + assert_ne!( + observed.state, + Activity::Ended, + "the wrapper's exit is the sole terminal writer" + ); + assert_eq!(observed.ask, HookAsk::None); + let ConditionReport::Fault(fault) = &observed.condition else { + panic!("the failure rides the condition axis") + }; + assert_eq!(fault.category, FaultCategory::RateLimit); + assert_eq!(fault.observed_at_ms, OBSERVED_AT_MS); + + // The version 3 statement: one frame, no exit, the ask still positively none. + let frame = observed.frame(); + assert_eq!(frame.state, Activity::Idle); + assert_eq!(frame.ask, HumanAsk::None); + assert_eq!(frame.exit, None, "a fault never carries a terminal outcome"); + assert_eq!(frame.condition, observed.condition); + assert_eq!(frame.reason.as_deref(), Some("apiError")); + + // The truncation word is the one `StopFailure` that raises nothing at all, while still + // reporting the turn's end exactly as it always did. + let truncated = observe_hook_event_at( + "StopFailure", + &serde_json::json!({"error": "max_output_tokens"}), + OBSERVED_AT_MS, + ) + .unwrap(); + assert_eq!(truncated.state, Activity::Idle); + assert_eq!(truncated.condition, ConditionReport::Unchanged); + assert_eq!(truncated.reason, Some("apiError")); + } + + /// The clear edges, and the two `SessionStart` sources that look like boundaries but are not. + #[test] + fn a_completed_turn_and_a_fresh_incarnation_are_the_only_edges_that_clear() { + let session = serde_json::json!({"session_id": "s-1"}); + let stop = observe_hook_event_at("Stop", &session, OBSERVED_AT_MS).unwrap(); + assert_eq!( + (stop.state, stop.condition.clone()), + (Activity::Idle, ConditionReport::Clear), + "a completed turn is progress a standing fault would have prevented" + ); + + // Genuine process boundaries: a new incarnation supersedes whatever the last one held. + for source in ["startup", "resume"] { + let mut payload = session.clone(); + payload["source"] = source.into(); + let boundary = observe_hook_event_at("SessionStart", &payload, OBSERVED_AT_MS).unwrap(); + assert_eq!(boundary.condition, ConditionReport::Clear, "{source}"); + assert_eq!(boundary.state, Activity::Idle, "{source}"); + } + // A `SessionStart` with no source word is treated as one, because that is what the event + // otherwise means. + assert_eq!( + observe_hook_event_at("SessionStart", &session, OBSERVED_AT_MS) + .unwrap() + .condition, + ConditionReport::Clear + ); + + // The MID-PROCESS sources. `compact` is the same session seeing one compaction a third + // time (see `observe_compaction`); `clear` empties the conversation inside the running + // process — same pty, same wrapper, same incarnation. A fault the provider still holds + // survives both, so clearing on either would silence it on every automatic compaction and + // every `/clear`. + for source in ["compact", "clear"] { + let mut payload = session.clone(); + payload["source"] = source.into(); + let mid_process = + observe_hook_event_at("SessionStart", &payload, OBSERVED_AT_MS).unwrap(); + assert_eq!( + mid_process.condition, + ConditionReport::Unchanged, + "{source}" + ); + assert_eq!(mid_process.state, Activity::Idle, "{source}"); + assert_eq!(mid_process.reason, Some("sessionStart"), "{source}"); + } + } + + /// An activity or ask edge has learned NOTHING about whether the provider is faulted. Carrying + /// the axis is also what preserves a standing fault's semantic clock: a stated `clear` here + /// would silence it, and a restated fault would re-mint `observedAtMs` on every tool call. + #[test] + fn an_activity_edge_never_clears_a_standing_condition() { + let session = serde_json::json!({"session_id": "s-1"}); + for event in [ + "UserPromptSubmit", + "PreToolUse", + "PostToolUse", + "PermissionRequest", + ] { + let observed = observe_hook_event_at(event, &session, OBSERVED_AT_MS).unwrap(); + assert_eq!(observed.state, Activity::Active, "{event}"); + assert_eq!(observed.condition, ConditionReport::Unchanged, "{event}"); + assert_eq!( + observed.frame().condition, + ConditionReport::Unchanged, + "the carry must reach the write, not just the mapping: {event}" + ); + } + + // A repeatable edge is CLOCK-INDEPENDENT: nothing it states may be minted from `now`, or + // two identical observations would be two different tuples. The conversation link is the + // one field that could be, so these edges state none and the standing link carries. + for event in [ + "UserPromptSubmit", + "PreToolUse", + "PostToolUse", + "PermissionRequest", + ] { + let first = observe_hook_event_at(event, &session, OBSERVED_AT_MS).unwrap(); + let later = observe_hook_event_at(event, &session, OBSERVED_AT_MS + 90_000).unwrap(); + assert_eq!(first.conversation, None, "{event}"); + assert_eq!( + first, later, + "a repeat must restate exactly the same tuple: {event}" + ); + assert_eq!( + first.frame(), + later.frame(), + "identical frames are what makes the repeat coalesce: {event}" + ); + } + + // The measured DQ-H1 denial residual on the tagged axis: "No" ends the turn with no + // further event, so the ask stands until the next prompt, which is the only edge that + // releases it — and that release still touches no condition. + let denied = observe_hook_event_at( + "PermissionRequest", + &serde_json::json!({"session_id": "s-1", "tool_name": "Bash"}), + OBSERVED_AT_MS, + ) + .unwrap(); + assert_eq!(denied.ask.tagged(), HumanAsk::Pending(AskKind::Permission)); + let next_prompt = + observe_hook_event_at("UserPromptSubmit", &session, OBSERVED_AT_MS).unwrap(); + assert_eq!(next_prompt.ask, HookAsk::None); + assert_eq!(next_prompt.condition, ConditionReport::Unchanged); + } + + /// The write-rate consequence of the rule above, on the path this build emits: a repeated + /// activity edge must not touch the record at all. If any axis were minted from `now`, the + /// second write would land, `transitions` would advance, and `sinceMs` — when the state was + /// ENTERED — would restart on a state the seat never left, on every tool call of every turn. + #[test] + fn a_repeated_activity_edge_neither_writes_nor_restarts_since() { + let tmp = tempfile::tempdir().unwrap(); + let record = harness_state_path(tmp.path()); + let payload = serde_json::json!({"session_id": "s-1"}); + let mut writer = harness_state::Writer::new( + tmp.path(), + "hetz.worker", + "claude", + Some("hetz.worker".to_string()), + ); + // The boundary event states the condition axis once; version 3 refuses an activity-only + // frame before that, which is the same bootstrap `run_observe` relies on in production. + hook_publish(&mut writer, "SessionStart", &payload); + let publish = |writer: &mut harness_state::Writer| { + hook_publish(writer, "PostToolUse", &payload).accepted() + }; + + assert!(publish(&mut writer)); + let first = fs::read(&record).unwrap(); + let since_ms = harness_state::read(&record, None).unwrap().since_ms; + + std::thread::sleep(Duration::from_millis(2)); + assert!(publish(&mut writer), "the record still says this"); + assert_eq!( + fs::read(&record).unwrap(), + first, + "a restated activity edge coalesces: nothing was written" + ); + assert_eq!( + harness_state::read(&record, None).unwrap().since_ms, + since_ms, + "`sinceMs` marks when the state was entered and must not restart" + ); + } + + /// The conversation bridge, from the one typed field every hook payload carries. + #[test] + fn hook_states_link_claudes_session_id_with_bounded_declared_evidence() { + let payload = serde_json::json!({ + "session_id": "abc-123", + "transcript_path": "/home/x/.claude/projects/abc-123.jsonl", + }); + let observed = observe_hook_event_at("Stop", &payload, OBSERVED_AT_MS).unwrap(); + let Some(ConversationState::Linked(link)) = observed.conversation.clone() else { + panic!("Claude's session id is typed evidence of its own conversation") + }; + assert_eq!(link.driver, "claude"); + assert_eq!( + link.conversation, "abc-123", + "the provider's identity rides verbatim" + ); + assert_ne!( + Some(link.conversation.clone()), + wrapperless_token(&payload), + "the prefixed token is st2's ownership namespace, not Claude's identity" + ); + // Claude compacts — this file counts the edges — so a prefix read once may be gone. + assert_eq!(link.history_mutability, HistoryMutability::Rewritable); + // Pinned knowledge of 2.1.x; st2 never probed the transcript. + assert_eq!(link.capability_evidence, CapabilityEvidence::Declared); + // Finite and positive: a consumer ages the claim instead of trusting it forever. + assert_eq!(link.verified_through_ms, OBSERVED_AT_MS); + // Identity and capability ONLY: the transcript path rides the same payload and is + // content-bearing, so it must not reach a replicated record. + assert!(!format!("{link:?}").contains("transcript"), "{link:?}"); + assert_eq!(observed.frame().conversation, observed.conversation); + + // Nothing to state is stated as nothing: `Unsupported` would be false (Claude plainly + // has conversations) and a half-stated link is what the reader degrades. + for silent in [serde_json::json!({}), serde_json::json!({"session_id": ""})] { + assert_eq!( + observe_hook_event_at("Stop", &silent, OBSERVED_AT_MS) + .unwrap() + .conversation, + None, + "{silent}" + ); + } + // A zero clock is not a finite verification bound, so the claim is withheld rather than + // written in a shape this build's own reader rejects. + assert_eq!( + observe_hook_event_at("Stop", &payload, 0) + .unwrap() + .conversation, + None + ); + } + + /// Source compatibility: on a writer that does not emit the condition axis, every Claude hook + /// writes exactly the bytes it always did. The condition and conversation axes are DROPPED + /// rather than half-written or stashed in a sidecar, and their absence reads as `absent` — + /// never as health. Production emits version 3 now, so the version 2 writer comes from the + /// test-only emitted-schema seam: this is the shape of every legacy record still on disk and + /// of anything a rollback writes. + #[test] + fn the_legacy_wire_is_unchanged_while_the_writer_omits_the_condition_axis() { + let tmp = tempfile::tempdir().unwrap(); + let record = harness_state_path(tmp.path()); + let payload = serde_json::json!({"session_id": "s-1", "error": "rate_limit"}); + let mut writer = harness_state::Writer::new( + tmp.path(), + "hetz.worker", + "claude", + Some("hetz.worker".to_string()), + ) + .with_emitted_schema(harness_state::SCHEMA_V2); + assert!( + !writer.writes_condition_axis(), + "the legacy projection is reached through the version 2 seam" + ); + assert!( + writer + .observe_unless_ended( + observe_hook_event("StopFailure", &payload) + .unwrap() + .observation() + ) + .unwrap() + ); + + let raw = fs::read_to_string(&record).unwrap(); + for key in ["condition", "conversationRef", "humanAsk"] { + assert!( + !raw.contains(&format!("\"{key}\"")), + "{key} must not reach the version 2 wire: {raw}" + ); + } + let observed = harness_state::read(&record, None).unwrap(); + assert_eq!(observed.state, Activity::Idle); + assert_eq!(observed.blocked_on, BlockedOn::None); + assert_eq!(observed.ask, Ask::None); + assert_eq!(observed.reason.as_deref(), Some("apiError")); + assert_eq!( + observed.condition, + harness_state::ConditionView::Absent, + "a record that never spoke about health is not healthy" + ); + } + + /// Claude's unobservable edges stay declared-unsupported: no mapping exists for an interrupt, + /// for `SessionEnd`, or for `Notification` (the only `auth_success` signal, unregistered — + /// which is why Claude has no paired clear). Pinned so a future build that starts emitting + /// them fails loudly here instead of silently doing nothing in the field. + #[test] + fn claudes_unobservable_edges_stay_declared() { + let session = serde_json::json!({"session_id": "s-1"}); + for event in [ + "Notification", + "SessionEnd", + "SubagentStart", + "SubagentStop", + "PermissionDenied", + "Interrupt", + // The compaction edges are harness-CONTEXT only and must never touch this axis. + "PreCompact", + "PostCompact", + ] { + assert_eq!(observe_hook_event(event, &session), None, "{event}"); + } + } + + /// A subagent's event must not raise, clear, or move anything — on either axis. + #[test] + fn subagent_events_never_raise_or_clear_a_condition() { + for (event, error) in [("StopFailure", "authentication_failed"), ("Stop", "")] { + let payload = serde_json::json!({ + "session_id": "s-1", + "agent_id": "sub-1", + "agent_type": "", + "error": error, + }); + assert_eq!(observe_hook_event(event, &payload), None, "{event}"); + } } /// T2: a hook that finishes after the wrapper reaped Claude must not replace the terminal @@ -1108,13 +1996,7 @@ mod tests { Some("hetz.worker".to_string()), ) .with_session(observer.session()); - assert!( - !late - .observe_unless_ended( - observe_hook_event("PostToolUse", &serde_json::Value::Null).unwrap() - ) - .unwrap() - ); + assert!(!hook_publish(&mut late, "PostToolUse", &serde_json::Value::Null).accepted()); assert_eq!( harness_state::read(&record, None).unwrap().state, Activity::Ended @@ -1131,11 +2013,7 @@ mod tests { .with_session("claude-session-fresh"); fallback.interrupt(); assert!( - !fallback - .observe_unless_ended( - observe_hook_event("SessionStart", &serde_json::Value::Null).unwrap() - ) - .unwrap() + !hook_publish(&mut fallback, "SessionStart", &serde_json::Value::Null).accepted() ); assert_eq!( harness_state::read(&record, None).unwrap().state, @@ -1152,13 +2030,7 @@ mod tests { Some("hetz.worker".to_string()), ) .with_ownership(next.session().to_string(), next.seq()); - assert!( - fresh - .observe_unless_ended( - observe_hook_event("SessionStart", &serde_json::Value::Null).unwrap() - ) - .unwrap() - ); + assert!(hook_publish(&mut fresh, "SessionStart", &serde_json::Value::Null).accepted()); assert_eq!( harness_state::read(&record, None).unwrap().state, Activity::Idle @@ -1175,12 +2047,11 @@ mod tests { let record = harness_state_path(tmp.path()); let payload_a = serde_json::json!({ "session_id": "aaa" }); let payload_b = serde_json::json!({ "session_id": "bbb" }); + // The whole ownership selection plus the write, exactly as `run_observe` performs them. let drive = |event: &str, payload: &serde_json::Value| { let mut writer = observe_writer(tmp.path(), "hetz.worker", None, event, payload, None, None); - writer - .observe_unless_ended(observe_hook_event(event, payload).unwrap()) - .unwrap() + hook_publish(&mut writer, event, payload).accepted() }; assert!(drive("SessionStart", &payload_a), "A claims"); @@ -1202,6 +2073,119 @@ mod tests { ); } + /// The wrapperless session boundary, decided in one predicate. + /// + /// The version 3 half is the load-bearing one: a claim writes this session's FENCE, which is + /// excluded from what an unstated axis inherits, so claiming on a mid-process `SessionStart` + /// would drop a standing fault by OWNERSHIP — the same laundering the mapping refuses to do + /// with `clear`, arriving through the other door. Pinned as a pure function of + /// `(writes_condition_axis, event, payload)` so BOTH arms stay decidable from one build; the + /// version 3 arm is now also proved end-to-end, against the real writer, by + /// `a_wrapperless_session_start_claims_only_for_a_fresh_incarnation`. + #[test] + fn a_mid_process_session_start_claims_only_while_the_condition_axis_is_unwritable() { + for source in [ + None, + Some("startup"), + Some("resume"), + Some("compact"), + Some("clear"), + ] { + let mut payload = serde_json::json!({"session_id": "aaa"}); + if let Some(source) = source { + payload["source"] = source.into(); + } + // The legacy wire: every `SessionStart` is the boundary and claims, unchanged. + assert!( + claims_wrapperless_session(false, "SessionStart", &payload), + "{source:?}" + ); + // With the condition axis on the wire, only a genuinely fresh incarnation claims. + let fresh = !matches!(source, Some("compact") | Some("clear")); + assert_eq!( + claims_wrapperless_session(true, "SessionStart", &payload), + fresh, + "{source:?}" + ); + } + + // No other event is a session boundary on either wire: token-only writers never claim. + let payload = serde_json::json!({"session_id": "aaa"}); + for event in [ + "Stop", + "StopFailure", + "UserPromptSubmit", + "PreToolUse", + "PostToolUse", + "PermissionRequest", + "PreCompact", + ] { + assert!( + !claims_wrapperless_session(false, event, &payload), + "{event}" + ); + assert!( + !claims_wrapperless_session(true, event, &payload), + "{event}" + ); + } + } + + /// The version 3 arm of the predicate above, end to end against the real writer — the proof + /// activation makes reachable. + /// + /// A mid-process `SessionStart` (`compact`, `clear`) keeps the token-only path, so no claim + /// fence is written and the standing fault this session already stated CARRIES: laundering it + /// away by ownership is exactly the defect the predicate exists to prevent. A genuinely fresh + /// incarnation (`startup`, `resume`) claims, and superseding the axis there is the truth. + #[test] + fn a_wrapperless_session_start_claims_only_for_a_fresh_incarnation() { + let seat = |source: &str| { + let tmp = tempfile::tempdir().unwrap(); + let record = harness_state_path(tmp.path()); + let drive = |event: &str, payload: &serde_json::Value| { + let mut writer = + observe_writer(tmp.path(), "hetz.worker", None, event, payload, None, None); + hook_publish(&mut writer, event, payload) + }; + // Claude's session id does NOT change across a compaction, so the successor differs + // from the predecessor by `source` alone — which is the whole question. + let mut session = serde_json::json!({"session_id": "aaa"}); + drive("SessionStart", &session); + let mut failed = session.clone(); + failed["error"] = "authentication_failed".into(); + drive("StopFailure", &failed); + let before = harness_state::read(&record, None).expect("a stated record"); + session["source"] = source.into(); + drive("SessionStart", &session); + ( + before, + harness_state::read(&record, None).expect("a stated record"), + ) + }; + + for mid_process in ["compact", "clear"] { + let (before, after) = seat(mid_process); + assert!( + matches!(before.condition, harness_state::ConditionView::Fault(_)), + "{mid_process}: the fault must stand before the boundary event" + ); + assert!( + matches!(after.condition, harness_state::ConditionView::Fault(_)), + "{mid_process}: a mid-process boundary must not launder a standing fault by \ + ownership" + ); + } + for fresh in ["startup", "resume"] { + let (_, after) = seat(fresh); + assert_eq!( + after.condition, + harness_state::ConditionView::Clear, + "{fresh}: a fresh incarnation legitimately supersedes the axis" + ); + } + } + /// The status-line payload captured verbatim from the version in the producer table, before /// the session's first API response. const PRE_TURN: &str = diff --git a/src/codex_app_server.rs b/src/codex_app_server.rs index 73b69b43..e7510031 100644 --- a/src/codex_app_server.rs +++ b/src/codex_app_server.rs @@ -31,7 +31,9 @@ use serde_json::{Value, json}; use sha2::{Digest as _, Sha256}; use tungstenite::{Message as WebSocketMessage, WebSocket}; -use crate::{ding, driver_diagnostic, harness_context, harness_state, message, run, status}; +use crate::{ + delivery_ledger, ding, driver_diagnostic, harness_context, harness_state, message, run, status, +}; const REQUIRED_CODEX_CLIENT_REQUESTS: &[&str] = &[ "hooks/list", @@ -43,6 +45,10 @@ const REQUIRED_CODEX_CLIENT_REQUESTS: &[&str] = &[ ]; const REQUIRED_CODEX_CLIENT_NOTIFICATIONS: &[&str] = &["initialized"]; const REQUIRED_CODEX_SERVER_NOTIFICATIONS: &[&str] = &[ + // The account window and the typed error notification are the fault axis's two native + // inputs; both are branched on, so both are admitted here and pinned by shape below. + "account/rateLimits/updated", + "error", "item/completed", "item/started", "thread/started", @@ -91,7 +97,6 @@ const CLASSIFIED_CODEX_THREAD_ITEMS: &[&str] = &[ const RUNTIME_SCHEMA: &str = "st2.codex-runtime.v1"; const BINDING_SCHEMA: &str = "st2.codex-thread-binding.v1"; const CONTROL_STATE_SCHEMA: &str = "st2.codex-control-state.v1"; -const DELIVERY_STATE_SCHEMA: &str = "st2.codex-delivery-state.v1"; const WRAPPER_DIAGNOSTIC_SCHEMA: &str = "st2.codex-wrapper-diagnostic.v1"; const CONTROL_TUI_LOADED_REQUEST_ID: u64 = 0; const CONTROL_SUBSCRIBE_REQUEST_ID: u64 = 1; @@ -295,6 +300,299 @@ fn codex_turn_outcome(turn: Option<&Value>) -> CodexTurnOutcome { } } +/// Every `CodexErrorInfo` enum word this build's fault table forks on. +/// +/// Pinned by [`verify_codex_protocol_schemas`]: a release that renamed or merged one of these +/// must refuse the launch rather than let st2 silently reclassify a live seat's failure into the +/// unclassified fall-through. The union's data-carrying arms are single-key objects rather than +/// words and are deliberately absent — losing one of those costs precision, not visibility. +const CODEX_CLASSIFIED_ERROR_WORDS: &[&str] = &[ + "badRequest", + "contextWindowExceeded", + "cyberPolicy", + "internalServerError", + "misalignmentPolicyViolation", + "other", + "rateLimitExceeded", + "sandboxError", + "serverOverloaded", + "sessionBudgetExceeded", + "threadRollbackFailed", + CODEX_PROVIDER_AUTH_REJECTED, + "usageLimitExceeded", +]; + +/// Every `RateLimitReachedType` word this build's fault table forks on, pinned for the same +/// reason as the error words above: the fork between a throttled window and an exhausted +/// allowance is exactly what these words decide. +const CODEX_RATE_LIMIT_REACHED_WORDS: &[&str] = &[ + "rate_limit_reached", + "workspace_member_credits_depleted", + "workspace_member_usage_limit_reached", + "workspace_owner_credits_depleted", + "workspace_owner_usage_limit_reached", +]; + +/// The provider's own explanation for a resume it refused, carried as the conversation bridge's +/// diagnostic reason. st2's own rejection words for that axis are reserved and unspellable here. +const CODEX_NO_PERSISTED_ROLLOUT: &str = "no persisted rollout"; + +/// Which signal raised a fault, so a paired clear names the EXACT key its own signal raised. +/// +/// The account rate-limit window is deliberately NOT a member. `account/rateLimits/updated` is a +/// sparse rolling update whose nullable fields mean "unavailable in this update", never +/// "recovered" — the generated schema says so in as many words — so no snapshot of it can be +/// read as a clear. A window fault this build raises is escaped only by a positive turn +/// completion, a new claim, or a new incarnation, until a merged or explicitly read snapshot +/// signal exists to prove the window lifted. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum CodexFaultSource { + /// A typed word Codex itself named, on a turn result, the error notification, or the account + /// window's own reached type. + Typed, + /// st2's own unclassified-protocol hold; leaving that hold is its paired clear. + UnknownProtocol, + /// A failure the harness reported without classifying it. It never REPLACES a fault that + /// already stands, because a typed word for the same episode says strictly more. + Unclassified, +} + +/// One statement about the fault axis, derived from exactly one signal. +#[derive(Debug, Clone, PartialEq, Eq)] +enum CodexConditionOp { + Raise { + fault: harness_state::FaultReport, + source: CodexFaultSource, + }, + /// Clear the fault this incarnation raised from that signal, by the exact key it raised. + ClearRaised(CodexFaultSource), + /// Clear the whole axis on a positive observation of progress a fault would have blocked. + ClearAll(harness_state::ProgressProof), +} + +/// A write this producer still owes the record. +#[derive(Debug, Clone)] +enum CodexPendingWrite { + /// The legacy observation, which is the shape this build's writer still emits. + Legacy(harness_state::Observation), + /// The version 3 tuple. + Frame(harness_state::Frame), +} + +/// What this producer can prove about its conversation identity. +#[derive(Debug, Clone, PartialEq, Eq)] +enum CodexConversationEvidence { + /// No thread bound yet — and nothing is published while unbound either. + Unbound, + /// A thread a typed provider response named, verified through that frame's instant. + Probed { + thread_id: String, + verified_through_ms: u64, + }, + /// The identity exists and is not reachable from this connection. + Unreachable { reason: &'static str }, +} + +/// Codex's own error vocabulary mapped onto the closed fault axis, plus the recovery class that +/// word means where the harness states no retry of its own. +/// +/// A presence-tested `match` with an explicit fall-through, never a lookup with a default: a +/// future or renamed word must stay VISIBLE under the most conservative truthful category rather +/// than be folded into a neighbour, so it becomes a harness-reported unclassified failure whose +/// recovery nobody claims to know. `other` — Codex's own catch-all — lands there by the same +/// route, because it names no cause at all. +fn codex_fault_class(word: &str) -> (harness_state::FaultCategory, harness_state::Recovery) { + use crate::harness_state::{FaultCategory as Category, Recovery}; + match word { + CODEX_PROVIDER_AUTH_REJECTED => (Category::Authentication, Recovery::Human), + "usageLimitExceeded" | "sessionBudgetExceeded" => (Category::Quota, Recovery::Human), + "rateLimitExceeded" => (Category::RateLimit, Recovery::Human), + "serverOverloaded" + | "internalServerError" + | "httpConnectionFailed" + | "responseStreamConnectionFailed" + | "responseStreamDisconnected" + | "responseTooManyFailedAttempts" => (Category::Provider, Recovery::Human), + "contextWindowExceeded" => (Category::Context, Recovery::Human), + "cyberPolicy" | "misalignmentPolicyViolation" => (Category::Policy, Recovery::Human), + "badRequest" | "activeTurnNotSteerable" => (Category::Configuration, Recovery::Human), + "threadRollbackFailed" | "sandboxError" => (Category::Harness, Recovery::Human), + _ => (Category::Harness, Recovery::Unknown), + } +} + +/// The one word a `CodexErrorInfo` names. +/// +/// The union has two shapes: bare enum strings, and single-key objects carrying data +/// (`httpConnectionFailed`, the response-stream family, `activeTurnNotSteerable`). An +/// `as_str`-only read would silently drop that whole family, so the object arm's single key IS +/// its word. Anything else — a null, a multi-key object, a shape this build cannot read — states +/// nothing rather than guessing. +fn codex_error_info_word(info: Option<&Value>) -> Option<&str> { + match info? { + Value::String(word) if !word.is_empty() => Some(word.as_str()), + Value::Object(fields) if fields.len() == 1 => fields + .keys() + .next() + .map(String::as_str) + .filter(|word| !word.is_empty()), + _ => None, + } +} + +/// One Codex error word as a fault a consumer can route, time, and age. +fn codex_fault( + word: &str, + will_retry: Option, + observed_at_ms: u64, +) -> harness_state::FaultReport { + let (category, stated) = codex_fault_class(word); + let recovery = match will_retry { + // `willRetry` is required on the error notification and absent from a turn result. Where + // Codex states it, a promised retry IS an automatic recovery; a refused retry leaves the + // table's own class, which is never optimistic. + Some(true) => harness_state::Recovery::Automatic, + Some(false) | None => stated, + }; + harness_state::FaultReport::new(category, recovery, observed_at_ms) + .with_code(format!("codex/{word}")) +} + +/// What one inbound control frame states about the fault axis. +/// +/// One frame yields at most one statement, and a frame that states nothing yields `None` — +/// never a clear, because an activity edge has observed nothing about the provider. Codex emits +/// its unclassified `thread/status/changed -> systemError` BEFORE the typed `error` notification +/// and the failed `turn/completed`, so the typed words arrive last and win. +fn codex_condition_edge( + message: &Value, + thread_id: &str, + observed_at_ms: u64, +) -> Option { + let method = message.get("method").and_then(Value::as_str)?; + let bound = message.pointer("/params/threadId").and_then(Value::as_str) == Some(thread_id); + match method { + "error" if bound => Some(CodexConditionOp::Raise { + fault: codex_fault( + codex_error_info_word(message.pointer("/params/error/codexErrorInfo"))?, + message + .pointer("/params/willRetry") + .and_then(Value::as_bool), + observed_at_ms, + ), + source: CodexFaultSource::Typed, + }), + "turn/completed" if bound => { + match message + .pointer("/params/turn/status") + .and_then(Value::as_str)? + { + // The only positive progress proof this protocol emits: a turn ran to completion + // against the provider, which every fault above would have prevented. + "completed" => Some(CodexConditionOp::ClearAll( + harness_state::ProgressProof::TurnCompleted, + )), + // A turn result carries no retry statement, so the table's class stands. + "failed" => Some(CodexConditionOp::Raise { + fault: codex_fault( + codex_error_info_word( + message.pointer("/params/turn/error/codexErrorInfo"), + )?, + None, + observed_at_ms, + ), + source: CodexFaultSource::Typed, + }), + // `interrupted`, `inProgress`, and any future word: no evidence either way, so a + // standing fault stands. + _ => None, + } + } + // The account window is seat-level and names no thread at all. + "account/rateLimits/updated" => { + codex_rate_limit_condition(message.pointer("/params/rateLimits")?, observed_at_ms) + } + _ => None, + } +} + +/// What one `account/rateLimits/updated` snapshot states about the fault axis. +/// +/// The notification is a SPARSE rolling update: its own schema says nullable account metadata +/// "may be unavailable in a rolling update and does not clear a previously observed value". So +/// every absent-or-null field here means UNAVAILABLE, never recovered — an omitted or null +/// `rateLimitReachedType` states nothing at all, and no snapshot of this notification can clear +/// a standing fault. A window fault is escaped only by a positive turn completion, a new claim, +/// or a new incarnation. +/// +/// For the same reason the classification is built only from fields actually PRESENT: calling a +/// reached window self-clearing means claiming the allowance behind it is intact, which is a +/// claim only the credit metadata can support. +fn codex_rate_limit_condition( + snapshot: &Value, + observed_at_ms: u64, +) -> Option { + use crate::harness_state::{FaultCategory as Category, FaultReport, Recovery}; + // Only this field says a window was REACHED. Occupancy alone classifies nothing: the numeric + // record already carries it, and Codex account-window occupancy does not decide availability + // without its separate credit metadata. + let reached = snapshot + .get("rateLimitReachedType") + .and_then(Value::as_str)?; + // Present credit metadata, or nothing. `hasCredits` and `unlimited` are both required on the + // snapshot Codex sends, so their absence here is the sparse update omitting the whole object + // — unavailable, and unavailable proves nothing in either direction. + let credits = snapshot.pointer("/credits"); + let has_credits = credits + .and_then(|credits| credits.get("hasCredits")) + .and_then(Value::as_bool); + let unlimited = credits + .and_then(|credits| credits.get("unlimited")) + .and_then(Value::as_bool); + // The wall behind the window: a limit reached with the balance proven empty is not a window + // that rolls over, it is a purchase a person has to make. + if has_credits == Some(false) && unlimited == Some(false) { + return Some(CodexConditionOp::Raise { + fault: FaultReport::new(Category::Quota, Recovery::Human, observed_at_ms) + .with_code("codex/creditsDepleted"), + source: CodexFaultSource::Typed, + }); + } + let solvent = has_credits == Some(true) || unlimited == Some(true); + let (category, recovery) = match reached { + // A throttled window whose allowance is PROVEN intact is the one arm that recovers by + // itself. Where the credit metadata is unavailable the same word proves only that a + // limit was reached, so the recovery stays unsayable — which pages rather than waits. + "rate_limit_reached" if solvent => (Category::RateLimit, Recovery::Automatic), + "rate_limit_reached" => (Category::RateLimit, Recovery::Unknown), + "workspace_owner_usage_limit_reached" + | "workspace_member_usage_limit_reached" + | "workspace_owner_credits_depleted" + | "workspace_member_credits_depleted" => (Category::Quota, Recovery::Human), + // The gate pins every word above, so this arm is reachable only through a release the + // gate refused to admit. It keeps the field's own family and a recovery nobody claims to + // know. + _ => (Category::RateLimit, Recovery::Unknown), + }; + let mut fault = FaultReport::new(category, recovery, observed_at_ms) + .with_code(format!("codex/rateLimitReached.{reached}")); + // `resetsAt` is EPOCH SECONDS in the captured snapshot, and nullable there too. A deadline + // belongs only to a recovery that happens by itself, and one preceding its own observation + // makes the whole record indeterminate — so an absent, unreadable, or inverted stamp is + // simply omitted. + if recovery == Recovery::Automatic + && let Some(resets_at) = snapshot.pointer("/primary/resetsAt").and_then(Value::as_i64) + && let Ok(resets_at_ms) = u64::try_from(resets_at.saturating_mul(1_000)) + && resets_at_ms >= observed_at_ms + { + fault = fault.with_observation_due(resets_at_ms); + } + Some(CodexConditionOp::Raise { + fault, + source: CodexFaultSource::Typed, + }) +} + impl CodexObservedState { /// Driver-side projection into the generic observed-harness-state vocabulary (#162). `Held` is /// a delivery predicate — the complement of steerable — and never leaks into the published @@ -360,6 +658,122 @@ impl CodexObservedState { }, } } + + /// Driver-side projection into the version 3 tuple: the same activity and ask facts the + /// legacy projection reports, with two deliberate differences. + /// + /// A terminal PROVIDER error is no longer an `ended` seat. `ended` is the incarnation's last + /// word and belongs to the process-exit owner alone ([`run_connected`]); a Codex thread that + /// failed a turn is a LIVE seat that is not working, which is exactly what the fault axis is + /// for. The failure reaches the record through [`Self::state_condition`] and the typed error + /// frames instead — and for the same reason a thread-level `systemError`, which the legacy + /// projection had to withhold entirely, is now reported: the fault is the provable part. + /// + /// The condition axis rides `Unchanged` on every frame. An activity edge has observed + /// nothing about the provider, and the publish site states the axis once per incarnation. + fn harness_frame(&self) -> Option { + use crate::harness_state::{ + Activity, AskKind, ConditionReport, Frame, HumanAsk, InputBuffer, + }; + // This producer reads the app-server control stream and cannot see the composer. + let frame = + |state, ask| Frame::new(state, InputBuffer::Unknown, ConditionReport::Unchanged, ask); + Some(match self { + CodexObservedState::AwaitingStatus => return None, + CodexObservedState::Idle => frame(Activity::Idle, HumanAsk::None), + // No work is in progress and the failure rides the fault axis. `providerAuth` is the + // same word OpenCode's `ProviderAuthError` already publishes. + CodexObservedState::TerminalError { reason } => frame(Activity::Idle, HumanAsk::None) + .with_reason(match reason { + CodexTerminalError::SystemError => "systemError", + CodexTerminalError::ProviderAuthRejected => "providerAuth", + }), + CodexObservedState::Active { .. } => frame(Activity::Active, HumanAsk::None), + CodexObservedState::Held { reason, .. } => match reason { + // Review's edges are model-emitted items inside a running turn: plain activity, + // no human, no ask. The delivery hold is a separate axis, and the protocol gate + // pins exactly two active flags — so no Codex signal can mean a review ask. + CodexHoldReason::Review => { + frame(Activity::Active, HumanAsk::None).with_reason("review") + } + CodexHoldReason::WaitingOnApproval => { + frame(Activity::Active, HumanAsk::Pending(AskKind::Permission)) + .with_reason("waitingOnApproval") + } + CodexHoldReason::WaitingOnUserInput => { + frame(Activity::Active, HumanAsk::Pending(AskKind::Question)) + .with_reason("waitingOnUserInput") + } + // Compaction is work the harness does on itself, never a fault. + CodexHoldReason::Compaction => { + frame(Activity::Active, HumanAsk::None).with_reason("compaction") + } + CodexHoldReason::UnknownProtocol => { + frame(Activity::Active, HumanAsk::None).with_reason("unknownProtocol") + } + // Codex positively reported active; st2 merely cannot name a steerable turn. + CodexHoldReason::ActiveWithoutTurn => { + frame(Activity::Active, HumanAsk::None).with_reason("activeWithoutTurn") + } + CodexHoldReason::ConflictingTurn => { + frame(Activity::Active, HumanAsk::None).with_reason("conflictingTurn") + } + CodexHoldReason::SystemError => { + frame(Activity::Idle, HumanAsk::None).with_reason("systemError") + } + // Nothing here is provable, and no absence may derive a definite state. + CodexHoldReason::NotLoaded | CodexHoldReason::UnknownStatus => return None, + }, + }) + } + + /// What the observed state ITSELF states about the fault axis, for the two failures that + /// carry no native word: st2's own unclassified-protocol hold, and a thread-level system + /// error Codex reports without naming a cause. + /// + /// Every state that projects a frame and is not the unclassified-protocol hold carries that + /// hold's paired clear, because a state only ever leaves that hold by the hold's own signal — + /// a typed exit item or a later thread status. No state states a clear of anything else. + fn state_condition(&self, observed_at_ms: u64) -> Option { + use crate::harness_state::{FaultCategory, FaultReport, Recovery}; + let harness_fault = |code: &str, recovery| { + FaultReport::new(FaultCategory::Harness, recovery, observed_at_ms) + .with_code(format!("codex/{code}")) + }; + Some(match self { + CodexObservedState::Held { + reason: CodexHoldReason::UnknownProtocol, + .. + } => CodexConditionOp::Raise { + // st2's own plumbing is the fault: a signal this build cannot classify holds + // delivery, and a person has to teach the adapter the word. + fault: harness_fault("unknownProtocol", Recovery::Human), + source: CodexFaultSource::UnknownProtocol, + }, + CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + } + | CodexObservedState::Held { + reason: CodexHoldReason::SystemError, + .. + } => CodexConditionOp::Raise { + // Visible under the most conservative truthful category, with a recovery nobody + // claims to know. It yields to the typed word Codex sends for the same episode. + fault: harness_fault("systemError", Recovery::Unknown), + source: CodexFaultSource::Unclassified, + }, + CodexObservedState::TerminalError { + reason: CodexTerminalError::ProviderAuthRejected, + } => CodexConditionOp::Raise { + fault: codex_fault(CODEX_PROVIDER_AUTH_REJECTED, None, observed_at_ms), + source: CodexFaultSource::Typed, + }, + _ if self.harness_frame().is_some() => { + CodexConditionOp::ClearRaised(CodexFaultSource::UnknownProtocol) + } + _ => return None, + }) + } } #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -464,51 +878,13 @@ struct PendingCodexDelivery { method: CodexDeliveryMethod, } -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "camelCase")] -enum CodexDeliveryPhase { - Attempted, - Accepted, -} - -/// One durable FIFO delivery attempt. -/// -/// `Attempted` is written before transport. A replacement control connection reconciles that -/// ambiguous attempt against the resumed thread before it may send the client ID again. `Accepted` -/// is written only after the exact completed typed user-message event and remains until normal -/// message archive precedence removes the inbox entry. -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "camelCase", deny_unknown_fields)] -struct CodexDeliveryState { - schema: String, - agent: String, - runtime_id: String, - runtime_incarnation: String, - thread_id: String, - filename: String, - client_id: String, - phase: CodexDeliveryPhase, -} - -impl CodexDeliveryState { - fn attempted( - runtime: &CodexRuntime, - thread_id: String, - filename: String, - client_id: String, - ) -> Self { - Self { - schema: DELIVERY_STATE_SCHEMA.to_string(), - agent: runtime.agent.clone(), - runtime_id: runtime.runtime_id.clone(), - runtime_incarnation: runtime.incarnation.clone(), - thread_id, - filename, - client_id, - phase: CodexDeliveryPhase::Attempted, - } - } -} +// One durable FIFO delivery attempt lives in the shared `crate::delivery_ledger`, which grades +// Codex's two receipts honestly: the JSON-RPC result of `turn/start`/`turn/steer` is +// `transportAccepted`, and only the exact completed typed user message — live, or found in a +// resumed thread's history — is `consumed`. Codex has no storage receipt and no scheduler +// admission signal, so it can never write the phases in between, and consumption is its true +// ceiling: reaching it releases FIFO ownership. Ordinary message archive precedence, which is the +// recipient agent's own act, still removes the inbox entry. /// The exact codex-cli version whose Rust source settled the occupancy arithmetic below, read at /// tag `rust-v0.151.0` (tag object `d8673cb68e349c208659b986697773d3145dbb14`) because the Nix @@ -799,7 +1175,6 @@ struct RejectedCodexDelivery { struct CodexInboxDelivery { config: CodexDeliveryConfig, - state_path: PathBuf, runtime: CodexRuntime, wake: Receiver<()>, _watcher: Option, @@ -807,7 +1182,7 @@ struct CodexInboxDelivery { next_presence_refresh: Instant, head: Option, suppressed: bool, - state: Option, + ledger: delivery_ledger::Ledger, pending: Option, rejected: Option, next_request_id: u64, @@ -816,9 +1191,20 @@ struct CodexInboxDelivery { /// and stop the heartbeat, so a state the pump can no longer see ages out instead of staying /// artificially fresh. harness_evidence: bool, - /// A projected transition whose write failed, retried on the next pump pass before any - /// heartbeat may re-stamp the contradicted on-disk state. - pending_observation: Option, + /// A projected write that failed, retried on the next pump pass before any heartbeat may + /// re-stamp the contradicted on-disk state. + pending_write: Option, + /// Whether this incarnation has stated the condition axis at least once. Version 3 cannot + /// write `absent`, so the first frame states it and every later frame carries it forward. + condition_stated: bool, + /// The fault this incarnation raised and believes stands, beside the signal that raised it, + /// so a paired clear names the EXACT key rather than a category or a code prefix. + standing_fault: Option<(harness_state::FaultKey, CodexFaultSource)>, + /// A fault-axis operation waiting for the observation it attaches to, or retried after a + /// failed write. + pending_condition: Option, + /// What this producer can prove about the conversation bridge. + conversation: CodexConversationEvidence, /// The numeric axis's producer, beside the categorical one. `None` only where the record has /// nowhere safe to stage — observability never blocks a launch. context: Option, @@ -831,7 +1217,7 @@ struct CodexInboxDelivery { impl CodexInboxDelivery { fn new( config: CodexDeliveryConfig, - state_path: PathBuf, + legacy_path: PathBuf, runtime: CodexRuntime, ) -> Result { fs::create_dir_all(&config.inbox).with_context(|| { @@ -844,7 +1230,18 @@ impl CodexInboxDelivery { // Scoped to inbox + status: this pump's own process group writes runtime records (presence // refreshes, harness-state transitions) into the same agent dir, and those must not wake it. let watcher = crate::watch::watch_delivery_inputs(&config.agent_dir, wake_tx); - let state = load_delivery_state(&state_path, &config.identity, runtime.runtime_id())?; + // The ledger's authority is `delivery-ledger.json`; the v1 path beside it is the one-shot + // adoption source and the rollback floor, never read as authority again. Codex's v1 + // `Accepted` was written only from the typed completed user message, so it adopts as + // `consumed` — the one adoption that may suppress a duplicate on its own evidence. + let identity = config.identity.clone(); + let ledger = delivery_ledger::Ledger::open( + &legacy_path, + delivery_ledger::Harness::Codex.profile(), + &config.identity, + runtime.runtime_id(), + |thread, filename| stable_client_user_message_id(&identity, thread, filename), + ); // The pty session whose liveness vouches for the record is the wrapper's task: the // runtime ID names the pty registry entry, and only aliases the identity on // driver-expanded seats — a hand-authored seat may declare a different task ID. @@ -907,7 +1304,6 @@ impl CodexInboxDelivery { ); Ok(Self { config, - state_path, runtime, wake, _watcher: watcher, @@ -915,13 +1311,17 @@ impl CodexInboxDelivery { next_presence_refresh: Instant::now(), head: None, suppressed: false, - state, + ledger, pending: None, rejected: None, next_request_id: FIRST_DELIVERY_REQUEST_ID, harness_writer, harness_evidence: false, - pending_observation: None, + pending_write: None, + condition_stated: false, + standing_fault: None, + pending_condition: None, + conversation: CodexConversationEvidence::Unbound, context, diagnostics, }) @@ -933,16 +1333,49 @@ impl CodexInboxDelivery { /// retried before any heartbeat, so a stale on-disk state is never kept fresh in /// contradiction of the latest observation. fn observe_harness(&mut self, observed: &CodexObservedState) { - match observed.harness_observation() { - Some(observation) => self.publish_observation(observation), - None => { - // Evidence lost: stop heartbeating, drop anything pending (it predates the gap), - // and mark the stream discontinuous so a state restated after the gap opens a - // fresh transition instead of claiming continuity across an unobserved interval. - self.harness_evidence = false; - self.pending_observation = None; - self.harness_writer.interrupt(); + // Version 2 carries no condition, tagged-ask, or conversation axis, and this build still + // emits it: the legacy projection rides EXACTLY as it did before, and not one of the + // version 3 operations below is reachable until the writer emits version 3. + if !self.harness_writer.writes_condition_axis() { + match observed.harness_observation() { + Some(observation) => self.publish_observation(observation), + None => self.lose_evidence(), + } + return; + } + match observed.harness_frame() { + Some(frame) => { + self.publish_frame(frame); + self.settle_condition(observed); + } + None => self.lose_evidence(), + } + } + + /// Attach whatever the fault axis now has to say, once there is an observation to attach it + /// to. + /// + /// Two statements can ride one frame: what the observed state itself says, and the typed + /// native edge that arrived with it. State first, native second — the native word is the more + /// specific claim about the same episode, and Codex reports its unclassified thread status + /// first. Without landed evidence NEITHER is applied: a condition write restates the + /// record's activity verbatim, so attaching one now would refresh a stale state. The edge + /// queues instead, keeping the typed native statement where both are waiting. + fn settle_condition(&mut self, observed: &CodexObservedState) { + let stated = observed.state_condition(message::now_ms()); + if !self.harness_evidence { + if let Some(op) = stated + && self.pending_condition.is_none() + { + self.pending_condition = Some(op); } + return; + } + if let Some(op) = stated { + self.apply_condition(op); + } + if let Some(op) = self.pending_condition.take() { + self.apply_condition(op); } } @@ -986,39 +1419,231 @@ impl CodexInboxDelivery { } } + /// Record what one inbound control frame states about the fault axis. + /// + /// Frame-level like [`Self::observe_context`] and [`Self::observe_provider_auth`], for the + /// same reason: it reads fields — the typed error notification's `willRetry`, the account + /// window's reached type — that no branch below it looks at, and every one of them may + /// `continue`. It returns whether an edge was recorded, because a fault changes the observed + /// record while leaving `CodexObservedState` identical: the publish predicate must be the + /// union of the two, or a fault would never reach the record at all. + /// + /// The edge is applied after the frame it arrived with is published, because a condition + /// attaches to an observation. A newer edge replaces an unapplied one: it is the later + /// statement about the same axis. + fn observe_condition(&mut self, message: &Value, thread_id: &str) -> bool { + if !self.harness_writer.writes_condition_axis() { + return false; + } + let Some(op) = codex_condition_edge(message, thread_id, message::now_ms()) else { + return false; + }; + self.pending_condition = Some(op); + true + } + + /// The bound thread is this session's conversation identity, proven by the typed provider + /// response that named it. + fn bind_conversation(&mut self, thread_id: &str) { + self.conversation = CodexConversationEvidence::Probed { + thread_id: thread_id.to_string(), + verified_through_ms: message::now_ms(), + }; + } + + /// The identity exists and this connection cannot reach it: a resume the provider refused. + fn conversation_unreachable(&mut self, reason: &'static str) { + self.conversation = CodexConversationEvidence::Unreachable { reason }; + } + + /// The conversation bridge as this producer can PROVE it, or `None` while there is nothing + /// proven to state — which leaves the record's axis exactly as it stands rather than + /// claiming an absence. + fn conversation_state(&self) -> Option { + match &self.conversation { + CodexConversationEvidence::Unbound => None, + CodexConversationEvidence::Probed { + thread_id, + verified_through_ms, + } => Some(harness_state::ConversationState::Linked( + harness_state::ConversationClaim { + driver: "codex".to_string(), + conversation: thread_id.clone(), + // Codex rewrites thread context through `contextCompaction`, which this + // adapter observes on the same stream: a prefix read once may be gone. + history_mutability: harness_state::HistoryMutability::Rewritable, + // Both bind paths carry a positive typed provider probe: the accepted + // `thread/resume` response, or the owning TUI's own `thread/started`. + capability_evidence: harness_state::CapabilityEvidence::Probed, + verified_through_ms: *verified_through_ms, + }, + )), + CodexConversationEvidence::Unreachable { reason } => Some( + harness_state::ConversationState::Unavailable(Some((*reason).to_string())), + ), + } + } + fn publish_observation(&mut self, observation: harness_state::Observation) { match self.harness_writer.observe(observation.clone()) { Ok(()) => { self.harness_evidence = true; - self.pending_observation = None; + self.pending_write = None; } Err(_) => { self.harness_evidence = false; self.harness_writer.interrupt(); - self.pending_observation = Some(observation); + self.pending_write = Some(CodexPendingWrite::Legacy(observation)); } } } - fn write_state(&mut self, state: CodexDeliveryState) -> Result<()> { - atomic_json(&self.state_path, &state)?; - self.state = Some(state); - Ok(()) + /// Publish one resolved version 3 tuple. + /// + /// The condition axis is stated exactly ONCE per incarnation and carried forward untouched + /// afterwards. Version 3 cannot write `absent`, and the claim this construction made removed + /// whatever a predecessor left, so a fresh incarnation stating `clear` states exactly what it + /// has observed: no fault. Every later frame states `unchanged`, because an activity edge has + /// observed nothing about the provider and a frame restating `clear` would erase a standing + /// fault on every turn boundary. + fn publish_frame(&mut self, frame: harness_state::Frame) { + let mut frame = frame; + if !self.condition_stated { + frame.condition = harness_state::ConditionReport::Clear; + } + if let Some(conversation) = self.conversation_state() { + frame = frame.with_conversation(conversation); + } + match self.harness_writer.publish(frame.clone()) { + Ok(outcome) if outcome.accepted() => { + self.harness_evidence = true; + self.pending_write = None; + self.condition_stated = true; + } + // A refusal is an ownership or version verdict, not a transient failure: a later + // session owns the record, or it speaks a version this writer does not emit. No + // retry changes either, so this writer stops claiming evidence and lets the record + // age out instead of spinning on a write that cannot land. + Ok(refused) => { + tracing::warn!("st2 codex: observed-state publish refused: {refused:?}"); + self.lose_evidence(); + } + Err(error) => { + tracing::warn!("st2 codex: observed-state publish failed: {error:#}"); + self.harness_evidence = false; + self.harness_writer.interrupt(); + self.pending_write = Some(CodexPendingWrite::Frame(frame)); + } + } } - fn clear_state(&mut self) -> Result<()> { - remove_state_file(&self.state_path)?; - self.state = None; - Ok(()) + /// Evidence lost: stop heartbeating, drop anything pending (it predates the gap), and mark + /// the stream discontinuous so a state restated after the gap opens a fresh transition + /// instead of claiming continuity across an interval nobody observed. The fault ledger is + /// deliberately untouched: a condition st2 observed does not stop holding because the + /// activity axis went dark. + fn lose_evidence(&mut self) { + self.harness_evidence = false; + self.pending_write = None; + self.harness_writer.interrupt(); + } + + /// Apply one fault-axis operation against what this incarnation believes stands. + fn apply_condition(&mut self, op: CodexConditionOp) { + // Version 2 has nowhere to carry a condition and the writer refuses the operation as a + // value, so retrying it would spin forever. Nothing reaches here until version 3. + if !self.harness_writer.writes_condition_axis() { + return; + } + let outcome = match &op { + CodexConditionOp::Raise { fault, source } => { + // A failure the harness reported without classifying it never REPLACES a fault + // that already stands: a typed word for the same episode says strictly more, + // and Codex reports its unclassified thread status before the typed word. + if *source == CodexFaultSource::Unclassified && self.standing_fault.is_some() { + return; + } + self.harness_writer.raise_fault(fault.clone()) + } + CodexConditionOp::ClearRaised(source) => { + // A paired clear names the EXACT key its own signal raised. Where this + // incarnation raised nothing from that signal there is nothing to clear, and a + // clear derived from a category would let one signal silence another's fault. + let Some((key, raised)) = self.standing_fault.clone() else { + return; + }; + if raised != *source { + return; + } + self.harness_writer.clear_fault(key) + } + CodexConditionOp::ClearAll(proof) => { + if self.standing_fault.is_none() { + return; + } + self.harness_writer.clear_all(*proof) + } + }; + match outcome { + Ok(landed) if landed.accepted() => { + self.standing_fault = match &op { + CodexConditionOp::Raise { fault, source } => Some((fault.key(), *source)), + CodexConditionOp::ClearRaised(_) | CodexConditionOp::ClearAll(_) => None, + }; + } + // The fault this clear named is not the one standing — it was already replaced. The + // record is right and the ledger was stale: benign, and never retried. + Ok(harness_state::WriteOutcome::Refused( + harness_state::Refusal::ConditionMismatch { current }, + )) => { + tracing::debug!("st2 codex: paired clear matched no standing fault: {current:?}"); + } + // A condition attaches to an observation, and the first fault of a session can + // arrive before its first frame: retried on the next pump pass. + Ok(harness_state::WriteOutcome::Refused(harness_state::Refusal::Unobserved)) => { + self.pending_condition = Some(op); + } + Ok(refused) => { + tracing::warn!("st2 codex: fault write refused: {refused:?}"); + } + Err(error) => { + tracing::warn!("st2 codex: fault write failed: {error:#}"); + self.pending_condition = Some(op); + } + } + } + + /// Reconcile the ledger to what the recipient still has unread, then re-assert the rollback + /// floor. Archive precedence is the recipient agent's act and the only settlement authority: + /// an entry whose file left the inbox releases ownership, and this pump never moves a file. + fn reconcile_inbox(&mut self, unread: &[message::Message]) -> Result<()> { + self.ledger + .prune(|filename| unread.iter().any(|message| message.filename == filename))?; + // Re-asserted on every pass while an entry is outstanding: a crash exactly at the floor + // write would otherwise leave a landing with no v1-readable lower bound. + self.ledger.reassert_floor() } fn refresh_if_due(&mut self) -> Result<()> { let now = Instant::now(); // A pending transition retries on EVERY pump pass — its write failed once and the // on-disk record contradicts the latest observation until it lands; only the heartbeat - // is presence-cadence work. - if let Some(pending) = self.pending_observation.clone() { - self.publish_observation(pending); + // is presence-cadence work. The fault edge retries behind it, in that order: a + // condition attaches to an observation, so it cannot land before one exists. + if let Some(pending) = self.pending_write.clone() { + match pending { + CodexPendingWrite::Legacy(observation) => self.publish_observation(observation), + CodexPendingWrite::Frame(frame) => self.publish_frame(frame), + } + } + // A fault attaches to an OBSERVATION and a condition write carries the record's activity + // forward verbatim, so a queued edge waits while this session has no landed observation: + // applying one across an evidence gap would restate — and re-stamp — a state the pump + // could no longer see. It attaches on the pass after evidence returns. + if self.harness_evidence + && let Some(pending) = self.pending_condition.take() + { + self.apply_condition(pending); } if now >= self.next_presence_refresh { // This wrapper owns the live provider session. It therefore owns the presence lease. @@ -1037,13 +1662,7 @@ impl CodexInboxDelivery { return Ok(()); } let unread = message::list_inbox(&self.config.inbox)?; - if self.state.as_ref().is_some_and(|state| { - unread - .iter() - .all(|message| message.filename != state.filename) - }) { - self.clear_state()?; - } + self.reconcile_inbox(&unread)?; if self.rejected.as_ref().is_some_and(|rejected| { unread .iter() @@ -1063,15 +1682,30 @@ impl CodexInboxDelivery { if self.pending.is_some() || !state.subscribed || self.suppressed { return Ok(None); } - if let Some(delivery_state) = self.state.as_ref() { - if delivery_state.thread_id == state.thread_id { - return Ok(None); - } - // A newly selected thread is a different delivery binding. An old binding's receipt - // must neither suppress nor acknowledge delivery to this thread. - self.clear_state()?; + // Fail closed: an unreadable ledger holds and surfaces rather than guessing. It never + // refuses to start — a control connection that will not start delivers nothing at all. + // The operator-visible surface is the existing typed boundary — the transport is + // unavailable — and the raw reason stays in tracing, so no unbounded prose reaches the + // record. Restating it is coalesced by the publisher, so a held pass costs no write. + if let Some(reason) = self.ledger.quarantined().map(str::to_string) { + tracing::warn!("st2 codex: delivery ledger is quarantined: {reason}"); + self.diagnostics.publish( + driver_diagnostic::Stage::Delivery, + driver_diagnostic::Reason::DeliveryUnavailable, + driver_diagnostic::Source::PromptTransport, + ); + return Ok(None); + } + // A newly selected thread is a different delivery binding. An old binding's receipt must + // neither suppress nor acknowledge delivery to this thread. + if self + .ledger + .binding() + .is_some_and(|binding| binding != state.thread_id()) + { + self.ledger.rebind(state.thread_id())?; } - let Some(head) = self.head.as_ref() else { + let Some(head) = self.head.clone() else { return Ok(None); }; if self.rejected.as_ref().is_some_and(|rejected| { @@ -1079,6 +1713,19 @@ impl CodexInboxDelivery { }) { return Ok(None); } + // Exactly one delivery is outstanding at a time on this transport: an entry bound to some + // other file holds the pump until archive precedence resolves it, so a message arriving + // out of filename order can never open a second concurrent delivery. + if !self.ledger.entries().is_empty() && self.ledger.entry(&head.filename).is_none() { + return Ok(None); + } + // An attempt this pump already owns is held until evidence settles or refuses it. Only an + // authoritative "no" — a rejected request, or a resumed history proving the client ID + // never landed — authorizes sending the same identity again; a carried-forward v1 record + // never does. + if self.ledger.retry(&head.filename) != delivery_ledger::RetryDecision::Retry { + return Ok(None); + } let method = match &state.observed { CodexObservedState::Idle | CodexObservedState::TerminalError { .. } => { CodexDeliveryMethod::Start @@ -1102,16 +1749,28 @@ impl CodexInboxDelivery { &self.config.catalog_root, &self.config.this_host, &self.config.identity, - head, + &head, ); let request = codex_delivery_request(request_id, state.thread_id(), &client_id, &text, &method); - self.write_state(CodexDeliveryState::attempted( - &self.runtime, - state.thread_id().to_string(), - filename.clone(), - client_id, - ))?; + // Durable ownership before transport: the v1-readable floor first, then the ledger's own + // `attempted`. + self.ledger.begin(delivery_ledger::Begin { + filename: filename.clone(), + binding: state.thread_id().to_string(), + correlation: delivery_ledger::Correlation::native(client_id.clone()), + // Codex's typed receipt is a live frame, so an attempt is acknowledged only by the + // incarnation that made it; an older one is settled by the resume sweep instead. + incarnation: Some(self.runtime.incarnation().to_string()), + legacy_floor: delivery_ledger::codex_floor( + &self.config.identity, + self.runtime.runtime_id(), + self.runtime.incarnation(), + state.thread_id(), + &filename, + &client_id, + ), + })?; self.pending = Some(PendingCodexDelivery { request_id, filename, @@ -1134,13 +1793,13 @@ impl CodexInboxDelivery { .take() .context("Codex delivery is not pending")?; if message.get("error").is_some() { - if !self - .state - .as_ref() - .is_some_and(|state| state.phase == CodexDeliveryPhase::Accepted) - { - self.clear_state()?; - } + // The request itself was refused: an authoritative negative acknowledgement about + // this attempt, and the only thing that re-authorizes the same client ID here. A + // delivery that already reached its ceiling cannot be un-settled by a late error. + self.ledger.negative( + &pending.filename, + delivery_ledger::NegativeReceipt::Rejected, + )?; self.rejected = Some(RejectedCodexDelivery { filename: pending.filename, observed: observed.clone(), @@ -1159,6 +1818,12 @@ impl CodexInboxDelivery { ); } } + // The request returned a well-formed result. That is a fact about the call, never about + // the model, so it grades no higher than `transportAccepted`. + self.ledger.record( + &pending.filename, + delivery_ledger::Evidence::TransportAccepted, + )?; self.rejected = None; Ok(true) } @@ -1169,26 +1834,37 @@ impl CodexInboxDelivery { { return Ok(false); } - let Some(delivery_state) = self.state.as_ref() else { + let Some(client_id) = message + .pointer("/params/item/clientId") + .and_then(Value::as_str) + else { return Ok(false); }; if message.pointer("/params/threadId").and_then(Value::as_str) != Some(state.thread_id()) - || delivery_state.thread_id != state.thread_id() - || delivery_state.runtime_incarnation != self.runtime.incarnation() || state.runtime_incarnation != self.runtime.incarnation() - || message - .pointer("/params/item/clientId") - .and_then(Value::as_str) - != Some(delivery_state.client_id.as_str()) { return Ok(false); } - if delivery_state.phase == CodexDeliveryPhase::Accepted { - return Ok(true); + // One correlation may carry several inbox files, so one typed receipt settles every entry + // it delivered — each on its own monotone entry. + let settled: Vec = self + .ledger + .correlated(client_id) + .into_iter() + .filter(|filename| { + self.ledger.entry(filename).is_some_and(|entry| { + entry.binding == state.thread_id() + && entry.incarnation.as_deref() == Some(self.runtime.incarnation()) + }) + }) + .collect(); + if settled.is_empty() { + return Ok(false); + } + for filename in &settled { + self.ledger + .record(filename, delivery_ledger::Evidence::Consumed)?; } - let mut accepted = delivery_state.clone(); - accepted.phase = CodexDeliveryPhase::Accepted; - self.write_state(accepted)?; Ok(true) } @@ -1198,12 +1874,17 @@ impl CodexInboxDelivery { if message.get("error").is_some() { return Ok(()); } - let Some(delivery_state) = self.state.as_ref() else { - return Ok(()); - }; - if delivery_state.thread_id != state.thread_id() - || delivery_state.phase == CodexDeliveryPhase::Accepted - { + let unsettled: Vec<(String, String)> = self + .ledger + .entries() + .iter() + .filter(|entry| { + entry.binding == state.thread_id() + && entry.phase < delivery_ledger::Phase::Consumed + }) + .map(|entry| (entry.filename.clone(), entry.correlation.value.clone())) + .collect(); + if unsettled.is_empty() { return Ok(()); } let turns = message @@ -1212,70 +1893,30 @@ impl CodexInboxDelivery { .context( "Codex thread/resume response has no typed turn history for delivery recovery", )?; - let accepted = turns.iter().any(|turn| { - turn.get("items") - .and_then(Value::as_array) - .is_some_and(|items| { - items.iter().any(|item| { - item.get("type").and_then(Value::as_str) == Some("userMessage") - && item.get("clientId").and_then(Value::as_str) - == Some(delivery_state.client_id.as_str()) + for (filename, client_id) in unsettled { + let accepted = turns.iter().any(|turn| { + turn.get("items") + .and_then(Value::as_array) + .is_some_and(|items| { + items.iter().any(|item| { + item.get("type").and_then(Value::as_str) == Some("userMessage") + && item.get("clientId").and_then(Value::as_str) + == Some(client_id.as_str()) + }) }) - }) - }); - if accepted { - let mut state = delivery_state.clone(); - state.phase = CodexDeliveryPhase::Accepted; - self.write_state(state) - } else { - self.clear_state() + }); + if accepted { + self.ledger + .record(&filename, delivery_ledger::Evidence::Consumed)?; + } else { + // An authoritative resumed history without the client ID proves the pre-crash + // attempt never landed. That absence is the receipt — retained, not erased — + // and only it may authorize sending the same stable ID again. + self.ledger + .negative(&filename, delivery_ledger::NegativeReceipt::Absent)?; + } } - } -} - -fn load_delivery_state( - path: &Path, - identity: &str, - runtime_id: &str, -) -> Result> { - let bytes = match fs::read(path) { - Ok(bytes) => bytes, - Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), - Err(error) => return Err(error.into()), - }; - let state: CodexDeliveryState = serde_json::from_slice(&bytes) - .with_context(|| format!("reading Codex delivery state {}", path.display()))?; - anyhow::ensure!( - state.schema == DELIVERY_STATE_SCHEMA, - "Codex delivery state has unsupported schema '{}'", - state.schema - ); - anyhow::ensure!( - state.agent == identity && state.runtime_id == runtime_id, - "Codex delivery state belongs to a different runtime" - ); - anyhow::ensure!( - !state.runtime_incarnation.is_empty() - && !state.thread_id.is_empty() - && message::is_message_filename(&state.filename), - "Codex delivery state has an invalid runtime binding or filename" - ); - anyhow::ensure!( - state.client_id - == stable_client_user_message_id(identity, &state.thread_id, &state.filename), - "Codex delivery state client ID does not match its binding" - ); - Ok(Some(state)) -} - -fn remove_state_file(path: &Path) -> Result<()> { - match fs::remove_file(path) { - Ok(()) => { - File::open(path.parent().context("state file has no parent")?)?.sync_all()?; - Ok(()) - } - Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), - Err(error) => Err(error.into()), + Ok(()) } } @@ -2058,19 +2699,30 @@ fn run_connected( Some(runtime.runtime_id().to_string()), ) .with_session(runtime.incarnation()); + // Wrapper-owned, so it goes through the shared terminal write: a codex TUI that died before + // the pump published anything leaves the condition axis unstated behind the claim fence, and + // version 3 refuses that write rather than serializing an axis nobody stated. This process + // watched a child, not a provider turn, so `clear` is the only axis it may state. let _ = match &result { - Ok(TuiEnd::Exited(status)) => harness_writer.ended(describe_tui_exit(Some(*status))), - Ok(TuiEnd::Stopped(status)) => harness_writer.ended(describe_tui_exit(*status)), + Ok(TuiEnd::Exited(status)) => crate::provider_session::write_terminal( + &mut harness_writer, + &describe_tui_exit(Some(*status)), + None, + harness_state::ConditionReport::Clear, + ), + Ok(TuiEnd::Stopped(status)) => crate::provider_session::write_terminal( + &mut harness_writer, + &describe_tui_exit(*status), + None, + harness_state::ConditionReport::Clear, + ), Err(error) => { let observed_exit = tui.try_wait().ok().flatten(); - harness_writer.observe( - harness_state::Observation::new( - harness_state::Activity::Ended, - harness_state::BlockedOn::None, - harness_state::InputBuffer::Unknown, - ) - .with_exit(describe_tui_exit(observed_exit)) - .with_reason(format!("{error}")), + crate::provider_session::write_terminal( + &mut harness_writer, + &describe_tui_exit(observed_exit), + Some(&format!("{error}")), + harness_state::ConditionReport::Clear, ) } }; @@ -2801,7 +3453,7 @@ fn pump_control( let mut control_state: Option = None; let mut subscription_pending = false; let mut peer_closed = false; - let delivery_state_path = control_state_path.with_file_name("delivery-state.json"); + let delivery_state_path = control_state_path.with_file_name(delivery_ledger::LEGACY_FILE); let mut delivery = delivery .map(|config| { CodexInboxDelivery::new(config, delivery_state_path.clone(), runtime.clone()) @@ -2920,6 +3572,10 @@ fn pump_control( atomic_json(control_state_path, &bound) .context("persisting Codex control state")?; if let Some(delivery) = delivery.as_mut() { + // The resumed thread's identity is proven by the typed resume response + // this frame just accepted, so the conversation bridge is stated from + // that probe rather than from a thread id st2 merely remembers. + delivery.bind_conversation(bound.thread_id()); delivery.observe_harness(&bound.observed); } control_state = Some(bound); @@ -2945,6 +3601,9 @@ fn pump_control( atomic_json(control_state_path, &bound) .context("persisting Codex fresh control state")?; if let Some(delivery) = delivery.as_mut() { + // Same probe on the fresh path: the owning TUI's typed `thread/started` + // named this thread, which is what `binding_candidate` just read. + delivery.bind_conversation(bound.thread_id()); delivery.observe_harness(&bound.observed); } control_state = Some(bound); @@ -2963,12 +3622,18 @@ fn pump_control( // another reading on the next model response, roughly 10-15 per turn — and the record // is honest about the gap through `ageMs` meanwhile, which is cheaper than teaching the // binding handshake to hold observability frames it has no state to attribute yet. - if let Some(delivery) = delivery.as_mut() { - delivery.observe_context(&message, state.thread_id()); - // The credential axis, taken here for the same reason: it reads a typed turn - // result no branch below looks at, and every one of them may `continue`. - delivery.observe_provider_auth(&message, state.thread_id()); - } + let condition_changed = match delivery.as_mut() { + Some(delivery) => { + delivery.observe_context(&message, state.thread_id()); + // The credential axis, taken here for the same reason: it reads a typed turn + // result no branch below looks at, and every one of them may `continue`. + delivery.observe_provider_auth(&message, state.thread_id()); + // The fault axis, same reason once more — and it is the only reader of + // `willRetry` and of the account window's reached type. + delivery.observe_condition(&message, state.thread_id()) + } + None => false, + }; let delivery_response = match delivery.as_mut() { Some(delivery) => { delivery @@ -3002,7 +3667,14 @@ fn pump_control( } changed } - SubscriptionAcceptance::Deferred => false, + SubscriptionAcceptance::Deferred => { + // The conversation identity exists and this connection cannot reach it: + // stated as unavailable rather than as a link nobody could resume. + if let Some(delivery) = delivery.as_mut() { + delivery.conversation_unreachable(CODEX_NO_PERSISTED_ROLLOUT); + } + false + } } } else { state @@ -3012,9 +3684,17 @@ fn pump_control( if changed { atomic_json(control_state_path, state) .context("persisting Codex observed control state")?; - if let Some(delivery) = delivery.as_mut() { - delivery.observe_harness(&state.observed); - } + } + // A fault edge changes the observed record while leaving `CodexObservedState` + // identical, so the publish predicate is the union of the two. The control-state + // FILE stays gated on `changed` alone: it carries no condition axis, and the + // observed record is the one place a fault belongs. + if (changed || condition_changed) + && let Some(delivery) = delivery.as_mut() + { + delivery.observe_harness(&state.observed); + } + if changed { let _ = events.send(ControlEvent::Observed); } if !state.subscribed @@ -3471,16 +4151,104 @@ fn verify_codex_protocol_schemas(schemas: &CodexProtocolSchemas) -> Result<()> { "TurnError.codexErrorInfo does not use CodexErrorInfo" ); let error_words = schema_variant_words(definitions, "CodexErrorInfo")?; - for word in [ - CODEX_PROVIDER_AUTH_REJECTED, - "rateLimitExceeded", - "usageLimitExceeded", - ] { + // Every word the fault table forks on, not only the credential and quota trio: a merged or + // renamed arm must refuse the launch rather than silently reclassify a live seat's failure + // into the unclassified fall-through. The data-carrying arms (`httpConnectionFailed` and the + // response-stream family) are objects rather than enum words and stay unpinned on purpose: + // losing one costs precision, and the fall-through still keeps the failure visible. + for word in CODEX_CLASSIFIED_ERROR_WORDS { anyhow::ensure!( - error_words.contains(word), + error_words.contains(*word), "CodexErrorInfo has no '{word}' word" ); } + // The typed error NOTIFICATION, which carries a failure Codex reports beside — and before — + // any turn result. `willRetry` is required and overrides the recovery class of every word + // above, so a release that made it optional must refuse the launch instead of leaving st2 to + // guess whether a fault clears itself. + let notified_error = required_schema_path(definitions, "ErrorNotification", &["error"])?; + anyhow::ensure!( + notified_error == schema_definition(definitions, "TurnError")?, + "ErrorNotification.error does not use TurnError" + ); + for field in ["threadId", "turnId"] { + let schema = required_schema_path(definitions, "ErrorNotification", &[field])?; + require_type( + definitions, + schema, + "string", + &format!("ErrorNotification.{field}"), + )?; + } + require_property_type( + definitions, + schema_definition(definitions, "ErrorNotification")?, + "willRetry", + "boolean", + true, + "ErrorNotification", + )?; + // The account rate-limit snapshot: `rateLimitReachedType` is the only field that classifies + // a window as reached — occupancy alone classifies nothing — the credit metadata is the wall + // behind it, and `resetsAt` is the automatic recovery's deadline. + let snapshot = required_schema_path( + definitions, + "AccountRateLimitsUpdatedNotification", + &["rateLimits"], + )?; + anyhow::ensure!( + snapshot == schema_definition(definitions, "RateLimitSnapshot")?, + "AccountRateLimitsUpdatedNotification.rateLimits does not use RateLimitSnapshot" + ); + let reached = nullable_schema( + definitions, + property( + definitions, + snapshot, + "rateLimitReachedType", + "RateLimitSnapshot", + )?, + "RateLimitSnapshot.rateLimitReachedType", + )?; + anyhow::ensure!( + reached == schema_definition(definitions, "RateLimitReachedType")?, + "RateLimitSnapshot.rateLimitReachedType does not use RateLimitReachedType" + ); + let reached_words = schema_enum(definitions, "RateLimitReachedType")?; + for word in CODEX_RATE_LIMIT_REACHED_WORDS { + anyhow::ensure!( + reached_words.contains(*word), + "RateLimitReachedType has no '{word}' word" + ); + } + let credits = nullable_schema( + definitions, + property(definitions, snapshot, "credits", "RateLimitSnapshot")?, + "RateLimitSnapshot.credits", + )?; + for field in ["hasCredits", "unlimited"] { + require_property_type( + definitions, + credits, + field, + "boolean", + true, + "CreditsSnapshot", + )?; + } + let primary = nullable_schema( + definitions, + property(definitions, snapshot, "primary", "RateLimitSnapshot")?, + "RateLimitSnapshot.primary", + )?; + require_property_type( + definitions, + primary, + "resetsAt", + "integer", + false, + "RateLimitWindow", + )?; for notification in ["ItemStartedNotification", "ItemCompletedNotification"] { let item = required_schema_path(definitions, notification, &["item"])?; anyhow::ensure!( @@ -4535,12 +5303,7 @@ mod tests { "oneOf": [ { "type": "string", - "enum": [ - "usageLimitExceeded", - "rateLimitExceeded", - "unauthorized", - "other" - ] + "enum": CODEX_CLASSIFIED_ERROR_WORDS }, object_schema( &["httpConnectionFailed"], @@ -4549,6 +5312,74 @@ mod tests { ] }), ); + definitions.insert( + "ErrorNotification".into(), + object_schema( + &["error", "threadId", "turnId", "willRetry"], + &[ + ("error", reference("TurnError")), + ("threadId", json!({ "type": "string" })), + ("turnId", json!({ "type": "string" })), + ("willRetry", json!({ "type": "boolean" })), + ], + ), + ); + definitions.insert( + "RateLimitReachedType".into(), + json!({ + "type": "string", + "enum": CODEX_RATE_LIMIT_REACHED_WORDS + }), + ); + definitions.insert( + "RateLimitWindow".into(), + object_schema( + &["usedPercent"], + &[ + ("usedPercent", json!({ "type": "integer" })), + ("windowDurationMins", json!({ "type": ["integer", "null"] })), + ("resetsAt", json!({ "type": ["integer", "null"] })), + ], + ), + ); + definitions.insert( + "CreditsSnapshot".into(), + object_schema( + &["hasCredits", "unlimited"], + &[ + ("hasCredits", json!({ "type": "boolean" })), + ("unlimited", json!({ "type": "boolean" })), + ("balance", json!({ "type": ["string", "null"] })), + ], + ), + ); + definitions.insert( + "RateLimitSnapshot".into(), + object_schema( + &[], + &[ + ( + "primary", + json!({ "anyOf": [reference("RateLimitWindow"), { "type": "null" }] }), + ), + ( + "credits", + json!({ "anyOf": [reference("CreditsSnapshot"), { "type": "null" }] }), + ), + ( + "rateLimitReachedType", + json!({ "anyOf": [reference("RateLimitReachedType"), { "type": "null" }] }), + ), + ], + ), + ); + definitions.insert( + "AccountRateLimitsUpdatedNotification".into(), + object_schema( + &["rateLimits"], + &[("rateLimits", reference("RateLimitSnapshot"))], + ), + ); for notification in ["TurnStartedNotification", "TurnCompletedNotification"] { definitions.insert( notification.into(), @@ -4819,6 +5650,79 @@ mod tests { ); } + /// Branching on the typed `error` notification means depending on it: a release that drops + /// the notification, its required retry statement, or any word the fault table reads must + /// refuse the launch rather than silently stop classifying a live seat's failures. + #[test] + fn the_gate_pins_the_error_notification_and_every_classified_word() { + let mut missing = compatible_protocol_schemas(); + missing + .server_notifications + .get_mut("oneOf") + .unwrap() + .as_array_mut() + .unwrap() + .retain(|arm| { + arm.pointer("/properties/method/enum/0").and_then(Value::as_str) != Some("error") + }); + let error = verify_codex_protocol_schemas(&missing).unwrap_err(); + assert!( + format!("{error:#}").contains("server notification"), + "{error:#}" + ); + + let mut unretried = compatible_protocol_schemas(); + unretried + .protocol + .pointer_mut("/definitions/ErrorNotification/required") + .unwrap() + .as_array_mut() + .unwrap() + .retain(|name| name.as_str() != Some("willRetry")); + let error = verify_codex_protocol_schemas(&unretried).unwrap_err(); + assert!(format!("{error:#}").contains("willRetry"), "{error:#}"); + + for word in [ + "sessionBudgetExceeded", + "contextWindowExceeded", + "serverOverloaded", + "cyberPolicy", + "badRequest", + "sandboxError", + "other", + ] { + let mut dropped = compatible_protocol_schemas(); + dropped + .protocol + .pointer_mut("/definitions/CodexErrorInfo/oneOf/0/enum") + .unwrap() + .as_array_mut() + .unwrap() + .retain(|value| value.as_str() != Some(word)); + let error = verify_codex_protocol_schemas(&dropped).unwrap_err(); + assert!( + format!("{error:#}").contains(&format!("CodexErrorInfo has no '{word}' word")), + "{error:#}" + ); + } + + // The account window's own reached-type vocabulary, for exactly the same reason: the + // fault table forks on these words too. + let mut reached = compatible_protocol_schemas(); + reached + .protocol + .pointer_mut("/definitions/RateLimitReachedType/enum") + .unwrap() + .as_array_mut() + .unwrap() + .retain(|value| value.as_str() != Some("rate_limit_reached")); + let error = verify_codex_protocol_schemas(&reached).unwrap_err(); + assert!( + format!("{error:#}").contains("rate_limit_reached"), + "{error:#}" + ); + } + #[test] fn protocol_rejection_reaches_the_declared_supervisor_once() { let tmp = tempfile::tempdir().unwrap(); @@ -5318,12 +6222,26 @@ mod tests { fn inbox_delivery(root: &Path, config: CodexDeliveryConfig) -> CodexInboxDelivery { CodexInboxDelivery::new( config, - root.join("state/delivery-state.json"), + root.join("state").join(delivery_ledger::LEGACY_FILE), CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(), ) .unwrap() } + /// Read the ledger back through its own loader and the real correlation derivation: a test + /// that read the bytes directly would not notice a record the pump itself would refuse. + fn ledger_entry(root: &Path, filename: &str) -> Option { + delivery_ledger::Ledger::open( + &root.join("state").join(delivery_ledger::LEGACY_FILE), + delivery_ledger::Harness::Codex.profile(), + "h.worker", + "h.worker", + |thread, file| stable_client_user_message_id("h.worker", thread, file), + ) + .entry(filename) + .cloned() + } + fn acknowledge_tui_thread_loaded(events: &Receiver) { let ControlEvent::TuiThreadLoaded(acknowledge) = events.recv_timeout(Duration::from_secs(10)).unwrap() @@ -5825,6 +6743,690 @@ mod tests { ); } + /// The 23-frame #263 capture read on the version 3 axes. The seat that used to project + /// `ended` on a provider failure now reads a LIVE seat carrying one typed quota fault: + /// `ended` is the incarnation's last word and belongs to the process-exit owner alone. + #[test] + fn captured_usage_limit_is_a_live_quota_fault_not_an_ended_seat() { + use crate::harness_state::{Activity, FaultCategory, HumanAsk, Recovery}; + let frames = include_str!("../tests/fixtures/codex_usage_limit_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .collect::>(); + let observed_at_ms = 1_788_000_000_000; + let mut state = subscribed_state(CodexObservedState::AwaitingStatus); + let mut stated = Vec::new(); + for frame in &frames { + state.observe(frame).unwrap(); + if let Some(op) = codex_condition_edge(frame, "thread-main", observed_at_ms) { + stated.push(( + frame["method"].as_str().unwrap_or_default().to_string(), + op, + )); + } + // No frame of the capture may project a terminal record. + if let Some(projected) = state.observed().harness_frame() { + assert_ne!(projected.state, Activity::Ended, "{frame}"); + assert!(projected.exit.is_none(), "{frame}"); + } + } + + // Exactly two captured frames state the fault, in the captured order. + assert_eq!( + stated + .iter() + .map(|(method, _)| method.as_str()) + .collect::>(), + ["error", "turn/completed"] + ); + for (method, op) in &stated { + let CodexConditionOp::Raise { fault, .. } = op else { + panic!("{method} must raise a fault: {op:?}") + }; + assert_eq!(fault.category, FaultCategory::Quota, "{method}"); + assert_eq!( + fault.code.as_deref(), + Some("codex/usageLimitExceeded"), + "{method}" + ); + // `willRetry` is false in the capture and a failed turn states no retry at all: + // neither may report an automatic recovery nobody promised. + assert_eq!(fault.recovery, Recovery::Human, "{method}"); + assert_eq!(fault.observed_at_ms, observed_at_ms, "{method}"); + assert_eq!(fault.next_observation_due_ms, None, "{method}"); + } + + let projected = state + .observed() + .harness_frame() + .expect("a faulted seat is still an observable seat"); + assert_eq!(projected.state, Activity::Idle); + assert_eq!(projected.ask, HumanAsk::None); + assert_eq!(projected.reason.as_deref(), Some("systemError")); + } + + /// The two captures walk identical methods and differ in ONE word of `codexErrorInfo`. That + /// word is the whole fork: a credential a person must repair against an allowance that ran + /// out, neither read from prose. + #[test] + fn a_rejected_credential_and_an_exhausted_allowance_diverge_on_one_word() { + use crate::harness_state::FaultCategory; + let error_frame = |fixture: &str| { + fixture + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .find(|frame| frame["method"] == "error") + .expect("each capture carries the typed error notification") + }; + for (fixture, category, code) in [ + ( + include_str!("../tests/fixtures/codex_provider_auth_inbound.jsonl"), + FaultCategory::Authentication, + "codex/unauthorized", + ), + ( + include_str!("../tests/fixtures/codex_usage_limit_inbound.jsonl"), + FaultCategory::Quota, + "codex/usageLimitExceeded", + ), + ] { + let Some(CodexConditionOp::Raise { fault, source }) = + codex_condition_edge(&error_frame(fixture), "thread-main", 11) + else { + panic!("{code} must raise a fault") + }; + assert_eq!(fault.category, category, "{code}"); + assert_eq!(fault.code.as_deref(), Some(code)); + assert_eq!(source, CodexFaultSource::Typed, "{code}"); + } + } + + /// A blanket clear is the one operation that can silence a fault nobody saw resolve, so it + /// rides exactly one positive edge: a turn that ran to completion against the provider. + #[test] + fn only_a_completed_turn_clears_every_fault() { + let completed = |status: &str, error: Value| { + json!({ + "method": "turn/completed", + "params": { + "threadId": "thread-main", + "turn": { "id": "turn-1", "status": status, "error": error } + } + }) + }; + assert_eq!( + codex_condition_edge(&completed("completed", Value::Null), "thread-main", 1), + Some(CodexConditionOp::ClearAll( + harness_state::ProgressProof::TurnCompleted + )) + ); + assert!(matches!( + codex_condition_edge( + &completed("failed", json!({ "codexErrorInfo": "sandboxError" })), + "thread-main", + 1 + ), + Some(CodexConditionOp::Raise { .. }) + )); + // Neither an interrupted turn nor one this version cannot classify is evidence either + // way, so a standing fault stands. + for status in ["interrupted", "inProgress", "futureStatus"] { + assert_eq!( + codex_condition_edge(&completed(status, Value::Null), "thread-main", 1), + None, + "{status}" + ); + } + assert_eq!( + codex_condition_edge(&completed("failed", Value::Null), "thread-main", 1), + None, + "a failed turn with no typed error states nothing rather than guessing" + ); + assert_eq!( + codex_condition_edge(&completed("completed", Value::Null), "thread-other", 1), + None, + "another thread's progress is not this seat's evidence" + ); + } + + /// `account/rateLimits/updated` is a SPARSE rolling update: absent-or-null means the field + /// was unavailable in this update, never that anything recovered. So no snapshot of it + /// states a clear, and the classification is built only from fields actually present — an + /// automatic recovery in particular is a claim about the allowance behind the window, which + /// only the credit metadata can support. + #[test] + fn a_sparse_window_snapshot_states_only_what_it_carries() { + use crate::harness_state::{FaultCategory, Recovery}; + let captured = include_str!("../tests/fixtures/codex_token_usage_inbound.jsonl") + .lines() + .map(|line| serde_json::from_str::(line).unwrap()) + .find(|frame| frame["method"] == "account/rateLimits/updated") + .expect("the capture carries one rate-limit snapshot"); + assert_eq!( + codex_condition_edge(&captured, "thread-main", 1), + None, + "the captured present-and-null reached type is unavailable, not a recovery" + ); + + let observed_at_ms = 1_788_000_000_000; + let snapshot = |reached: Value, extra: Value| { + let mut limits = json!({ + "primary": { + "usedPercent": 100, + "windowDurationMins": 10_080, + "resetsAt": 1_788_452_803_i64 + }, + "rateLimitReachedType": reached + }); + if let Some(fields) = extra.as_object() { + for (key, value) in fields { + limits[key] = value.clone(); + } + } + json!({ + "method": "account/rateLimits/updated", + "params": { "rateLimits": limits } + }) + }; + let solvent = json!({ "credits": { "hasCredits": true, "unlimited": false } }); + + // Nothing about the fault axis is stated by a snapshot whose reached type is null, + // omitted, or carried only as occupancy — and never a clear, whatever stands. + for (label, frame) in [ + ("null", snapshot(Value::Null, solvent.clone())), + ( + "omitted", + json!({ + "method": "account/rateLimits/updated", + "params": { "rateLimits": { "primary": { "usedPercent": 44 } } } + }), + ), + ( + "no params", + json!({ "method": "account/rateLimits/updated" }), + ), + ] { + assert_eq!( + codex_condition_edge(&frame, "thread-main", observed_at_ms), + None, + "{label}" + ); + } + + // A reached window whose allowance is PROVEN intact is the one arm that recovers by + // itself, and `resetsAt` is EPOCH SECONDS in the capture. + let Some(CodexConditionOp::Raise { fault, source }) = codex_condition_edge( + &snapshot(json!("rate_limit_reached"), solvent), + "thread-main", + observed_at_ms, + ) else { + panic!("a reached window is a fault") + }; + assert_eq!(fault.category, FaultCategory::RateLimit); + assert_eq!( + fault.code.as_deref(), + Some("codex/rateLimitReached.rate_limit_reached") + ); + assert_eq!(fault.recovery, Recovery::Automatic); + assert_eq!(fault.next_observation_due_ms, Some(1_788_452_803_000)); + assert!(fault.next_observation_due_ms.unwrap() >= fault.observed_at_ms); + // Every rate-limit fault is a typed word Codex named, so the window's own snapshots can + // never name a paired clear for it: only a positive turn completion escapes it. + assert_eq!(source, CodexFaultSource::Typed); + + // The same word with the credit metadata UNAVAILABLE proves only that a limit was + // reached: no automatic recovery is claimed, and no deadline rides an unsayable one. + let Some(CodexConditionOp::Raise { fault, .. }) = codex_condition_edge( + &snapshot(json!("rate_limit_reached"), json!({})), + "thread-main", + observed_at_ms, + ) else { + panic!("a reached window is still a fault") + }; + assert_eq!(fault.category, FaultCategory::RateLimit); + assert_eq!(fault.recovery, Recovery::Unknown); + assert_eq!(fault.next_observation_due_ms, None); + + // A proven-empty balance is a purchase, not a window: quota, human, no deadline. + let Some(CodexConditionOp::Raise { fault, .. }) = codex_condition_edge( + &snapshot( + json!("rate_limit_reached"), + json!({ "credits": { "hasCredits": false, "unlimited": false } }), + ), + "thread-main", + observed_at_ms, + ) else { + panic!("a depleted balance is a fault") + }; + assert_eq!(fault.category, FaultCategory::Quota); + assert_eq!(fault.code.as_deref(), Some("codex/creditsDepleted")); + assert_eq!(fault.recovery, Recovery::Human); + assert_eq!(fault.next_observation_due_ms, None); + + // The words that name an exhausted allowance need no credit metadata to classify, and + // carry no deadline because nothing rolls over by itself. + for word in [ + "workspace_owner_usage_limit_reached", + "workspace_member_usage_limit_reached", + "workspace_owner_credits_depleted", + "workspace_member_credits_depleted", + ] { + let Some(CodexConditionOp::Raise { fault, .. }) = codex_condition_edge( + &snapshot(json!(word), json!({})), + "thread-main", + observed_at_ms, + ) else { + panic!("{word} must raise a fault") + }; + assert_eq!(fault.category, FaultCategory::Quota, "{word}"); + assert_eq!(fault.recovery, Recovery::Human, "{word}"); + assert_eq!(fault.next_observation_due_ms, None, "{word}"); + assert_eq!( + fault.code, + Some(format!("codex/rateLimitReached.{word}")), + "{word}" + ); + } + + // A word the gate never admitted keeps the field's own family with an unsayable + // recovery, and still no deadline. + let Some(CodexConditionOp::Raise { fault, .. }) = codex_condition_edge( + &snapshot(json!("future_limit_nobody_has_seen"), json!({})), + "thread-main", + observed_at_ms, + ) else { + panic!("an unadmitted word must stay visible") + }; + assert_eq!(fault.category, FaultCategory::RateLimit); + assert_eq!(fault.recovery, Recovery::Unknown); + assert_eq!(fault.next_observation_due_ms, None); + } + + /// A fault observed while the activity axis is dark stays QUEUED. A condition operation + /// restates the record's activity verbatim, so attaching one without a landed observation of + /// this session's would refresh a state the pump could no longer see — the exact stale-state + /// refresh the pending-transition retry exists to prevent. + #[test] + fn a_fault_queued_during_an_evidence_gap_waits_for_the_next_observation() { + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let mut delivery = inbox_delivery(tmp.path(), config); + let queued = CodexConditionOp::Raise { + fault: harness_state::FaultReport::new( + harness_state::FaultCategory::Quota, + harness_state::Recovery::Human, + 1_788_000_000_000, + ) + .with_code("codex/usageLimitExceeded"), + source: CodexFaultSource::Typed, + }; + delivery.pending_condition = Some(queued.clone()); + assert!( + !delivery.harness_evidence, + "no observation has landed for this session yet" + ); + + delivery.next_presence_refresh = Instant::now() + status::STATUS_REFRESH; + delivery.refresh_if_due().unwrap(); + assert_eq!( + delivery.pending_condition, + Some(queued.clone()), + "with nothing to attach to, the edge waits instead of restating an activity" + ); + + // An indeterminate projection loses evidence without dropping the queued fault: a + // condition st2 observed does not stop holding because the activity axis went dark. + delivery.observe_harness(&CodexObservedState::Held { + reason: CodexHoldReason::NotLoaded, + turn_id: None, + }); + assert!(!delivery.harness_evidence); + assert_eq!(delivery.pending_condition, Some(queued)); + + // Evidence returning is what lets it attach. On the version 2 wire the writer refuses + // the operation as a value, so the edge is consumed on that pass rather than retried + // forever against a record that cannot carry it. + delivery.observe_harness(&CodexObservedState::Idle); + assert!(delivery.harness_evidence); + delivery.next_presence_refresh = Instant::now() + status::STATUS_REFRESH; + delivery.refresh_if_due().unwrap(); + assert_eq!(delivery.pending_condition, None); + } + + /// `willRetry` is required on the typed error notification and is the only thing that may + /// call a Codex fault self-clearing. Every other word keeps the table's class, and a word + /// this build does not know stays visible instead of being folded into a neighbour. + #[test] + fn will_retry_flips_the_recovery_class_and_an_unknown_word_stays_visible() { + use crate::harness_state::{FaultCategory, Recovery}; + let notification = |word: Value, will_retry: bool| { + json!({ + "method": "error", + "params": { + "error": { "codexErrorInfo": word }, + "willRetry": will_retry, + "threadId": "thread-main", + "turnId": "turn-1" + } + }) + }; + let fault_of = |frame: &Value| match codex_condition_edge(frame, "thread-main", 7) { + Some(CodexConditionOp::Raise { fault, .. }) => fault, + other => panic!("{other:?} is not a raised fault"), + }; + assert_eq!( + fault_of(¬ification(json!("rateLimitExceeded"), true)).recovery, + Recovery::Automatic + ); + assert_eq!( + fault_of(¬ification(json!("rateLimitExceeded"), false)).recovery, + Recovery::Human, + "a refused retry keeps the table's own class" + ); + + for (word, category) in [ + ("unauthorized", FaultCategory::Authentication), + ("usageLimitExceeded", FaultCategory::Quota), + ("sessionBudgetExceeded", FaultCategory::Quota), + ("rateLimitExceeded", FaultCategory::RateLimit), + ("serverOverloaded", FaultCategory::Provider), + ("internalServerError", FaultCategory::Provider), + ("contextWindowExceeded", FaultCategory::Context), + ("cyberPolicy", FaultCategory::Policy), + ("misalignmentPolicyViolation", FaultCategory::Policy), + ("badRequest", FaultCategory::Configuration), + ("threadRollbackFailed", FaultCategory::Harness), + ("sandboxError", FaultCategory::Harness), + ] { + let fault = fault_of(¬ification(json!(word), false)); + assert_eq!(fault.category, category, "{word}"); + assert_eq!(fault.code, Some(format!("codex/{word}")), "{word}"); + } + + // The data-carrying arms are objects, not strings: the single key IS the word, and an + // `as_str`-only read would silently drop that whole family. + let object = fault_of(¬ification( + json!({ "httpConnectionFailed": { "httpStatusCode": 503 } }), + false, + )); + assert_eq!(object.category, FaultCategory::Provider); + assert_eq!(object.code.as_deref(), Some("codex/httpConnectionFailed")); + assert_eq!( + fault_of(¬ification(json!({ "activeTurnNotSteerable": {} }), false)).category, + FaultCategory::Configuration + ); + + for word in ["other", "futureFailureNobodyHasSeen"] { + let fault = fault_of(¬ification(json!(word), false)); + assert_eq!(fault.category, FaultCategory::Harness, "{word}"); + assert_eq!(fault.recovery, Recovery::Unknown, "{word}"); + assert_eq!(fault.code, Some(format!("codex/{word}")), "{word}"); + } + assert_eq!( + fault_of(¬ification(json!("other"), true)).recovery, + Recovery::Automatic, + "an unknown word the harness promises to retry is automatic on that promise" + ); + + assert_eq!( + codex_condition_edge(¬ification(json!("unauthorized"), false), "thread-other", 7), + None, + "another thread's error is not this seat's fault" + ); + assert_eq!( + codex_condition_edge(¬ification(json!({}), false), "thread-main", 7), + None, + "a union shape this build cannot read states nothing" + ); + } + + /// The projection never writes the incarnation's last word: `ended` has exactly two sites, + /// both in `run_connected`, and both observe a real process exit. It also cannot spell a + /// review ask — the protocol gate pins `ThreadActiveFlag` to exactly two words, neither of + /// which means review. + #[test] + fn no_codex_frame_reports_an_ended_seat_or_a_review_ask() { + use crate::harness_state::{Activity, AskKind, ConditionReport, HumanAsk}; + let held = |reason| CodexObservedState::Held { + reason, + turn_id: Some("turn-1".into()), + }; + let mut projected = vec![ + ( + "awaitingStatus".to_string(), + CodexObservedState::AwaitingStatus.harness_frame(), + ), + ("idle".to_string(), CodexObservedState::Idle.harness_frame()), + ( + "active".to_string(), + CodexObservedState::Active { + turn_id: "turn-1".into(), + } + .harness_frame(), + ), + ]; + for reason in [ + CodexTerminalError::SystemError, + CodexTerminalError::ProviderAuthRejected, + ] { + projected.push(( + format!("terminal {reason:?}"), + CodexObservedState::TerminalError { reason }.harness_frame(), + )); + } + for reason in [ + CodexHoldReason::ActiveWithoutTurn, + CodexHoldReason::ConflictingTurn, + CodexHoldReason::Review, + CodexHoldReason::Compaction, + CodexHoldReason::UnknownProtocol, + CodexHoldReason::NotLoaded, + CodexHoldReason::SystemError, + CodexHoldReason::UnknownStatus, + CodexHoldReason::WaitingOnApproval, + CodexHoldReason::WaitingOnUserInput, + ] { + projected.push((format!("held {reason:?}"), held(reason).harness_frame())); + } + for (label, frame) in &projected { + let Some(frame) = frame else { continue }; + assert_ne!(frame.state, Activity::Ended, "{label}"); + assert!(frame.exit.is_none(), "{label}"); + assert_ne!(frame.ask, HumanAsk::Pending(AskKind::Review), "{label}"); + assert_ne!( + frame.ask, + HumanAsk::Unknown, + "{label}: this producer does see the ask surface" + ); + assert_eq!( + frame.condition, + ConditionReport::Unchanged, + "{label}: an activity edge states nothing about the fault axis" + ); + } + + // The rows nothing can prove stay withheld, exactly as the legacy projection withholds + // them: no absence may derive a definite state. + for state in [ + CodexObservedState::AwaitingStatus, + held(CodexHoldReason::NotLoaded), + held(CodexHoldReason::UnknownStatus), + ] { + assert_eq!(state.harness_frame(), None, "{state:?}"); + } + // Codex reported a thread-level system error: the seat is live and not working, and the + // failure rides the fault axis instead of a fabricated terminal record. + assert_eq!( + held(CodexHoldReason::SystemError) + .harness_frame() + .unwrap() + .state, + Activity::Idle + ); + assert_eq!( + held(CodexHoldReason::WaitingOnApproval) + .harness_frame() + .unwrap() + .ask, + HumanAsk::Pending(AskKind::Permission) + ); + assert_eq!( + held(CodexHoldReason::WaitingOnUserInput) + .harness_frame() + .unwrap() + .ask, + HumanAsk::Pending(AskKind::Question) + ); + } + + /// Two failures have no native word at all: st2's own unclassified-protocol hold, and a + /// thread-level system error Codex reports without naming a cause. Both stay visible, and + /// each is escaped only by the signal that raised it. + #[test] + fn a_faulted_thread_state_states_its_own_fault_without_a_native_word() { + use crate::harness_state::{FaultCategory, Recovery}; + let held = |reason| CodexObservedState::Held { + reason, + turn_id: None, + }; + let Some(CodexConditionOp::Raise { fault, source }) = + held(CodexHoldReason::UnknownProtocol).state_condition(9) + else { + panic!("an unclassified protocol signal is a harness fault") + }; + assert_eq!(fault.category, FaultCategory::Harness); + assert_eq!(fault.code.as_deref(), Some("codex/unknownProtocol")); + assert_eq!(fault.recovery, Recovery::Human); + assert_eq!(source, CodexFaultSource::UnknownProtocol); + assert_eq!( + CodexObservedState::Idle.state_condition(9), + Some(CodexConditionOp::ClearRaised( + CodexFaultSource::UnknownProtocol + )), + "leaving the hold by its own signal is that fault's exact paired clear" + ); + + for state in [ + held(CodexHoldReason::SystemError), + CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + }, + ] { + let Some(CodexConditionOp::Raise { fault, source }) = state.state_condition(9) else { + panic!("{state:?} must stay visible") + }; + assert_eq!(fault.category, FaultCategory::Harness, "{state:?}"); + assert_eq!(fault.code.as_deref(), Some("codex/systemError"), "{state:?}"); + assert_eq!(fault.recovery, Recovery::Unknown, "{state:?}"); + assert_eq!(source, CodexFaultSource::Unclassified, "{state:?}"); + } + + let Some(CodexConditionOp::Raise { fault, source }) = CodexObservedState::TerminalError { + reason: CodexTerminalError::ProviderAuthRejected, + } + .state_condition(9) + else { + panic!("the state machine already classified this credential") + }; + assert_eq!(fault.category, FaultCategory::Authentication); + assert_eq!(fault.code.as_deref(), Some("codex/unauthorized")); + assert_eq!(fault.recovery, Recovery::Human); + assert_eq!(source, CodexFaultSource::Typed); + + for state in [ + CodexObservedState::AwaitingStatus, + held(CodexHoldReason::NotLoaded), + held(CodexHoldReason::UnknownStatus), + ] { + assert_eq!(state.state_condition(9), None, "{state:?}"); + } + } + + /// The conversation bridge is stated only from typed provider identity this driver already + /// holds, and a resume the provider refused is stated as unavailable rather than linked. + #[test] + fn a_bound_thread_states_a_probed_conversation_and_a_refused_resume_does_not() { + use crate::harness_state::{CapabilityEvidence, ConversationState, HistoryMutability}; + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let mut delivery = inbox_delivery(tmp.path(), config); + assert_eq!( + delivery.conversation_state(), + None, + "nothing is claimed before a thread is bound" + ); + + delivery.bind_conversation("thread-main"); + let Some(ConversationState::Linked(link)) = delivery.conversation_state() else { + panic!("a bound thread is a linked conversation") + }; + assert_eq!(link.driver, "codex"); + assert_eq!(link.conversation, "thread-main"); + assert_eq!(link.history_mutability, HistoryMutability::Rewritable); + assert_eq!(link.capability_evidence, CapabilityEvidence::Probed); + assert!(link.verified_through_ms > 0); + + delivery.conversation_unreachable(CODEX_NO_PERSISTED_ROLLOUT); + assert_eq!( + delivery.conversation_state(), + Some(ConversationState::Unavailable(Some( + CODEX_NO_PERSISTED_ROLLOUT.to_string() + ))) + ); + } + + /// The version 2 wire has no condition axis: on a writer emitting it, every version 3 + /// operation stays inert and the legacy projection — including its terminal reading of a + /// provider failure — is exactly what it was. Production emits version 3 now, so the version + /// 2 writer is substituted through the test-only seam; this is the shape of every legacy + /// record still on disk. + #[test] + fn version_two_delivery_keeps_the_legacy_projection_and_states_no_condition() { + use crate::harness_state::{self, Activity}; + let tmp = tempfile::tempdir().unwrap(); + let config = delivery_config(tmp.path()); + let record_path = harness_state::harness_state_path(&config.agent_dir); + let mut delivery = inbox_delivery(tmp.path(), config); + // The whole session on the version 2 wire. `CodexInboxDelivery::new` performed a + // PRODUCTION claim, so the seam substitutes the writer's version and the now-foreign + // version 3 fence that claim wrote is removed: a version 2 build would have written a + // version 2 fence there, and what this test pins is what the pump writes afterwards. + delivery + .harness_writer + .emit_schema(harness_state::SCHEMA_V2); + std::fs::remove_file(&record_path).unwrap(); + assert!( + !delivery.harness_writer.writes_condition_axis(), + "the legacy projection is reached through the version 2 seam" + ); + + let error = json!({ + "method": "error", + "params": { + "error": { "codexErrorInfo": "usageLimitExceeded" }, + "willRetry": false, + "threadId": "thread-main", + "turnId": "turn-failed" + } + }); + assert!( + !delivery.observe_condition(&error, "thread-main"), + "a condition edge cannot change a record whose wire has no condition axis" + ); + assert_eq!(delivery.pending_condition, None); + + delivery.observe_harness(&CodexObservedState::TerminalError { + reason: CodexTerminalError::SystemError, + }); + let observed = harness_state::read(&record_path, None).expect("record written"); + assert_eq!(observed.state, Activity::Ended); + assert_eq!(observed.reason.as_deref(), Some("systemError")); + assert!( + delivery.pending_write.is_none(), + "the legacy write landed exactly as before" + ); + } + #[test] fn idle_session_refreshes_stale_presence_without_inbox_activity() { let tmp = tempfile::tempdir().unwrap(); @@ -5930,8 +7532,8 @@ mod tests { let idle = subscribed_state(CodexObservedState::Idle); let request = delivery.maybe_request(&idle).unwrap().unwrap(); assert_eq!( - delivery.state.as_ref().unwrap().phase, - CodexDeliveryPhase::Attempted, + delivery.ledger.entry(&filename).unwrap().phase, + delivery_ledger::Phase::Attempted, "submission ownership is durable before transport" ); assert!( @@ -5943,9 +7545,9 @@ mod tests { .unwrap() ); assert_eq!( - delivery.state.as_ref().unwrap().phase, - CodexDeliveryPhase::Attempted, - "JSON success is not typed acceptance" + delivery.ledger.entry(&filename).unwrap().phase, + delivery_ledger::Phase::TransportAccepted, + "a well-formed JSON result is transport, never typed acceptance" ); assert_eq!(delivery.maybe_request(&idle).unwrap(), None); assert!(config.inbox.join(&filename).is_file()); @@ -6023,11 +7625,12 @@ mod tests { .unwrap() ); assert_eq!( - load_delivery_state(&state_path, "h.worker", "h.worker") - .unwrap() - .unwrap() - .phase, - CodexDeliveryPhase::Accepted + ledger_entry(tmp.path(), &filename).unwrap().phase, + delivery_ledger::Phase::Consumed + ); + assert!( + !state_path.exists(), + "consumption released the v1 rollback floor" ); assert!(config.inbox.join(&filename).is_file()); @@ -6049,7 +7652,11 @@ mod tests { assert_eq!(replacement.maybe_request(&idle).unwrap(), None); assert!( !state_path.exists(), - "archive precedence clears the receipt" + "no outstanding delivery, no rollback floor" + ); + assert!( + ledger_entry(tmp.path(), &filename).is_none(), + "archive precedence — the recipient agent's own act — releases the ledger entry" ); } @@ -6100,21 +7707,42 @@ mod tests { ) .unwrap(); assert_eq!( - recovered.state.as_ref().unwrap().phase, - CodexDeliveryPhase::Accepted + recovered.ledger.entry(&filename).unwrap().phase, + delivery_ledger::Phase::Consumed, + "a resumed history carrying the client ID is the same typed receipt, found late" ); assert_eq!(recovered.maybe_request(&idle).unwrap(), None); assert!(config.inbox.join(&filename).is_file()); - // An authoritative resumed history without the client ID proves that the pre-send record - // did not reach typed acceptance. Only then may the same stable ID be retried. - recovered.state.as_mut().unwrap().phase = CodexDeliveryPhase::Attempted; - atomic_json( - &tmp.path().join("state/delivery-state.json"), - recovered.state.as_ref().unwrap(), + // An authoritative resumed history WITHOUT the client ID proves the pre-crash attempt + // never landed. Only that absence may re-authorize the same stable ID — so it needs its + // own scenario, because the delivery above is settled and can never be un-settled. + let absent_tmp = tempfile::tempdir().unwrap(); + let absent_config = delivery_config(absent_tmp.path()); + let absent_filename = message::send_to_inbox( + &absent_config.inbox, + "h.sender", + Some("absent"), + None, + &[], + "body", ) .unwrap(); - recovered + let mut attempted = inbox_delivery(absent_tmp.path(), absent_config.clone()); + let absent_client_id = attempted.maybe_request(&idle).unwrap().unwrap() + ["params"]["clientUserMessageId"] + .as_str() + .unwrap() + .to_string(); + drop(attempted); + + let mut replacement = inbox_delivery(absent_tmp.path(), absent_config); + assert_eq!( + replacement.maybe_request(&idle).unwrap(), + None, + "an ambiguous attempt is held and surfaced, never replayed on its own" + ); + replacement .reconcile_resume( &json!({ "id": CONTROL_SUBSCRIBE_REQUEST_ID, @@ -6123,20 +7751,35 @@ mod tests { &idle, ) .unwrap(); - assert!(recovered.state.is_none()); - let retry = recovered.maybe_request(&idle).unwrap().unwrap(); - assert_eq!(retry["params"]["clientUserMessageId"], client_id); + assert_eq!( + replacement + .ledger + .entry(&absent_filename) + .unwrap() + .negative, + Some(delivery_ledger::NegativeReceipt::Absent), + "the absence is retained as evidence, not erased" + ); + let retry = replacement.maybe_request(&idle).unwrap().unwrap(); + assert_eq!(retry["params"]["clientUserMessageId"], absent_client_id); } + /// v1 refused to START on a record whose client ID contradicted its own binding, which is the + /// worst available failure: the control connection never comes up and nothing is delivered at + /// all. The ledger fails closed instead — it starts, authorizes no transport, retains the + /// reason, and destroys no evidence. #[test] - fn malformed_delivery_state_fails_closed() { + fn a_tampered_v1_record_fails_closed_without_refusing_to_start() { let tmp = tempfile::tempdir().unwrap(); let config = delivery_config(tmp.path()); + let filename = + message::send_to_inbox(&config.inbox, "h.sender", Some("held"), None, &[], "body") + .unwrap(); let state_path = tmp.path().join("state/delivery-state.json"); atomic_json( &state_path, &json!({ - "schema": DELIVERY_STATE_SCHEMA, + "schema": delivery_ledger::CODEX_LEGACY_SCHEMA, "agent": "h.worker", "runtimeId": "h.worker", "runtimeIncarnation": "incarnation-test", @@ -6147,15 +7790,36 @@ mod tests { }), ) .unwrap(); - let error = match CodexInboxDelivery::new( - config, - state_path, - CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(), - ) { - Ok(_) => panic!("accepted malformed delivery state"), - Err(error) => error, + let mut delivery = inbox_delivery(tmp.path(), config.clone()); + assert!( + delivery + .ledger + .quarantined() + .is_some_and(|reason| reason.contains("does not match its binding")), + "the refusal names itself" + ); + assert_eq!( + delivery + .maybe_request(&subscribed_state(CodexObservedState::Idle)) + .unwrap(), + None, + "a quarantined ledger authorizes no transport" + ); + // The operator-visible surface is the existing typed delivery boundary. No new + // vocabulary, and the raw quarantine reason stays in tracing rather than the record. + let driver_diagnostic::Observed::Failure(failure) = + driver_diagnostic::read(&driver_diagnostic::path(&config.agent_dir)) + else { + panic!("a quarantined ledger must be diagnosed") }; - assert!(error.to_string().contains("client ID does not match")); + assert_eq!(failure.stage, driver_diagnostic::Stage::Delivery); + assert_eq!(failure.reason, driver_diagnostic::Reason::DeliveryUnavailable); + assert_eq!(failure.source, driver_diagnostic::Source::PromptTransport); + assert!( + state_path.exists(), + "a record we refuse to read is not a record we may destroy" + ); + assert!(config.inbox.join(&filename).is_file()); } #[test] @@ -6279,17 +7943,10 @@ mod tests { server.join().unwrap(); let _ = shutdown.shutdown(Shutdown::Both); pump.join().unwrap(); - assert!(delivery_config(tmp.path()).inbox.join(filename).is_file()); + assert!(delivery_config(tmp.path()).inbox.join(&filename).is_file()); assert_eq!( - load_delivery_state( - &tmp.path().join("state/delivery-state.json"), - "h.worker", - "h.worker", - ) - .unwrap() - .unwrap() - .phase, - CodexDeliveryPhase::Accepted + ledger_entry(tmp.path(), &filename).unwrap().phase, + delivery_ledger::Phase::Consumed ); } @@ -6400,14 +8057,20 @@ mod tests { let client_id = stable_client_user_message_id("h.worker", "thread-main", &filename); let prior_runtime = CodexRuntime::fresh("h.worker".into(), "h.worker".into()).unwrap(); let delivery_state_path = tmp.path().join("state/delivery-state.json"); + // The v1 record a prior binary left, in exactly the shape it wrote it. This is the + // migration boundary: no ledger exists, so the new pump adopts this attempt. atomic_json( &delivery_state_path, - &CodexDeliveryState::attempted( - &prior_runtime, - "thread-main".into(), - filename.clone(), - client_id.clone(), - ), + &json!({ + "schema": delivery_ledger::CODEX_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "runtimeIncarnation": prior_runtime.incarnation(), + "threadId": "thread-main", + "filename": &filename, + "clientId": &client_id, + "phase": "attempted" + }), ) .unwrap(); @@ -6512,12 +8175,19 @@ mod tests { let _ = shutdown.shutdown(Shutdown::Both); pump.join().unwrap(); - let recovered = load_delivery_state(&delivery_state_path, "h.worker", "h.worker") - .unwrap() - .unwrap(); - assert_eq!(recovered.phase, CodexDeliveryPhase::Accepted); - assert_eq!(recovered.client_id, client_id); - assert!(delivery_config(tmp.path()).inbox.join(filename).is_file()); + let recovered = ledger_entry(tmp.path(), &filename).expect("the adopted attempt survives"); + assert_eq!(recovered.phase, delivery_ledger::Phase::Consumed); + assert_eq!(recovered.correlation.value, client_id); + assert_eq!( + recovered.adopted_from.as_deref(), + Some(delivery_ledger::CODEX_LEGACY_SCHEMA), + "the receipt landed on the entry carried forward from v1, not on a second attempt" + ); + assert!( + !delivery_state_path.exists(), + "consumption released the v1 record it was adopted from" + ); + assert!(delivery_config(tmp.path()).inbox.join(&filename).is_file()); } #[test] diff --git a/src/delivery_ledger.rs b/src/delivery_ledger.rs new file mode 100644 index 00000000..8b5e4960 --- /dev/null +++ b/src/delivery_ledger.rs @@ -0,0 +1,1784 @@ +//! The shared native-delivery ledger. +//! +//! Every native transport used to keep its own single-binding `delivery-state.json` holding one +//! `{binding, filename, correlation, phase in {Attempted, Accepted}}`. `Accepted` meant something +//! different in each driver — a typed completed user message on Codex (the model consumed it), a +//! `GET 200` on OpenCode (the server merely stored it) — and nothing on disk said which. This +//! module replaces that with one evidence-graded ledger: +//! +//! * **Fresh namespace.** The record lives at `/delivery-ledger.json` under schema +//! [`LEDGER_SCHEMA`]. The v1 path is never written as authority again. An in-place schema bump +//! was measured to be unrecoverable: v1's Codex loader `ensure!`s its schema and denies unknown +//! fields, so a v2 body there refuses to start the control connection at all, and v1's OpenCode +//! loader silently discards it and re-POSTs the same message id — which appends its parts a +//! second time on 1.18.19. +//! * **Monotone phases.** [`Phase`] orders `attempted < transportAccepted < persisted < admitted +//! < consumed`. A write that would lower a phase is refused, so a restart at any persistence +//! boundary can only ever read a true lower bound of what happened. +//! * **Per-filename identity under a shared correlation.** Entries are keyed by message filename +//! and carry the durable native correlation they were transported under. N filenames may share +//! one correlation value, so a bounded multi-message FIFO prefix is N monotone entries rather +//! than one record naming one file. +//! * **Honest adapter grading.** A [`Profile`] declares what its harness can actually prove and +//! what it could prove if every signal were wired. Evidence a harness cannot honestly produce is +//! refused rather than recorded, and OpenCode's `persisted` therefore never reads as consumption. +//! * **One-shot label-downgrading adoption.** A pre-existing v1 record is carried forward exactly +//! once, at the evidence it actually proved: Codex `Accepted` → [`Phase::Consumed`], OpenCode +//! `Accepted` → [`Phase::Persisted`], either `Attempted` → [`Phase::Attempted`]. Adoption is +//! recorded as such and authorizes no transport by itself. +//! * **A rollback-readable floor.** Before the first transport of a delivery this module writes a +//! v1-*shaped* `Attempted` record at the old path and re-asserts it while the entry is +//! outstanding. It is never advanced, so it cannot contradict the ledger and no old binary can +//! read it as acceptance — it is the lower bound a rolled-back binary needs in order not to +//! re-POST a delivery this binary started. +//! * **Fail closed, never fail to start.** An unreadable or foreign ledger quarantines the pump: +//! no transport is authorized and the reason is retained. It never propagates as a startup error, +//! because refusing to start is strictly worse than holding. +//! +//! What this module deliberately does not own: the inbox. Archive remains the recipient agent's +//! act and the sole settlement authority (`message::archive_msg`). "Release" here means the ledger +//! stops holding FIFO ownership and drops the rollback floor — never that a file moves. + +use anyhow::{Context, Result}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use std::fs; +use std::io::Write; +use std::path::{Path, PathBuf}; + +use crate::message; + +/// The ledger's own schema. Bumping this is a fresh-namespace decision, not an in-place edit. +pub const LEDGER_SCHEMA: &str = "st2.delivery-ledger.v1"; +/// The ledger filename, a sibling of the v1 state file inside the same per-harness state dir. +pub const LEDGER_FILE: &str = "delivery-ledger.json"; +/// The legacy v1 filename: adoption source and rollback floor, never authority. +pub const LEGACY_FILE: &str = "delivery-state.json"; +/// Codex's v1 delivery-state schema. +pub const CODEX_LEGACY_SCHEMA: &str = "st2.codex-delivery-state.v1"; +/// OpenCode's v1 delivery-state schema. +pub const OPENCODE_LEGACY_SCHEMA: &str = "st2.opencode-delivery-state.v1"; + +/// A native transport with a durable delivery record. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Harness { + Codex, + OpenCode, +} + +impl Harness { + pub fn as_str(self) -> &'static str { + match self { + Self::Codex => "codex", + Self::OpenCode => "opencode", + } + } + + /// Fail closed on a harness this build does not know: an unrecognized name is refused rather + /// than defaulted, because a default would grade some other adapter's evidence with these + /// rules. + pub fn parse(name: &str) -> Result { + match name { + "codex" => Ok(Self::Codex), + "opencode" => Ok(Self::OpenCode), + other => anyhow::bail!("unknown native delivery harness '{other}'"), + } + } + + pub fn legacy_schema(self) -> &'static str { + match self { + Self::Codex => CODEX_LEGACY_SCHEMA, + Self::OpenCode => OPENCODE_LEGACY_SCHEMA, + } + } + + /// The v1 field naming the thread or session a delivery was bound to. + fn legacy_binding_key(self) -> &'static str { + match self { + Self::Codex => "threadId", + Self::OpenCode => "sessionId", + } + } + + /// The v1 field carrying the durable correlation the transport was sent under. + fn legacy_correlation_key(self) -> &'static str { + match self { + Self::Codex => "clientId", + Self::OpenCode => "messageId", + } + } + + /// What v1's `Accepted` label actually proved on this harness. + fn adopted_accepted_phase(self) -> Phase { + match self { + // Written only from the typed `item/completed{userMessage, clientId}` inside a turn: + // the model received it. + Self::Codex => Phase::Consumed, + // Written on `GET /session/{s}/message/{m}` returning 200: the server stored it. That + // is not scheduling, so mapping it to consumption would make the stored-but-never- + // admitted class permanently unretryable. + Self::OpenCode => Phase::Persisted, + } + } + + pub fn profile(self) -> Profile { + match self { + Self::Codex => Profile { + harness: self, + correlation: CorrelationKind::Native, + ceiling: Phase::Consumed, + ceiling_if_wired: Phase::Consumed, + // The same `clientUserMessageId` re-sent blind is not a proven no-op; Codex earns + // a retry from a `thread/resume` sweep that proves definite absence. + idempotent_resend: false, + retry: RetryPolicy::AtMostOnce, + }, + Self::OpenCode => Profile { + harness: self, + correlation: CorrelationKind::Native, + // What this build can observe. `session.next.prompt.admitted` is not wired, so + // persistence is the ceiling — and because it is below what the harness could + // prove, persistence never releases ownership. + ceiling: Phase::Persisted, + ceiling_if_wired: Phase::Admitted, + // Measured on 1.18.19: a second POST with the same messageID appends its parts + // again into the same message. Re-sending is a duplicate, not an idempotent retry. + idempotent_resend: false, + retry: RetryPolicy::AtMostOnce, + }, + } + } +} + +/// What a harness's evidence can prove, and what it could prove if every signal were wired. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct Profile { + pub harness: Harness, + pub correlation: CorrelationKind, + /// The highest phase this build's evidence can actually reach. + pub ceiling: Phase, + /// The highest phase the harness itself could prove. When it exceeds [`Profile::ceiling`], + /// reaching the ceiling is not a settlement — the missing signal is merely unread. + pub ceiling_if_wired: Phase, + /// Whether re-sending the same correlation is a measured no-op. + pub idempotent_resend: bool, + /// The declared policy where correlation cannot prove anything either way. + pub retry: RetryPolicy, +} + +impl Profile { + /// Fail closed on evidence this harness cannot honestly produce. + fn graded(&self, evidence: Evidence) -> Result { + let phase = match evidence { + Evidence::TransportAccepted => Phase::TransportAccepted, + Evidence::Persisted => Phase::Persisted, + Evidence::Admitted => Phase::Admitted, + Evidence::Consumed => Phase::Consumed, + }; + anyhow::ensure!( + self.proves(phase), + "{} delivery evidence cannot prove phase {phase:?}", + self.harness.as_str() + ); + Ok(phase) + } + + fn proves(&self, phase: Phase) -> bool { + match self.harness { + // Codex has a JSON-RPC result (transport) and a typed completed user message + // (consumption). It has no storage receipt and no scheduler admission signal. + Harness::Codex => matches!( + phase, + Phase::Attempted | Phase::TransportAccepted | Phase::Consumed + ), + // OpenCode has a POST status (transport), a durable message read-back (storage) and, + // once wired, prompt admission. It never proves the model consumed anything. + Harness::OpenCode => matches!( + phase, + Phase::Attempted | Phase::TransportAccepted | Phase::Persisted | Phase::Admitted + ), + } + } +} + +/// The declared policy for a harness whose correlation cannot settle a replay either way. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RetryPolicy { + /// Hold and surface rather than risk a duplicate. The conservative default. + AtMostOnce, + /// Accept a possible duplicate rather than risk a lost delivery. + AtLeastOnce, +} + +/// The monotone delivery phase. Declaration order is the ordering. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum Phase { + /// Durable before transport. Says only that this process was about to send. + Attempted, + /// The transport call itself succeeded. Says nothing about the harness's own state. + TransportAccepted, + /// The harness durably holds the exact correlated message. Storage, not scheduling. + Persisted, + /// The harness's scheduler took it as input. + Admitted, + /// The model received it. + Consumed, +} + +/// How durable a delivery's correlation is. Only [`CorrelationKind::Native`] can settle a replay. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum CorrelationKind { + /// A durable harness-side identity st2 chose and can re-query. + Native, + /// An attributable content echo. Confirms only a uniquely attributable match. + Content, + /// Live-only acknowledgement; nothing survives the process. + None, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct Correlation { + pub kind: CorrelationKind, + pub value: String, +} + +impl Correlation { + pub fn native(value: impl Into) -> Self { + Self { + kind: CorrelationKind::Native, + value: value.into(), + } + } +} + +/// An authoritative "no" about a specific attempt. The only thing besides a durable idempotent +/// correlation that may authorize a retry. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub enum NegativeReceipt { + /// The harness authoritatively does not hold the correlated message. + Absent, + /// The transport refused this attempt. + Rejected, +} + +/// Positive evidence a driver extracted from its harness. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Evidence { + TransportAccepted, + Persisted, + Admitted, + Consumed, +} + +/// One message's delivery, keyed by inbox filename. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct Entry { + pub filename: String, + /// The thread or session this delivery is bound to. A different binding is a different + /// delivery: its receipt may neither suppress nor acknowledge delivery to another one. + pub binding: String, + pub correlation: Correlation, + pub phase: Phase, + /// The runtime incarnation that made the attempt. Live evidence acknowledges only its own + /// incarnation; a pre-crash attempt is settled by a history sweep, never by a live frame. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub incarnation: Option, + /// The v1 schema this entry was carried forward from, when it was. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub adopted_from: Option, + /// Whether fresh evidence has authorized another transport. Adoption sets this false and never + /// true: carrying a record forward is not evidence about the world. + pub retry_eligible: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub negative: Option, + /// The exact v1-shaped `Attempted` record written at the legacy path for this entry, retained + /// so it can be re-asserted after a restart while the entry is still outstanding. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub legacy_floor: Option, +} + +/// The on-disk ledger. Additive-tolerant on read: unknown fields are ignored, but an unknown +/// schema, harness, phase, or owner is not. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +struct Record { + schema: String, + harness: String, + agent: String, + runtime_id: String, + entries: Vec, +} + +/// Whether the ledger still holds FIFO ownership of a delivery. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Retention { + /// Ownership released: stop offering, drop the rollback floor. Never an archive. + Release, + Hold(HoldReason), +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum HoldReason { + /// Transport happened; nothing about the harness's state is proved yet. + AmbiguousAttempt, + /// The harness holds it, but the evidence that would settle it is merely unread. + UnreadReceipt, + /// The harness said no. The item is re-offered. + NegativeReceipt, + /// Carried forward from v1: enough to suppress a duplicate, never enough to retry. + AdoptedWithoutFreshEvidence, + /// The ledger could not be read. Nothing may be transported. + Quarantined, + /// Already settled: ownership was released, so nothing is owed. + Settled, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum RetryDecision { + Retry, + Hold(HoldReason), +} + +/// A first transport's durable pre-conditions. +#[derive(Debug, Clone)] +pub struct Begin { + pub filename: String, + pub binding: String, + pub correlation: Correlation, + pub incarnation: Option, + /// The v1-shaped rollback floor an old binary must be able to read. Build it with + /// [`codex_floor`] or [`opencode_floor`]. + pub legacy_floor: Value, +} + +pub struct Ledger { + path: PathBuf, + legacy_path: PathBuf, + profile: Profile, + record: Record, + quarantine: Option, +} + +impl Ledger { + /// Open the ledger beside `legacy_path`, adopting the v1 record exactly once if this is the + /// first run on the new schema. + /// + /// `correlate(binding, filename)` recomputes the harness's durable correlation. It is the same + /// derivation the transport uses, so a record whose correlation does not match its own binding + /// is provably not this agent's and fails closed. + /// + /// Never returns an error: an unreadable record quarantines the pump instead of refusing to + /// start, because a driver that will not start delivers nothing at all. + pub fn open( + legacy_path: &Path, + profile: Profile, + agent: &str, + runtime_id: &str, + correlate: impl Fn(&str, &str) -> String, + ) -> Self { + let mut ledger = Self { + path: legacy_path.with_file_name(LEDGER_FILE), + legacy_path: legacy_path.to_path_buf(), + profile, + record: Record { + schema: LEDGER_SCHEMA.to_string(), + harness: profile.harness.as_str().to_string(), + agent: agent.to_string(), + runtime_id: runtime_id.to_string(), + entries: Vec::new(), + }, + quarantine: None, + }; + let outcome = match fs::read(&ledger.path) { + Ok(bytes) => ledger.load(&bytes, &correlate), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + ledger.adopt(&correlate) + } + Err(error) => Err(error).with_context(|| { + format!("reading delivery ledger {}", ledger.path.display()) + }), + }; + if let Err(error) = outcome { + ledger.record.entries.clear(); + ledger.quarantine = Some(format!("{error:#}")); + } else { + ledger.rebind_floor_runtime_ids(); + } + ledger + } + + fn load(&mut self, bytes: &[u8], correlate: &impl Fn(&str, &str) -> String) -> Result<()> { + let record: Record = serde_json::from_slice(bytes) + .with_context(|| format!("reading delivery ledger {}", self.path.display()))?; + anyhow::ensure!( + record.schema == LEDGER_SCHEMA, + "delivery ledger has unsupported schema '{}'", + record.schema + ); + anyhow::ensure!( + Harness::parse(&record.harness)? == self.profile.harness, + "delivery ledger belongs to harness '{}'", + record.harness + ); + // The agent identity is the durable owner; the runtime id is a mutable address. A seat + // relaunched under a new runtime is the SAME recipient holding the same outstanding + // deliveries, so drift rebinds the record rather than quarantining or discarding + // evidence — discarding it is exactly how a held ambiguous delivery becomes a duplicate. + // A rebind is bookkeeping, never evidence about the world: entries cross unchanged, + // including the incarnation that made each attempt, so ambiguous entries stay held, a + // live typed receipt still cannot settle a pre-crash attempt, and nothing about the + // rebind authorizes a resend. + anyhow::ensure!( + record.agent == self.record.agent, + "delivery ledger belongs to a different agent" + ); + let owner_runtime_id = self.record.runtime_id.clone(); + for entry in &record.entries { + Self::validate(&self.profile, entry)?; + } + anyhow::ensure!( + record + .entries + .windows(2) + .all(|pair| pair[0].binding == pair[1].binding), + "delivery ledger holds entries from more than one binding" + ); + // A bounded multi-message prefix shares ONE correlation: the durable identity its head was + // transported under. So a correlation is anchored when SOME entry carrying it derives it + // from its own binding and filename — requiring that of every entry would make a batch + // unloadable, and requiring it of none would accept a value nothing here was ever sent + // with. + if self.profile.correlation == CorrelationKind::Native { + for entry in &record.entries { + anyhow::ensure!( + record.entries.iter().any(|anchor| { + anchor.correlation.value == entry.correlation.value + && anchor.correlation.value + == correlate(&anchor.binding, &anchor.filename) + }), + "delivery ledger entry correlation does not match its binding" + ); + } + } + self.record = record; + // Provenance, not a fence: the rebound runtime id needs no write of its own and rides + // along on the next durable write this pump makes. + self.record.runtime_id = owner_runtime_id; + Ok(()) + } + + /// Carry the current runtime id onto every retained rollback floor. + /// + /// A rolled-back Codex loader compares `runtimeId`, so a floor still naming the runtime that + /// wrote it would be refused after a relaunch and the rollback would have no lower bound at + /// all — the duplicate class the floor exists to remove. Only that one mutable address is + /// rewritten. `clientId`, `runtimeIncarnation`, `threadId`/`sessionId`, `filename` and the + /// never-advanced `phase` are left exactly as written: they identify the attempt itself, and + /// v1 revalidates the client ID against the thread and filename it still carries. The floor + /// stays a true lower bound, and the corrected bytes land on the next per-pass re-assert, + /// which both drivers perform before any transport. + fn rebind_floor_runtime_ids(&mut self) { + let runtime_id = Value::String(self.record.runtime_id.clone()); + for entry in &mut self.record.entries { + if let Some(object) = entry.legacy_floor.as_mut().and_then(Value::as_object_mut) + && object.get("runtimeId") != Some(&runtime_id) + { + object.insert("runtimeId".to_string(), runtime_id.clone()); + } + } + } + + /// Per-entry structural validation, independent of any correlation grouping. + fn validate(profile: &Profile, entry: &Entry) -> Result<()> { + anyhow::ensure!( + message::is_message_filename(&entry.filename) && !entry.binding.is_empty(), + "delivery ledger entry has an invalid binding or filename" + ); + anyhow::ensure!( + entry.correlation.kind == profile.correlation, + "delivery ledger entry has a correlation kind this harness does not use" + ); + anyhow::ensure!( + profile.proves(entry.phase), + "delivery ledger entry records a phase {} cannot prove", + profile.harness.as_str() + ); + Ok(()) + } + + /// One-shot carry-forward of the v1 record, at the evidence its label actually proved. + /// + /// Adoption runs only when no ledger exists, and persists whatever it carried — so the ledger + /// it writes is what makes adoption happen once. With nothing to adopt it writes nothing, + /// because a driver that never delivers should leave no record behind. A legacy record naming + /// another schema or another agent is ignored rather than quarantined — it is not this + /// ledger's authority — but a record of ours whose correlation contradicts its own binding + /// fails closed. + fn adopt(&mut self, correlate: &impl Fn(&str, &str) -> String) -> Result<()> { + let Some(entry) = self.read_legacy(correlate)? else { + return Ok(()); + }; + self.record.entries.push(entry); + self.persist() + } + + fn read_legacy(&self, correlate: &impl Fn(&str, &str) -> String) -> Result> { + let bytes = match fs::read(&self.legacy_path) { + Ok(bytes) => bytes, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => { + return Err(error).with_context(|| { + format!("reading v1 delivery state {}", self.legacy_path.display()) + }); + } + }; + let legacy: Value = match serde_json::from_slice(&bytes) { + Ok(legacy) => legacy, + // Unreadable bytes at the v1 path are not authority and cannot be adopted; the new + // ledger simply starts empty rather than quarantining a working pump. + Err(_) => return Ok(None), + }; + let harness = self.profile.harness; + let string = |key: &str| legacy.get(key).and_then(Value::as_str).unwrap_or_default(); + // The agent identity is the durable owner, and it is the whole ownership test here. + // + // The runtime id is deliberately NOT compared, for a different reason on each harness. + // OpenCode's v1 filter never looked at it, so comparing it would be stricter than v1 and + // would drop a record the old binary would have acted on. Codex's v1 loader did compare + // it — and hard-errored, refusing to start — but a drifted record still describes a real + // attempt this recipient made, so it is carried forward at the phase it proved with + // `retryEligible: false`. Held that way, the `thread/resume` sweep can settle it or + // refuse it BEFORE anything is sent, which is strictly better than ignoring it and + // opening a second delivery for the same message. + if string("schema") != harness.legacy_schema() || string("agent") != self.record.agent { + return Ok(None); + } + let binding = string(harness.legacy_binding_key()).to_string(); + let filename = string("filename").to_string(); + let value = string(harness.legacy_correlation_key()).to_string(); + anyhow::ensure!( + message::is_message_filename(&filename) && !binding.is_empty(), + "v1 delivery state has an invalid binding or filename" + ); + anyhow::ensure!( + value == correlate(&binding, &filename), + "v1 delivery state correlation does not match its binding" + ); + let phase = match string("phase") { + "attempted" => Phase::Attempted, + "accepted" => harness.adopted_accepted_phase(), + other => anyhow::bail!("v1 delivery state has an unknown phase '{other}'"), + }; + let incarnation = legacy + .get("runtimeIncarnation") + .and_then(Value::as_str) + .map(str::to_string); + Ok(Some(Entry { + filename, + binding, + correlation: Correlation { + kind: CorrelationKind::Native, + value, + }, + phase, + incarnation, + adopted_from: Some(harness.legacy_schema().to_string()), + // Adoption is a carried-forward claim, never an observation: it can suppress a + // duplicate but it authorizes no transport until fresh evidence arrives. + retry_eligible: false, + negative: None, + // The record we adopted IS this entry's floor: it already exists at the v1 path, it is + // v1-readable, and it is never advanced from here. + legacy_floor: Some(legacy), + })) + } + + /// The reason nothing may be transported, if the record could not be read. + pub fn quarantined(&self) -> Option<&str> { + self.quarantine.as_deref() + } + + pub fn entries(&self) -> &[Entry] { + &self.record.entries + } + + pub fn entry(&self, filename: &str) -> Option<&Entry> { + self.record + .entries + .iter() + .find(|entry| entry.filename == filename) + } + + /// The binding every held entry shares, if any. + pub fn binding(&self) -> Option<&str> { + self.record + .entries + .first() + .map(|entry| entry.binding.as_str()) + } + + /// The filenames delivered under one correlation. A bounded multi-message prefix is several + /// entries sharing one value, so one receipt settles all of them at once. + pub fn correlated(&self, value: &str) -> Vec { + self.record + .entries + .iter() + .filter(|entry| entry.correlation.value == value) + .map(|entry| entry.filename.clone()) + .collect() + } + + /// Durably own an attempt before transporting it. Both durable writes — the ledger entry and + /// the v1-readable floor — complete before this call returns, and the caller transports only + /// afterwards. A crash between them leaves either nothing, or a floor that adoption reads + /// back as exactly this ambiguous attempt. + pub fn begin(&mut self, begin: Begin) -> Result { + anyhow::ensure!( + self.quarantine.is_none(), + "delivery ledger is quarantined: {}", + self.quarantine.as_deref().unwrap_or_default() + ); + anyhow::ensure!( + begin.correlation.kind == self.profile.correlation, + "{} delivery cannot use this correlation kind", + self.profile.harness.as_str() + ); + // Deliberately not written yet: v1 holds ONE record, so the floor that belongs there is + // the oldest outstanding attempt, which is only knowable once this entry has landed. + let held = self + .record + .entries + .iter() + .position(|entry| entry.filename == begin.filename); + let entry = match held { + Some(index) => { + let entry = &mut self.record.entries[index]; + entry.binding = begin.binding; + entry.correlation = begin.correlation; + entry.incarnation = begin.incarnation; + // Monotone: a repeat attempt never lowers what was already proved. + entry.phase = entry.phase.max(Phase::Attempted); + // A live attempt is ambiguous again, and it supersedes the receipt that authorized + // it — the next retry must earn its own evidence. It is also no longer a merely + // carried-forward claim: this build transported it. + entry.negative = None; + entry.retry_eligible = false; + entry.adopted_from = None; + entry.legacy_floor = Some(begin.legacy_floor); + entry.clone() + } + None => { + let entry = Entry { + filename: begin.filename, + binding: begin.binding, + correlation: begin.correlation, + phase: Phase::Attempted, + incarnation: begin.incarnation, + adopted_from: None, + retry_eligible: false, + negative: None, + legacy_floor: Some(begin.legacy_floor), + }; + self.record.entries.push(entry.clone()); + entry + } + }; + self.persist()?; + // The floor lands AFTER the entry, and it is the OLDEST outstanding one: a batch must + // leave the earliest attempt's lower bound at the v1 path, never the most recent. + self.reassert_floor()?; + Ok(entry) + } + + /// Record positive evidence. Refuses evidence the harness cannot prove, and never lowers a + /// phase. Returns the entry's phase after the write, or `None` when no such entry is held. + pub fn record(&mut self, filename: &str, evidence: Evidence) -> Result> { + anyhow::ensure!( + self.quarantine.is_none(), + "delivery ledger is quarantined: {}", + self.quarantine.as_deref().unwrap_or_default() + ); + let phase = self.profile.graded(evidence)?; + let Some(index) = self + .record + .entries + .iter() + .position(|entry| entry.filename == filename) + else { + return Ok(None); + }; + { + let entry = &mut self.record.entries[index]; + if phase <= entry.phase { + return Ok(Some(entry.phase)); + } + entry.phase = phase; + // Positive evidence supersedes an earlier "no" about the same attempt. + entry.negative = None; + entry.retry_eligible = false; + } + self.persist()?; + self.settle()?; + Ok(Some(phase)) + } + + /// Record an authoritative "no". Ignored once the entry is settled: a late refusal cannot + /// un-consume a delivery. + pub fn negative(&mut self, filename: &str, receipt: NegativeReceipt) -> Result { + anyhow::ensure!( + self.quarantine.is_none(), + "delivery ledger is quarantined: {}", + self.quarantine.as_deref().unwrap_or_default() + ); + let retention = self.retention(filename); + if retention == Retention::Release { + return Ok(retention); + } + let Some(index) = self + .record + .entries + .iter() + .position(|entry| entry.filename == filename) + else { + return Ok(retention); + }; + { + let entry = &mut self.record.entries[index]; + if entry.negative == Some(receipt) && entry.retry_eligible { + return Ok(Retention::Hold(HoldReason::NegativeReceipt)); + } + entry.negative = Some(receipt); + entry.retry_eligible = true; + } + self.persist()?; + Ok(Retention::Hold(HoldReason::NegativeReceipt)) + } + + /// Whether the ledger still holds FIFO ownership of `filename`. + /// + /// Release requires either scheduler-or-better evidence, or reaching a ceiling that is the + /// harness's true ceiling. A ceiling below what the harness could prove is not a settlement: + /// OpenCode `persisted` holds with [`HoldReason::UnreadReceipt`] rather than releasing. + pub fn retention(&self, filename: &str) -> Retention { + if self.quarantine.is_some() { + return Retention::Hold(HoldReason::Quarantined); + } + let Some(entry) = self.entry(filename) else { + return Retention::Release; + }; + if entry.negative.is_some() { + return Retention::Hold(HoldReason::NegativeReceipt); + } + if entry.phase >= Phase::Admitted { + return Retention::Release; + } + if entry.phase == self.profile.ceiling + && self.profile.ceiling == self.profile.ceiling_if_wired + { + return Retention::Release; + } + if entry.phase >= Phase::Persisted { + return Retention::Hold(HoldReason::UnreadReceipt); + } + if entry.adopted_from.is_some() && !entry.retry_eligible { + return Retention::Hold(HoldReason::AdoptedWithoutFreshEvidence); + } + Retention::Hold(HoldReason::AmbiguousAttempt) + } + + /// Whether another transport of `filename` is authorized. + pub fn retry(&self, filename: &str) -> RetryDecision { + if self.quarantine.is_some() { + return RetryDecision::Hold(HoldReason::Quarantined); + } + let Some(entry) = self.entry(filename) else { + // Nothing is held, so nothing is being repeated: a first transport is not a retry. + return RetryDecision::Retry; + }; + match self.retention(filename) { + Retention::Release => return RetryDecision::Hold(HoldReason::Settled), + Retention::Hold(HoldReason::NegativeReceipt) => return RetryDecision::Retry, + Retention::Hold(_) => {} + } + if entry.adopted_from.is_some() && !entry.retry_eligible { + return RetryDecision::Hold(HoldReason::AdoptedWithoutFreshEvidence); + } + if entry.phase >= Phase::Persisted { + return RetryDecision::Hold(HoldReason::UnreadReceipt); + } + if entry.phase > Phase::Attempted { + // The transport call itself landed. Nothing readable proves the harness did not get + // it, so a second send is a duplicate risk, not a recovery. + return RetryDecision::Hold(HoldReason::AmbiguousAttempt); + } + match entry.correlation.kind { + CorrelationKind::Native if self.profile.idempotent_resend => RetryDecision::Retry, + CorrelationKind::Native => RetryDecision::Hold(HoldReason::AmbiguousAttempt), + CorrelationKind::Content | CorrelationKind::None => match self.profile.retry { + RetryPolicy::AtLeastOnce => RetryDecision::Retry, + RetryPolicy::AtMostOnce => RetryDecision::Hold(HoldReason::AmbiguousAttempt), + }, + } + } + + /// Drop every entry whose binding is not `binding`. A newly selected thread or session is a + /// different delivery binding, and the old one's receipt may neither suppress nor acknowledge + /// delivery to this one. + pub fn rebind(&mut self, binding: &str) -> Result<()> { + let before = self.record.entries.len(); + self.record.entries.retain(|entry| entry.binding == binding); + if self.record.entries.len() == before { + return Ok(()); + } + self.persist()?; + self.settle() + } + + /// Reconcile the ledger to what the recipient still has unread. Archive is the recipient's + /// act and the settlement authority: an entry whose file left the inbox is released, never + /// re-offered, and never archived from here. + pub fn prune(&mut self, is_unread: impl Fn(&str) -> bool) -> Result<()> { + let before = self.record.entries.len(); + self.record + .entries + .retain(|entry| is_unread(&entry.filename)); + if self.record.entries.len() == before { + return Ok(()); + } + self.persist()?; + self.settle() + } + + /// The floor of the OLDEST outstanding delivery. v1 holds a single record, so when several + /// entries are outstanding the one lower bound it can carry must be the earliest attempt. + /// Message filenames start with their unix-ms send time, so lexicographic order is FIFO + /// order — the same grammar the bus already relies on. + fn oldest_floor(&self) -> Option { + self.record + .entries + .iter() + .filter(|entry| entry.legacy_floor.is_some()) + .min_by(|left, right| left.filename.cmp(&right.filename)) + .and_then(|entry| entry.legacy_floor.clone()) + } + + /// Re-assert the rollback floor while an entry is outstanding. Called on every pass: a crash + /// exactly at the floor write would otherwise leave a landing with no v1-readable lower bound. + pub fn reassert_floor(&mut self) -> Result<()> { + let Some(floor) = self.oldest_floor() else { + return Ok(()); + }; + self.write_legacy(&floor) + } + + /// Drop the floor of every released entry, then converge the v1 path: the head outstanding + /// floor, or nothing at all. + fn settle(&mut self) -> Result<()> { + let released: Vec = self + .record + .entries + .iter() + .filter(|entry| entry.legacy_floor.is_some()) + .filter(|entry| self.retention(&entry.filename) == Retention::Release) + .map(|entry| entry.filename.clone()) + .collect(); + if !released.is_empty() { + for entry in &mut self.record.entries { + if released.contains(&entry.filename) { + entry.legacy_floor = None; + } + } + self.persist()?; + } + match self.oldest_floor() { + Some(floor) => self.write_legacy(&floor), + None => remove_file(&self.legacy_path), + } + } + + /// Write the v1 rollback floor, skipping a byte-identical restatement. An outstanding + /// delivery re-asserts its floor on every pass, and an unread message stays outstanding for + /// as long as the recipient leaves it in the inbox — so restating must cost a read, not an + /// fsync per pass. + fn write_legacy(&self, floor: &Value) -> Result<()> { + let bytes = serde_json::to_vec(floor)?; + if fs::read(&self.legacy_path).is_ok_and(|current| current == bytes) { + return Ok(()); + } + atomic_bytes(&self.legacy_path, &bytes).with_context(|| { + format!( + "writing v1 delivery rollback floor {}", + self.legacy_path.display() + ) + }) + } + + fn persist(&self) -> Result<()> { + atomic_json(&self.path, &self.record) + .with_context(|| format!("writing delivery ledger {}", self.path.display())) + } +} + +/// Codex's v1-shaped rollback floor. The key set is exact: v1's `CodexDeliveryState` denies +/// unknown fields, so an extra or missing key makes the old binary refuse to start. +pub fn codex_floor( + agent: &str, + runtime_id: &str, + runtime_incarnation: &str, + thread_id: &str, + filename: &str, + client_id: &str, +) -> Value { + json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": agent, + "runtimeId": runtime_id, + "runtimeIncarnation": runtime_incarnation, + "threadId": thread_id, + "filename": filename, + "clientId": client_id, + "phase": "attempted", + }) +} + +/// OpenCode's v1-shaped rollback floor. v1 recomputes `stable_message_id(identity, session, +/// filename)` and silently discards a record that does not match, so `message_id` must be the +/// exact derived value or the floor buys nothing. +pub fn opencode_floor( + agent: &str, + runtime_id: &str, + session_id: &str, + filename: &str, + message_id: &str, +) -> Value { + json!({ + "schema": OPENCODE_LEGACY_SCHEMA, + "agent": agent, + "runtimeId": runtime_id, + "sessionId": session_id, + "filename": filename, + "messageId": message_id, + "phase": "attempted", + }) +} + +/// Durability, not just atomicity: a crash between the rename and the harness's acceptance of the +/// transport would otherwise lose the receipt and let the pump re-send duplicate content. The +/// bytes reach disk before the rename and the directory entry afterwards. +fn atomic_json(path: &Path, value: &impl Serialize) -> Result<()> { + atomic_bytes(path, &serde_json::to_vec(value)?) +} + +fn atomic_bytes(path: &Path, bytes: &[u8]) -> Result<()> { + let parent = path.parent().context("ledger file has no parent")?; + fs::create_dir_all(parent)?; + let temp = parent.join(format!(".{}.tmp", std::process::id())); + let mut file = fs::File::create(&temp)?; + file.write_all(bytes)?; + file.sync_all()?; + drop(file); + if let Err(error) = fs::rename(&temp, path) { + let _ = fs::remove_file(&temp); + return Err(error.into()); + } + if let Ok(dir) = fs::File::open(parent) { + let _ = dir.sync_all(); + } + Ok(()) +} + +fn remove_file(path: &Path) -> Result<()> { + match fs::remove_file(path) { + Ok(()) => { + if let Some(parent) = path.parent() + && let Ok(dir) = fs::File::open(parent) + { + let _ = dir.sync_all(); + } + Ok(()) + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()), + Err(error) => Err(error.into()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + const FILE_A: &str = "1786380000000-aaa111.md"; + const FILE_B: &str = "1786380000001-bbb222.md"; + + /// The one derivation both the transport and the ledger use, so a record that does not match + /// its own binding is provably not ours. + fn correlate(binding: &str, filename: &str) -> String { + format!("corr:{binding}:{filename}") + } + + fn legacy_path(dir: &Path) -> PathBuf { + dir.join("state").join(LEGACY_FILE) + } + + fn open(dir: &Path, harness: Harness) -> Ledger { + Ledger::open( + &legacy_path(dir), + harness.profile(), + "h.worker", + "h.worker", + correlate, + ) + } + + fn begin(ledger: &mut Ledger, harness: Harness, binding: &str, filename: &str) -> Entry { + let value = correlate(binding, filename); + let floor = match harness { + Harness::Codex => codex_floor( + "h.worker", + "h.worker", + "incarnation-1", + binding, + filename, + &value, + ), + Harness::OpenCode => opencode_floor("h.worker", "h.worker", binding, filename, &value), + }; + ledger + .begin(Begin { + filename: filename.to_string(), + binding: binding.to_string(), + correlation: Correlation::native(value), + incarnation: match harness { + Harness::Codex => Some("incarnation-1".to_string()), + Harness::OpenCode => None, + }, + legacy_floor: floor, + }) + .unwrap() + } + + #[test] + fn phase_is_monotonic_and_durable_across_every_write_boundary() { + let tmp = tempfile::tempdir().unwrap(); + let mut ledger = open(tmp.path(), Harness::Codex); + begin(&mut ledger, Harness::Codex, "thread-main", FILE_A); + + // Every boundary is re-readable, and re-reading never loses a phase. + for evidence in [Evidence::TransportAccepted, Evidence::Consumed] { + ledger.record(FILE_A, evidence).unwrap(); + let reopened = open(tmp.path(), Harness::Codex); + assert_eq!( + reopened.entry(FILE_A).unwrap().phase, + ledger.entry(FILE_A).unwrap().phase, + "the phase on disk is the phase in memory" + ); + } + assert_eq!(ledger.entry(FILE_A).unwrap().phase, Phase::Consumed); + + // A lower phase is refused, not written. + assert_eq!( + ledger.record(FILE_A, Evidence::TransportAccepted).unwrap(), + Some(Phase::Consumed), + "a late lower reading never regresses the record" + ); + assert_eq!( + open(tmp.path(), Harness::Codex).entry(FILE_A).unwrap().phase, + Phase::Consumed + ); + } + + #[test] + fn v1_accepted_adopts_at_the_evidence_it_actually_proved() { + // Codex wrote `Accepted` only from the typed completed user message: consumption. + let codex = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(codex.path()), + &json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "runtimeIncarnation": "incarnation-0", + "threadId": "thread-main", + "filename": FILE_A, + "clientId": correlate("thread-main", FILE_A), + "phase": "accepted", + }), + ) + .unwrap(); + let ledger = open(codex.path(), Harness::Codex); + assert_eq!(ledger.quarantined(), None); + let entry = ledger.entry(FILE_A).unwrap(); + assert_eq!(entry.phase, Phase::Consumed); + assert_eq!(entry.adopted_from.as_deref(), Some(CODEX_LEGACY_SCHEMA)); + assert_eq!( + ledger.retention(FILE_A), + Retention::Release, + "a Codex acceptance is a true ceiling" + ); + + // OpenCode wrote `Accepted` on a GET 200: storage, never scheduling. + let opencode = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(opencode.path()), + &json!({ + "schema": OPENCODE_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "sessionId": "ses_target", + "filename": FILE_A, + "messageId": correlate("ses_target", FILE_A), + "phase": "accepted", + }), + ) + .unwrap(); + let ledger = open(opencode.path(), Harness::OpenCode); + let entry = ledger.entry(FILE_A).unwrap(); + assert_eq!(entry.phase, Phase::Persisted); + assert_eq!( + ledger.retention(FILE_A), + Retention::Hold(HoldReason::UnreadReceipt), + "persistence is not consumption and never releases ownership" + ); + } + + #[test] + fn a_runtime_id_change_rebinds_the_record_and_holds_its_entries() { + let tmp = tempfile::tempdir().unwrap(); + let ledger_file = legacy_path(tmp.path()).with_file_name(LEDGER_FILE); + let mut ledger = open(tmp.path(), Harness::OpenCode); + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_A); + + // The seat is relaunched under a new runtime id. The agent — the durable owner — is + // unchanged, so this is the same recipient still holding the same outstanding delivery. + // Quarantining or discarding here is exactly how a held ambiguous delivery becomes a + // duplicate. + let mut rebound = Ledger::open( + &legacy_path(tmp.path()), + Harness::OpenCode.profile(), + "h.worker", + "h.worker.relaunched", + correlate, + ); + assert_eq!(rebound.quarantined(), None, "drift is not corruption"); + let entry = rebound + .entry(FILE_A) + .expect("the evidence is preserved, never discarded"); + assert_eq!(entry.phase, Phase::Attempted); + assert_eq!( + rebound.retry(FILE_A), + RetryDecision::Hold(HoldReason::AmbiguousAttempt), + "a rebind is bookkeeping, never evidence: it authorizes no resend of its own" + ); + + // The rebound id is provenance, not a fence: it needs no write of its own and rides along + // on the next durable write. + rebound.record(FILE_A, Evidence::Persisted).unwrap(); + let on_disk: Value = serde_json::from_slice(&fs::read(&ledger_file).unwrap()).unwrap(); + assert_eq!(on_disk["runtimeId"], "h.worker.relaunched"); + + // A different AGENT is a different recipient, and that still fails closed. + let foreign = Ledger::open( + &legacy_path(tmp.path()), + Harness::OpenCode.profile(), + "h.other", + "h.worker", + correlate, + ); + assert!( + foreign + .quarantined() + .is_some_and(|reason| reason.contains("different agent")) + ); + assert!(foreign.entries().is_empty()); + } + + #[test] + fn a_runtime_drifted_v1_record_is_adopted_so_reconciliation_can_settle_it_before_send() { + // OpenCode: v1's filter never compared `runtimeId` — schema, agent, the filename grammar + // and the recomputed messageID were the whole test — so adopting on a stricter rule would + // drop a record the old binary WOULD have acted on, which is the duplicate-POST class + // returning. + let opencode = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(opencode.path()), + &json!({ + "schema": OPENCODE_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker.previous", + "sessionId": "ses_target", + "filename": FILE_A, + "messageId": correlate("ses_target", FILE_A), + "phase": "attempted", + }), + ) + .unwrap(); + let ledger = open(opencode.path(), Harness::OpenCode); + assert_eq!(ledger.quarantined(), None); + assert!( + ledger.entry(FILE_A).is_some(), + "OpenCode's v1 loader ignored runtimeId, so adoption must too" + ); + + // Codex: v1 DID compare it, and hard-errored. But the record still describes a real + // attempt this recipient made, so it is carried forward at the phase it proved and held — + // ignoring it would open a second delivery for the same message, while holding it lets + // the resume sweep settle or refuse it before anything is sent. + let codex = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(codex.path()), + &json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker.previous", + "runtimeIncarnation": "incarnation-0", + "threadId": "thread-main", + "filename": FILE_A, + "clientId": correlate("thread-main", FILE_A), + "phase": "accepted", + }), + ) + .unwrap(); + let mut ledger = open(codex.path(), Harness::Codex); + assert_eq!(ledger.quarantined(), None); + let entry = ledger + .entry(FILE_A) + .expect("a drifted record is adopted, never ignored"); + assert_eq!( + entry.phase, + Phase::Consumed, + "adopted at the phase its label actually proved" + ); + assert!(!entry.retry_eligible); + assert_eq!(entry.adopted_from.as_deref(), Some(CODEX_LEGACY_SCHEMA)); + assert_eq!(entry.incarnation.as_deref(), Some("incarnation-0")); + + // An ambiguous drifted attempt is held for the sweep, and only its verdict moves it. + let attempted = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(attempted.path()), + &json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker.previous", + "runtimeIncarnation": "incarnation-0", + "threadId": "thread-main", + "filename": FILE_A, + "clientId": correlate("thread-main", FILE_A), + "phase": "attempted", + }), + ) + .unwrap(); + ledger = open(attempted.path(), Harness::Codex); + assert_eq!( + ledger.retry(FILE_A), + RetryDecision::Hold(HoldReason::AdoptedWithoutFreshEvidence), + "no send before the sweep" + ); + ledger.negative(FILE_A, NegativeReceipt::Absent).unwrap(); + assert_eq!( + ledger.retry(FILE_A), + RetryDecision::Retry, + "a resumed history proving absence is what authorizes the resend" + ); + } + + #[test] + fn a_rebind_rewrites_only_the_runtime_id_on_every_retained_floor() { + let tmp = tempfile::tempdir().unwrap(); + let legacy = legacy_path(tmp.path()); + let mut ledger = open(tmp.path(), Harness::Codex); + begin(&mut ledger, Harness::Codex, "thread-main", FILE_A); + let before: Value = serde_json::from_slice(&fs::read(&legacy).unwrap()).unwrap(); + assert_eq!(before["runtimeId"], "h.worker"); + + // The seat is relaunched under a new runtime id. A rolled-back Codex loader compares + // `runtimeId`, so a floor still naming the previous runtime would be refused and the + // rollback would have no lower bound at all. + let mut rebound = Ledger::open( + &legacy_path(tmp.path()), + Harness::Codex.profile(), + "h.worker", + "h.worker.relaunched", + correlate, + ); + rebound.reassert_floor().unwrap(); + let after: Value = serde_json::from_slice(&fs::read(&legacy).unwrap()).unwrap(); + assert_eq!(after["runtimeId"], "h.worker.relaunched"); + + // Only that one mutable address moved. Everything identifying the attempt is untouched, + // including the client ID v1 revalidates against the thread and filename it still carries, + // and the never-advanced phase. + for key in [ + "schema", + "agent", + "runtimeIncarnation", + "threadId", + "filename", + "clientId", + "phase", + ] { + assert_eq!(after[key], before[key], "{key} must not change"); + } + assert_eq!(after["phase"], "attempted"); + assert_eq!( + after["clientId"].as_str().unwrap(), + correlate("thread-main", FILE_A), + "the floor still passes v1's own client-ID recomputation" + ); + assert_eq!( + after.as_object().unwrap().len(), + before.as_object().unwrap().len(), + "no key is added or removed: v1 denies unknown fields" + ); + + // The ledger's own entry is unchanged too — a rebind is bookkeeping, not evidence. + let entry = rebound.entry(FILE_A).unwrap(); + assert_eq!(entry.phase, Phase::Attempted); + assert_eq!(entry.incarnation.as_deref(), Some("incarnation-1")); + assert_eq!(entry.correlation.value, correlate("thread-main", FILE_A)); + } + + #[test] + fn the_v1_path_holds_the_oldest_outstanding_floor_at_every_crash_position() { + let tmp = tempfile::tempdir().unwrap(); + let legacy = legacy_path(tmp.path()); + let mut ledger = open(tmp.path(), Harness::OpenCode); + + // Two distinct filenames with two distinct floors, under one binding. The newer one is + // begun FIRST on purpose: the answer must not depend on insertion order. + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_B); + assert_eq!( + serde_json::from_slice::(&fs::read(&legacy).unwrap()).unwrap()["filename"], + FILE_B + ); + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_A); + assert_eq!(ledger.entries().len(), 2); + + // v1 holds ONE record, so it must name the OLDEST outstanding attempt — a rolled-back + // binary reconciles the earliest delivery, and the later one is still held by the ledger. + let oldest = |legacy: &Path| { + serde_json::from_slice::(&fs::read(legacy).unwrap()).unwrap()["filename"] + .as_str() + .unwrap() + .to_string() + }; + assert_eq!(oldest(&legacy), FILE_A); + + // Crash position: restart. The floor still names the oldest and both entries stay held. + let mut restarted = open(tmp.path(), Harness::OpenCode); + fs::remove_file(&legacy).unwrap(); + restarted.reassert_floor().unwrap(); + assert_eq!(oldest(&legacy), FILE_A); + for filename in [FILE_A, FILE_B] { + assert_eq!( + restarted.retry(filename), + RetryDecision::Hold(HoldReason::AmbiguousAttempt), + "{filename} is held, not replayed" + ); + } + + // The recipient archives the oldest: the floor advances to the next outstanding attempt + // and never regresses onto a released one. + restarted.prune(|filename| filename == FILE_B).unwrap(); + assert_eq!(oldest(&legacy), FILE_B); + + // And when the last outstanding entry goes, so does the floor. + restarted.prune(|_| false).unwrap(); + assert!(!legacy.exists()); + } + + #[test] + fn adoption_alone_never_authorizes_a_transport() { + let tmp = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(tmp.path()), + &json!({ + "schema": OPENCODE_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "sessionId": "ses_target", + "filename": FILE_A, + "messageId": correlate("ses_target", FILE_A), + "phase": "attempted", + }), + ) + .unwrap(); + let mut ledger = open(tmp.path(), Harness::OpenCode); + let entry = ledger.entry(FILE_A).unwrap(); + assert_eq!(entry.phase, Phase::Attempted); + assert!(!entry.retry_eligible); + assert_eq!( + ledger.retry(FILE_A), + RetryDecision::Hold(HoldReason::AdoptedWithoutFreshEvidence) + ); + assert_eq!( + ledger.retention(FILE_A), + Retention::Hold(HoldReason::AdoptedWithoutFreshEvidence), + "an ambiguous carried-forward attempt is held and surfaced, never replayed" + ); + + // Only fresh evidence about the world moves it: an authoritative absence. + ledger.negative(FILE_A, NegativeReceipt::Absent).unwrap(); + assert_eq!(ledger.retry(FILE_A), RetryDecision::Retry); + } + + #[test] + fn adoption_happens_exactly_once_and_never_re_reads_a_stale_v1_record() { + let tmp = tempfile::tempdir().unwrap(); + let floor = json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "runtimeIncarnation": "incarnation-0", + "threadId": "thread-main", + "filename": FILE_A, + "clientId": correlate("thread-main", FILE_A), + "phase": "attempted", + }); + atomic_json(&legacy_path(tmp.path()), &floor).unwrap(); + let mut ledger = open(tmp.path(), Harness::Codex); + assert_eq!(ledger.entry(FILE_A).unwrap().phase, Phase::Attempted); + + // The delivery settles and the ledger releases it, clearing the floor. + ledger.record(FILE_A, Evidence::Consumed).unwrap(); + assert!( + !legacy_path(tmp.path()).exists(), + "release clears the v1 floor" + ); + + // A stale v1 record reappearing (an old binary, a restored backup) is not re-adopted: the + // ledger file already exists, so adoption is spent. + atomic_json(&legacy_path(tmp.path()), &floor).unwrap(); + let reopened = open(tmp.path(), Harness::Codex); + assert_eq!(reopened.entry(FILE_A).unwrap().phase, Phase::Consumed); + } + + #[test] + fn a_negative_receipt_is_retained_and_re_offered() { + let tmp = tempfile::tempdir().unwrap(); + let mut ledger = open(tmp.path(), Harness::OpenCode); + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_A); + assert_eq!( + ledger.retry(FILE_A), + RetryDecision::Hold(HoldReason::AmbiguousAttempt), + "an ambiguous attempt on a non-idempotent transport is held" + ); + + ledger.negative(FILE_A, NegativeReceipt::Absent).unwrap(); + // The receipt survives the process that observed it. + let reopened = open(tmp.path(), Harness::OpenCode); + assert_eq!( + reopened.entry(FILE_A).unwrap().negative, + Some(NegativeReceipt::Absent) + ); + assert_eq!(reopened.retry(FILE_A), RetryDecision::Retry); + assert_eq!( + reopened.retention(FILE_A), + Retention::Hold(HoldReason::NegativeReceipt), + "a refused delivery is retained, never released" + ); + + // A settled delivery cannot be un-settled by a late "no". + let mut settled = open(tmp.path(), Harness::OpenCode); + settled.record(FILE_A, Evidence::Admitted).unwrap(); + assert_eq!( + settled.negative(FILE_A, NegativeReceipt::Rejected).unwrap(), + Retention::Release + ); + assert_eq!(settled.entry(FILE_A).unwrap().phase, Phase::Admitted); + assert_eq!(settled.entry(FILE_A).unwrap().negative, None); + } + + #[test] + fn release_requires_the_harness_scheduler_or_a_true_ceiling() { + let tmp = tempfile::tempdir().unwrap(); + let mut opencode = open(tmp.path(), Harness::OpenCode); + begin(&mut opencode, Harness::OpenCode, "ses_target", FILE_A); + for (evidence, expected) in [ + ( + Evidence::TransportAccepted, + Retention::Hold(HoldReason::AmbiguousAttempt), + ), + ( + Evidence::Persisted, + Retention::Hold(HoldReason::UnreadReceipt), + ), + (Evidence::Admitted, Retention::Release), + ] { + opencode.record(FILE_A, evidence).unwrap(); + assert_eq!(opencode.retention(FILE_A), expected, "{evidence:?}"); + } + + // Codex's ceiling is the harness's own ceiling, so reaching it settles. + let codex_dir = tempfile::tempdir().unwrap(); + let mut codex = open(codex_dir.path(), Harness::Codex); + begin(&mut codex, Harness::Codex, "thread-main", FILE_A); + codex.record(FILE_A, Evidence::TransportAccepted).unwrap(); + assert_eq!( + codex.retention(FILE_A), + Retention::Hold(HoldReason::AmbiguousAttempt), + "a transport result is not a receipt" + ); + codex.record(FILE_A, Evidence::Consumed).unwrap(); + assert_eq!(codex.retention(FILE_A), Retention::Release); + } + + #[test] + fn unknown_and_dishonest_adapter_evidence_fails_closed() { + assert!(Harness::parse("gemini").is_err()); + + let tmp = tempfile::tempdir().unwrap(); + let mut opencode = open(tmp.path(), Harness::OpenCode); + begin(&mut opencode, Harness::OpenCode, "ses_target", FILE_A); + let error = opencode.record(FILE_A, Evidence::Consumed).unwrap_err(); + assert!( + error.to_string().contains("cannot prove"), + "OpenCode has no consumption signal: {error:#}" + ); + assert_eq!(opencode.entry(FILE_A).unwrap().phase, Phase::Attempted); + + let codex_dir = tempfile::tempdir().unwrap(); + let mut codex = open(codex_dir.path(), Harness::Codex); + begin(&mut codex, Harness::Codex, "thread-main", FILE_A); + assert!(codex.record(FILE_A, Evidence::Persisted).is_err()); + assert!(codex.record(FILE_A, Evidence::Admitted).is_err()); + } + + #[test] + fn an_unreadable_ledger_quarantines_the_pump_instead_of_refusing_to_start() { + let tmp = tempfile::tempdir().unwrap(); + let mut ledger = open(tmp.path(), Harness::Codex); + begin(&mut ledger, Harness::Codex, "thread-main", FILE_A); + + // A foreign schema at the ledger path: exactly the shape an in-place bump would have left. + atomic_json( + &legacy_path(tmp.path()).with_file_name(LEDGER_FILE), + &json!({ + "schema": "st2.delivery-ledger.v2", + "harness": "codex", + "agent": "h.worker", + "runtimeId": "h.worker", + "entries": [], + }), + ) + .unwrap(); + let mut quarantined = open(tmp.path(), Harness::Codex); + assert!( + quarantined + .quarantined() + .is_some_and(|reason| reason.contains("unsupported schema")), + "the reason is retained" + ); + assert!(quarantined.entries().is_empty()); + assert_eq!( + quarantined.retention(FILE_A), + Retention::Hold(HoldReason::Quarantined) + ); + assert_eq!( + quarantined.retry(FILE_A), + RetryDecision::Hold(HoldReason::Quarantined) + ); + let error = quarantined + .begin(Begin { + filename: FILE_A.to_string(), + binding: "thread-main".to_string(), + correlation: Correlation::native(correlate("thread-main", FILE_A)), + incarnation: None, + legacy_floor: json!({}), + }) + .unwrap_err(); + assert!(error.to_string().contains("quarantined")); + } + + #[test] + fn a_tampered_correlation_fails_closed_without_deleting_evidence() { + let tmp = tempfile::tempdir().unwrap(); + atomic_json( + &legacy_path(tmp.path()), + &json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "runtimeIncarnation": "incarnation-0", + "threadId": "thread-main", + "filename": FILE_A, + "clientId": "st2:tampered", + "phase": "attempted", + }), + ) + .unwrap(); + let ledger = open(tmp.path(), Harness::Codex); + assert!( + ledger + .quarantined() + .is_some_and(|reason| reason.contains("does not match its binding")) + ); + assert!( + legacy_path(tmp.path()).exists(), + "a record we refuse to read is not a record we may destroy" + ); + } + + #[test] + fn a_v1_readable_floor_precedes_the_first_transport_and_clears_only_on_release() { + let tmp = tempfile::tempdir().unwrap(); + let legacy = legacy_path(tmp.path()); + let mut ledger = open(tmp.path(), Harness::OpenCode); + assert!(!legacy.exists()); + + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_A); + let floor: Value = serde_json::from_slice(&fs::read(&legacy).unwrap()).unwrap(); + assert_eq!(floor["schema"], OPENCODE_LEGACY_SCHEMA); + assert_eq!(floor["phase"], "attempted"); + assert_eq!(floor["messageId"], correlate("ses_target", FILE_A)); + + // The floor never advances, so it can never contradict the ledger and no old binary can + // read it as acceptance. + ledger.record(FILE_A, Evidence::Persisted).unwrap(); + let held: Value = serde_json::from_slice(&fs::read(&legacy).unwrap()).unwrap(); + assert_eq!(held["phase"], "attempted"); + assert_eq!(held, floor); + + // Re-asserted while outstanding: a crash exactly at the floor write must not leave a + // landing without a v1-readable lower bound. + fs::remove_file(&legacy).unwrap(); + ledger.reassert_floor().unwrap(); + assert_eq!( + serde_json::from_slice::(&fs::read(&legacy).unwrap()).unwrap(), + floor + ); + + // Cleared only at release. + ledger.record(FILE_A, Evidence::Admitted).unwrap(); + assert!(!legacy.exists()); + ledger.reassert_floor().unwrap(); + assert!(!legacy.exists(), "a released entry re-asserts nothing"); + } + + #[test] + fn a_bounded_multi_message_prefix_is_n_entries_under_one_correlation() { + let tmp = tempfile::tempdir().unwrap(); + let mut ledger = open(tmp.path(), Harness::Codex); + // One transport, two messages: the shape a bounded FIFO prefix needs. + let value = correlate("thread-main", FILE_A); + for filename in [FILE_A, FILE_B] { + ledger + .begin(Begin { + filename: filename.to_string(), + binding: "thread-main".to_string(), + correlation: Correlation::native(value.clone()), + incarnation: Some("incarnation-1".to_string()), + legacy_floor: codex_floor( + "h.worker", + "h.worker", + "incarnation-1", + "thread-main", + FILE_A, + &value, + ), + }) + .unwrap(); + } + assert_eq!(ledger.correlated(&value), vec![FILE_A, FILE_B]); + + // And the batch is loadable: a shared correlation is anchored by the head it was sent + // under, so a restart mid-batch reads both entries back instead of quarantining. + let reopened = open(tmp.path(), Harness::Codex); + assert_eq!(reopened.quarantined(), None); + assert_eq!(reopened.correlated(&value), vec![FILE_A, FILE_B]); + + // One receipt settles every filename it carried, each on its own monotone entry. + for filename in ledger.correlated(&value) { + ledger.record(&filename, Evidence::Consumed).unwrap(); + } + assert!( + ledger + .entries() + .iter() + .all(|entry| entry.phase == Phase::Consumed) + ); + assert_eq!(ledger.retention(FILE_B), Retention::Release); + + // The recipient archiving one of them releases only that one. + ledger.prune(|filename| filename == FILE_B).unwrap(); + assert_eq!(ledger.entries().len(), 1); + assert_eq!(ledger.entries()[0].filename, FILE_B); + } + + #[test] + fn released_and_archived_entries_leave_no_residue() { + let tmp = tempfile::tempdir().unwrap(); + let legacy = legacy_path(tmp.path()); + let mut ledger = open(tmp.path(), Harness::OpenCode); + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_A); + assert!(legacy.exists()); + + // Archive precedence — the recipient's act — releases ownership even mid-attempt. + ledger.prune(|_| false).unwrap(); + assert!(ledger.entries().is_empty()); + assert!(!legacy.exists(), "no outstanding entry, no floor"); + + // And a rebind drops the other binding's entry with its floor. + begin(&mut ledger, Harness::OpenCode, "ses_old", FILE_A); + ledger.rebind("ses_new").unwrap(); + assert!(ledger.entries().is_empty()); + assert!(!legacy.exists()); + assert_eq!(ledger.binding(), None); + + // The ledger file itself is retained even when empty: its existence is what makes v1 + // adoption one-shot. + let reopened = open(tmp.path(), Harness::OpenCode); + assert!(reopened.entries().is_empty()); + assert_eq!(reopened.quarantined(), None); + } + + #[test] + fn a_crash_at_any_persistence_boundary_never_duplicates_a_held_delivery() { + // The boundaries a restart can land between, in order: floor write, ledger attempted, + // transport, evidence write. At every one, a replacement process must hold rather than + // re-transport, because nothing it can read proves the harness did not get the message. + let tmp = tempfile::tempdir().unwrap(); + let legacy = legacy_path(tmp.path()); + + // Crash after the floor write, before the ledger's own attempted record. + let value = correlate("ses_target", FILE_A); + atomic_json( + &legacy, + &opencode_floor("h.worker", "h.worker", "ses_target", FILE_A, &value), + ) + .unwrap(); + let recovered = open(tmp.path(), Harness::OpenCode); + assert_eq!( + recovered.retry(FILE_A), + RetryDecision::Hold(HoldReason::AdoptedWithoutFreshEvidence), + "the floor is adopted as the ambiguous attempt it is" + ); + + // Crash after the ledger's attempted record, before or during transport. + let mut ledger = open(tmp.path(), Harness::OpenCode); + begin(&mut ledger, Harness::OpenCode, "ses_target", FILE_A); + let recovered = open(tmp.path(), Harness::OpenCode); + assert_eq!( + recovered.retry(FILE_A), + RetryDecision::Hold(HoldReason::AmbiguousAttempt) + ); + assert!(legacy.exists(), "the rollback floor survives the crash"); + + // Crash after persistence was proved: still held, still never re-sent. + ledger.record(FILE_A, Evidence::Persisted).unwrap(); + let recovered = open(tmp.path(), Harness::OpenCode); + assert_eq!(recovered.entry(FILE_A).unwrap().phase, Phase::Persisted); + assert_eq!( + recovered.retry(FILE_A), + RetryDecision::Hold(HoldReason::UnreadReceipt) + ); + assert_eq!( + recovered.retention(FILE_A), + Retention::Hold(HoldReason::UnreadReceipt) + ); + } +} diff --git a/src/harness_context.rs b/src/harness_context.rs index 74b71998..f5843abf 100644 --- a/src/harness_context.rs +++ b/src/harness_context.rs @@ -51,10 +51,14 @@ pub const SCHEMA_V1: &str = "st2.harness-context.v1"; pub const SCHEMA_V2: &str = "st2.harness-context.v2"; /// Whether the immutable-ID writer is active. **On**, with the rest of the DELTA-003 activation -/// cohort, for the same reason as its sibling [`crate::harness_state::EMIT_SCHEMA_V2`]: the driver -/// wrappers hand this writer a raw immutable agent ID, and version 1's `agent` means a bus -/// identity. [`read`] already accepts both versions and reports which namespace each names, so the +/// cohort, for the same reason its sibling record family's selector +/// ([`crate::harness_state::EMIT_SCHEMA_V3`]) carries that cohort forward: the driver wrappers +/// hand this writer a raw immutable agent ID, and version 1's `agent` means a bus identity. +/// [`read`] already accepts both versions and reports which namespace each names, so the /// reader-first precondition is met. One named constant, one reversal point, defaulted on. +/// +/// This record family has exactly two versions and no version 3: the numeric reading is identical +/// in both, so the sibling's fault-axis cutover has nothing to do here. pub const EMIT_SCHEMA_V2: bool = true; /// The version this build writes. Coalescing and compaction-counter continuity are scoped to it: diff --git a/src/harness_state.rs b/src/harness_state.rs index 186f5c54..b9d6149e 100644 --- a/src/harness_state.rs +++ b/src/harness_state.rs @@ -54,27 +54,30 @@ pub const SCHEMA_V2: &str = "st2.harness-state.v2"; /// others. pub const SCHEMA_V3: &str = "st2.harness-state.v3"; -/// Whether the immutable-ID writer is active. **On**, with the rest of the DELTA-003 activation -/// cohort (raw-ID `ST_AGENT`, ID-keyed runtime ownership, message record version 2, PTY schema 2). +/// Whether the fault-axis writer is active. **On**: this build emits version 3. /// -/// The driver wrappers hand this writer a raw immutable agent ID, and a version suffix is the read -/// contract for this record family: stamping that ID under version 1, whose `agent` means a bus -/// identity, would misattribute it to whichever subject holds those bytes as a route. The -/// reader-first precondition is already met — [`read`] accepts both versions and reports which -/// namespace each one names. The constant stays named so the cohort remains visible and one -/// reversal point exists; it is not a per-record switch to flip alone. +/// This is the one and only writer-selection point for this record family, and version 3 +/// activation SUPERSEDED the immutable-ID selector that preceded it rather than adding a second +/// switch beside it. The DELTA-003 cohort it used to gate (raw-ID `ST_AGENT`, ID-keyed runtime +/// ownership, message record version 2, PTY schema 2) is subsumed: version 3's `agent` means the +/// immutable agent ID exactly as version 2's did, so turning this off falls back to version 2 and +/// never to version 1 — the cohort cannot be un-activated by this constant. /// -/// Version 3 activation SUPERSEDES this constant rather than adding a second selector beside it. -/// This build is reader-first again: it reads, strictly validates, and projects version 3 while -/// its writer stays on version 2, so exactly one writer-selection point exists to replace when -/// the version 3 producers land, and no seat is ever written into a shape its readers do not yet -/// interpret. -pub const EMIT_SCHEMA_V2: bool = true; +/// The reader-first precondition was met before the flip and stays met after it: [`read`] accepts +/// versions 1, 2, and 3, reports which namespace each `agent` names, and projects a legacy +/// record's condition as explicitly absent rather than as health. The fence is version-independent +/// in both directions — a version 2 claim refuses a version 3 record, and a version 3 claim +/// refuses to be superseded by one — so a mixed fleet during a rollout is safe from either side. +/// +/// The constant stays named so exactly one reversal point exists, and the `#[cfg(test)]` +/// [`Writer::with_emitted_schema`] seam keeps version 2 emission provable in this build: legacy +/// records must stay readable and the refused fence direction must stay testable. +pub const EMIT_SCHEMA_V3: bool = true; /// The version this build writes. Every ownership decision below — sequence adoption, own-record /// coalescing, heartbeat eligibility — is scoped to it: a writer owns only the shape it emits, so -/// a v1 straggler still refuses to replace a v2 record and vice versa. -const SCHEMA: &str = if EMIT_SCHEMA_V2 { SCHEMA_V2 } else { SCHEMA_V1 }; +/// a v2 straggler still refuses to replace a v3 record and vice versa. +const SCHEMA: &str = if EMIT_SCHEMA_V3 { SCHEMA_V3 } else { SCHEMA_V2 }; /// Whether a record's schema is one this version can interpret. Versions 1 and 2 describe the same /// axes and differ only in the meaning of `agent`, which [`RecordSubject`] carries. Version 3 @@ -1470,16 +1473,28 @@ impl Writer { } /// Emit a different version than [`SCHEMA`]. TEST ONLY, and deliberately not a second - /// production selector: version 3 activation replaces [`SCHEMA`] itself, which every writer - /// reads, so this seam exists purely to prove the version 3 serialization, ownership, and - /// condition semantics byte-for-byte in a build whose production writer is still on version - /// 2. + /// production selector: [`EMIT_SCHEMA_V3`] replaces [`SCHEMA`] itself, which every writer + /// reads. The seam exists because activation makes one direction of every version rule + /// unreachable from the production writer: with [`SCHEMA`] on version 3, the legacy byte + /// contract, the projection guarantee, and the refused fence direction (a version 2 claim + /// over a version 3 record) can only be exercised by naming version 2 explicitly. Both are + /// live facts — every legacy record on disk was written that way, and a rolled-back or + /// not-yet-replaced peer still writes that way — so both stay proved in one binary. + /// `pub(crate)` because the adapter modules own their own legacy-projection proofs and need + /// the same seam; it is compiled out of every shipped binary. #[cfg(test)] - fn with_emitted_schema(mut self, schema: &'static str) -> Self { + pub(crate) fn with_emitted_schema(mut self, schema: &'static str) -> Self { self.schema = schema; self } + /// [`Writer::with_emitted_schema`] in place, for a writer a constructor already handed back + /// inside a larger driver value. + #[cfg(test)] + pub(crate) fn emit_schema(&mut self, schema: &'static str) { + self.schema = schema; + } + /// Mark this writer's observation stream discontinuous: its evidence was lost and has since /// returned. The next observation opens a fresh transition even if it restates the /// pre-interruption tuple — continuity (`sinceMs`, the counter) must never be claimed across @@ -2821,8 +2836,8 @@ impl Snapshot { /// disk. Both writable shapes are decoded, because a hooks-only session must be able to claim a /// version 3 record for exactly the reasons it can claim a version 2 one — bytes it cannot decode /// would otherwise read as "somebody's live record" forever, and after the writer cutover that -/// would be every record. Split out from [`claim_wrapperless`] so the version 3 eligibility is -/// provable in a build whose own writer is still version 2. +/// would be every record. Split out from [`claim_wrapperless`] so eligibility is provable for a +/// version this build's own writer is not currently emitting, in either direction. fn wrapperless_eligible(ours: &str, current: &CurrentRecord, now_ms: u64) -> bool { let snapshot = match current { CurrentRecord::Absent => return true, @@ -3186,26 +3201,126 @@ static TMP_COUNTER: AtomicU64 = AtomicU64::new(0); mod tests { use super::*; + /// The LEGACY writer surface, pinned to version 2 through the emitted-schema seam. + /// + /// Everything reached through [`Writer::observe`] and [`Writer::ended`] is a version 2 + /// statement by construction: the legacy pair is the whole vocabulary those calls have, and + /// version 3 refuses a write whose condition axis nobody stated. Pinning the helper is what + /// keeps the compatibility seam honest after activation — these tests assert the bytes and + /// the ownership rules a version 2 record still has, which every legacy record on disk and + /// every rolled-back seat depends on. The production version 3 surface is proved through + /// [`v3_writer`] and the producer block below, which now emits exactly what this build ships. fn writer(dir: &Path) -> Writer { Writer::new(dir, "hetz.worker", "codex", Some("worker".to_string())) + .with_emitted_schema(SCHEMA_V2) } - /// A new session arriving the way real wrappers do: a written claim, then adoption. + /// A new session arriving the way real wrappers do: a written claim, then adoption. Claim and + /// writer emit the SAME version — a claim is written by the version its writer emits, so a + /// mismatched pair would be refused by the schema fence rather than testing anything. fn takeover(dir: &Path, harness: &'static str) -> Writer { + takeover_emitting(dir, harness, SCHEMA_V2) + } + + fn takeover_emitting(dir: &Path, harness: &'static str, schema: &'static str) -> Writer { let token = session_token(); - let seq = claim(dir, "hetz.worker", harness, &token).unwrap(); + let seq = claim_emitting(dir, "hetz.worker", harness, &token, schema).unwrap(); Writer::new(dir, "hetz.worker", harness, Some("worker".to_string())) + .with_emitted_schema(schema) .with_ownership(token, seq) } + /// [`claim`] and [`claim_wrapperless`] as a build emitting `schema` would perform them: the + /// same locked bodies and the same eligibility, with only the emitted version substituted. + /// The REFUSED fence direction is unreachable from a build that writes the newest version, so + /// this seam is the only way it stays provable in one binary — which is exactly what a + /// staged rollout with an older peer still running needs proved. + fn claim_emitting( + dir: &Path, + agent: &str, + harness: &'static str, + token: &str, + schema: &'static str, + ) -> anyhow::Result { + let writer = Writer::new(dir, agent, harness, None).with_emitted_schema(schema); + let _lock = writer.locked()?; + claim_locked(&writer, token) + } + + fn claim_wrapperless_emitting( + dir: &Path, + agent: &str, + harness: &'static str, + token: &str, + schema: &'static str, + ) -> anyhow::Result> { + let writer = Writer::new(dir, agent, harness, None).with_emitted_schema(schema); + let _lock = writer.locked()?; + if !wrapperless_eligible( + writer.schema, + &read_current(&writer.path), + crate::message::now_ms(), + ) { + return Ok(None); + } + claim_locked(&writer, token).map(Some) + } + + + /// The version the legacy-surface block below writes. Every test that states its observation + /// through [`Writer::observe`], [`Writer::ended`], or a written claim is a VERSION 2 test by + /// construction: the legacy pair is the whole vocabulary those calls have, and version 3 + /// refuses a write whose condition axis nobody stated. Pinning the block keeps the + /// compatibility contract — the bytes, the ownership rules, and the claim behaviour every + /// legacy record on disk and every rolled-back seat depends on — proved after activation. + const LEGACY_SCHEMA: &str = SCHEMA_V2; + + fn legacy_new( + dir: &Path, + agent: impl Into, + harness: &'static str, + pty_session: Option, + ) -> Writer { + Writer::new(dir, agent, harness, pty_session).with_emitted_schema(LEGACY_SCHEMA) + } + + fn legacy_claim( + dir: &Path, + agent: impl Into, + harness: &'static str, + token: &str, + ) -> anyhow::Result { + let agent: String = agent.into(); + claim_emitting(dir, &agent, harness, token, LEGACY_SCHEMA) + } + + fn legacy_claim_wrapperless( + dir: &Path, + agent: impl Into, + harness: &'static str, + token: &str, + ) -> anyhow::Result> { + let agent: String = agent.into(); + claim_wrapperless_emitting(dir, &agent, harness, token, LEGACY_SCHEMA) + } + /// One record on disk under an explicit schema, with a live stamp and an orphan token, so /// every eligibility clause except the schema fence says "claimable". + /// + /// The BODY follows the schema: the two writable shapes are structurally disjoint, so a + /// version 3 label over a legacy body is unreadable bytes rather than a version 3 record, and + /// planting that would test the undecodable path instead of the fence. fn planted(dir: &Path, schema: &str, agent: &str, seq: u64) -> PathBuf { let path = harness_state_path(dir); + let axes = if schema == SCHEMA_V3 { + r#""condition":{"kind":"clear"},"ask":{"kind":"none"}"# + } else { + r#""blockedOn":"none","ask":"none""# + }; fs::write( &path, format!( - r#"{{"schema":"{schema}","agent":"{agent}","harness":"codex","state":"ended","blockedOn":"none","inputBuffer":"unknown","exit":"exit 0","incarnation":"","seq":{seq},"sinceMs":1,"writtenAtMs":{},"transitions":4}}"#, + r#"{{"schema":"{schema}","agent":"{agent}","harness":"codex","state":"ended","inputBuffer":"unknown",{axes},"exit":"exit 0","incarnation":"","seq":{seq},"sinceMs":1,"writtenAtMs":{},"transitions":4}}"#, crate::message::now_ms() ), ) @@ -3442,7 +3557,7 @@ mod tests { let skew_ms = duration_ms(HARNESS_STATE_FUTURE_SKEW); let raw = |written_at_ms: u64| { serde_json::to_vec(&Record { - schema: SCHEMA.to_string(), + schema: SCHEMA_V2.to_string(), agent: "hetz.worker".to_string(), harness: "codex".to_string(), state: Activity::Active, @@ -3622,29 +3737,49 @@ mod tests { ); } - /// The activation cohort is on: the driver wrappers hand this writer a raw immutable agent ID, - /// so the record it writes must DECLARE version 2. Stamping that ID under version 1 — whose - /// `agent` means a bus identity — is the misattribution this version exists to prevent, and it - /// is exactly what the old writer default did. + /// The production writer's own declaration, after activation: this build emits version 3, so + /// the record it writes must DECLARE version 3 and carry the condition axis version 3's own + /// reader requires. The `agent` meaning the immutable-ID cohort activated is carried forward + /// unchanged — version 3's `agent` is the immutable ID exactly as version 2's was — so the + /// misattribution the cohort exists to prevent stays prevented, and stamping that ID under + /// version 1 remains impossible from this build. + /// + /// The claim fence is deliberately included: a production session's FIRST write follows a + /// fence whose axis is not a statement, so this exercises the same bootstrap every wrapper + /// and adapter does. #[test] - fn the_writer_declares_version_two_so_its_agent_field_means_an_immutable_id() { - assert!(EMIT_SCHEMA_V2); - assert_eq!(SCHEMA, SCHEMA_V2); + fn the_writer_declares_version_three_and_still_means_an_immutable_id() { + assert!(EMIT_SCHEMA_V3); + assert_eq!(SCHEMA, SCHEMA_V3); let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); - let mut writer = takeover(tmp.path(), "codex"); - writer.observe(active()).unwrap(); - let record: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); - assert_eq!(record.schema, SCHEMA_V2); - // And the reader hands that value back in the ID namespace, not as a route. + let mut writer = takeover_emitting(tmp.path(), "codex", SCHEMA); + writer + .publish(Frame::new( + Activity::Active, + InputBuffer::Unknown, + ConditionReport::Clear, + HumanAsk::None, + )) + .unwrap(); + let record = record_json(&path); + assert_eq!(record["schema"], SCHEMA_V3); + assert_eq!(record["condition"]["kind"], "clear"); + assert!( + record.get("blockedOn").is_none(), + "version 3 replaced the overloaded axis rather than keeping both: {record}" + ); + // And the reader hands the subject back in the ID namespace, not as a route. assert_eq!( read(&path, None).unwrap().subject, - Some(RecordSubject::AgentId(record.agent.clone())) + Some(RecordSubject::AgentId( + record["agent"].as_str().unwrap().to_string() + )) ); - // Writing version 2 does not retype the version-1 records already on disk: a v1 record - // still reads with bus-identity meaning, which is the whole point of keeping the pair. + // Writing version 3 does not retype the legacy records already on disk: a v1 record still + // reads with bus-identity meaning, which is the whole point of keeping the pair readable. let v1 = format!( r#"{{"schema":"st2.harness-state.v1","agent":"hetz.worker","harness":"codex","state":"active","blockedOn":"none","inputBuffer":"empty","sinceMs":1,"writtenAtMs":{},"transitions":3}}"#, crate::message::now_ms() @@ -3656,6 +3791,11 @@ mod tests { observed.subject, Some(RecordSubject::BusIdentity("hetz.worker".into())) ); + assert_eq!( + observed.condition, + ConditionView::Absent, + "a legacy record's condition is explicitly absent, never health" + ); } /// The takeover fence is ONE-WAY across supported versions. The defect this pins: during the @@ -3706,26 +3846,31 @@ mod tests { assert!(claim_may_supersede(SCHEMA_V1, Some("st2.harness-state.v4"))); } - /// This build writes v2, so the v1 → v2 migration claim is the one it can exercise - /// end-to-end: it lands exactly once, keeps the sequence monotonic across the version change - /// (the counter is not a meaning, and restarting it would sit below a lingering predecessor's - /// claim and fence the new session out), and leaves the record declaring v2. + /// This build writes v3, so the v1 → v3 migration claim is the one it exercises end-to-end: + /// it lands exactly once, keeps the sequence monotonic across the version change (the counter + /// is not a meaning, and restarting it would sit below a lingering predecessor's claim and + /// fence the new session out), and leaves the record declaring the version this build emits. + /// The claim skips no version on the way: a v1 record is superseded directly, because the + /// fence is a comparison and not a migration ladder. #[test] - fn a_version_two_claim_migrates_a_version_one_record_once_and_keeps_the_sequence() { - assert_eq!(SCHEMA, SCHEMA_V2, "this build writes the newer version"); + fn a_production_claim_migrates_a_version_one_record_once_and_keeps_the_sequence() { + assert_eq!(SCHEMA, SCHEMA_V3, "this build writes the newest version"); let tmp = tempfile::tempdir().unwrap(); let path = planted(tmp.path(), SCHEMA_V1, "hetz.worker", 7); let token = session_token(); let seq = claim(tmp.path(), "0199b8f4-8d3a-7c21-9a44-6f85b7320ea1", "codex", &token).unwrap(); assert_eq!(seq, 8, "the sequence continues past the version change"); - let record: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); - assert_eq!(record.schema, SCHEMA_V2); - assert_eq!(record.agent, "0199b8f4-8d3a-7c21-9a44-6f85b7320ea1"); + assert_eq!(record_json(&path)["schema"], SCHEMA_V3); + assert_eq!( + record_json(&path)["agent"], + "0199b8f4-8d3a-7c21-9a44-6f85b7320ea1" + ); // The claim itself is a fence, not an observation, so it reads indeterminate (`claimed`) - // and proves no subject. The session's first real observation is what a consumer joins to - // a catalog, and that must land in the ID namespace. + // and proves no subject. The session's first real statement is what a consumer joins to a + // catalog, and that must land in the ID namespace — and because a fence states nothing, + // that first statement is the one that has to state the condition axis. assert!(read(&path, None).unwrap().subject.is_none()); let mut owner = Writer::new( tmp.path(), @@ -3734,7 +3879,29 @@ mod tests { Some("worker".into()), ) .with_ownership(token, seq); - owner.observe(active()).unwrap(); + assert_eq!( + owner + .publish(Frame::new( + Activity::Active, + InputBuffer::Unknown, + ConditionReport::Unchanged, + HumanAsk::None, + )) + .unwrap(), + WriteOutcome::Refused(Refusal::Unstated), + "the fence is not carried forward, so nothing is inherited to ride on" + ); + assert!( + owner + .publish(Frame::new( + Activity::Active, + InputBuffer::Unknown, + ConditionReport::Clear, + HumanAsk::None, + )) + .unwrap() + .landed() + ); let observed = read(&path, None).unwrap(); assert_eq!(observed.state, Activity::Active); assert_eq!( @@ -3784,7 +3951,7 @@ mod tests { .unwrap(); let before = fs::read(&path).unwrap(); - let mut token_only = Writer::new(tmp.path(), "hetz.worker", "codex", Some("worker".into())); + let mut token_only = legacy_new(tmp.path(), "hetz.worker", "codex", Some("worker".into())); token_only.session = token.clone(); assert!( !token_only.observe_unless_ended(active()).unwrap(), @@ -3846,7 +4013,7 @@ mod tests { // A wrapper relaunch: claim the state record, and the sibling goes with it. let token = session_token(); - claim(&agent_dir, "hetz.worker", "claude", &token).unwrap(); + legacy_claim(&agent_dir, "hetz.worker", "claude", &token).unwrap(); assert!( harness_context::read(&context_path).is_none(), "the new incarnation must read `no context yet`" @@ -3863,7 +4030,7 @@ mod tests { ); // Claiming a seat that never had a context record is not an error. - claim(&agent_dir, "hetz.worker", "claude", &session_token()).unwrap(); + legacy_claim(&agent_dir, "hetz.worker", "claude", &session_token()).unwrap(); // …and the wrapperless boundary, which routes through the same body. It is eligible only // over a seat no wrapper holds, so it gets its own. @@ -3876,7 +4043,7 @@ mod tests { .unwrap(); let wrapperless = format!("{WRAPPERLESS_PREFIX}abc"); assert!( - claim_wrapperless(&hooks_dir, "hetz.hooked", "claude", &wrapperless) + legacy_claim_wrapperless(&hooks_dir, "hetz.hooked", "claude", &wrapperless) .unwrap() .is_some(), "the claim must actually have happened" @@ -3889,7 +4056,7 @@ mod tests { let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); let predecessor = Record { - schema: SCHEMA.to_string(), + schema: SCHEMA_V2.to_string(), agent: "hetz.worker".to_string(), harness: "codex".to_string(), state: Activity::Active, @@ -3947,10 +4114,10 @@ mod tests { // The channel and wrapper are sibling processes of ONE session and share its token — // that sharing is what makes the wrapper's terminal record the session's last word. let token = session_token(); - let mut channel = Writer::new(tmp.path(), "hetz.worker", "pi", Some("worker".into())) + let mut channel = legacy_new(tmp.path(), "hetz.worker", "pi", Some("worker".into())) .with_session(token.clone()); let mut wrapper = - Writer::new(tmp.path(), "hetz.worker", "pi", Some("worker".into())).with_session(token); + legacy_new(tmp.path(), "hetz.worker", "pi", Some("worker".into())).with_session(token); assert!(channel.observe_unless_ended(active()).unwrap()); wrapper.ended("signal 9").unwrap(); @@ -4041,7 +4208,7 @@ mod tests { let mut writer = takeover(tmp.path(), "codex"); assert!(writer.observe_unless_ended(active()).unwrap()); let record: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); - assert_eq!(record.schema, SCHEMA); + assert_eq!(record.schema, SCHEMA_V2); assert_eq!(record.state, Activity::Active); assert_eq!(record.transitions, 9); } @@ -4069,7 +4236,7 @@ mod tests { fn live_observations_require_a_pty_session_and_unfenced_live_records_read_unknown() { let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); - let mut unfenced = Writer::new(tmp.path(), "hetz.worker", "codex", None); + let mut unfenced = legacy_new(tmp.path(), "hetz.worker", "codex", None); assert!( unfenced.observe(active()).is_err(), "live states need a fence" @@ -4183,16 +4350,16 @@ mod tests { let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); let token = session_token(); - let mut wrapper = Writer::new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) + let mut wrapper = legacy_new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) .with_session(token.clone()); - let mut hook = Writer::new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) + let mut hook = legacy_new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) .with_session(token); hook.observe(active()).unwrap(); let entered: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); // A sibling's restatement coalesces (no transition churn across hook processes)… - let mut hook2 = Writer::new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) + let mut hook2 = legacy_new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) .with_session(entered.incarnation.clone()); hook2.observe(active()).unwrap(); let restated: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); @@ -4248,8 +4415,8 @@ mod tests { // The wrapper writes the claim and exports it; the hook adopts the pair. let token = session_token(); - let seq = claim(tmp.path(), "hetz.worker", "claude", &token).unwrap(); - let mut hook = Writer::new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) + let seq = legacy_claim(tmp.path(), "hetz.worker", "claude", &token).unwrap(); + let mut hook = legacy_new(tmp.path(), "hetz.worker", "claude", Some("worker".into())) .with_ownership(token.clone(), seq); hook.observe(Observation::new( Activity::Idle, @@ -4260,7 +4427,7 @@ mod tests { assert_eq!(read(&path, None).unwrap().state, Activity::Idle); // A later session claims past it; the adopted writer becomes the straggler. - let mut next = Writer::new(tmp.path(), "hetz.worker", "claude", Some("worker".into())); + let mut next = legacy_new(tmp.path(), "hetz.worker", "claude", Some("worker".into())); next.observe(active()).unwrap(); let after = fs::read(&path).unwrap(); hook.observe(Observation::new( @@ -4306,7 +4473,7 @@ mod tests { .map(|_| { let dir = dir.clone(); std::thread::spawn(move || { - claim(&dir, "hetz.worker", "codex", &session_token()).unwrap() + legacy_claim(&dir, "hetz.worker", "codex", &session_token()).unwrap() }) }) .collect(); @@ -4326,7 +4493,7 @@ mod tests { writer(tmp.path()).observe(active()).unwrap(); let token = session_token(); - let seq = claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); + let seq = legacy_claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); let observed = read(&path, None).unwrap(); assert_eq!( observed.state, @@ -4336,7 +4503,7 @@ mod tests { assert_eq!(observed.reason.as_deref(), Some("claimed")); assert_eq!(observed.exit, None); - let mut successor = Writer::new(tmp.path(), "hetz.worker", "codex", Some("worker".into())) + let mut successor = legacy_new(tmp.path(), "hetz.worker", "codex", Some("worker".into())) .with_ownership(token, seq); successor.observe(active()).unwrap(); assert_eq!(read(&path, None).unwrap().state, Activity::Active); @@ -4353,7 +4520,7 @@ mod tests { fs::write(&path, v2).unwrap(); let token = session_token(); - let mut adopted = Writer::new(tmp.path(), "hetz.worker", "codex", Some("worker".into())) + let mut adopted = legacy_new(tmp.path(), "hetz.worker", "codex", Some("worker".into())) .with_ownership(token, 5); adopted.observe(active()).unwrap(); assert_eq!( @@ -4372,7 +4539,7 @@ mod tests { let mut claimed = takeover(tmp.path(), "codex"); claimed.observe(active()).unwrap(); let record: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); - assert_eq!(record.schema, SCHEMA, "only the written claim supersedes"); + assert_eq!(record.schema, SCHEMA_V2, "only the written claim supersedes"); } /// W8-7: a beyond-skew future stamp (a backward clock correction's leftover) must not make @@ -4408,21 +4575,21 @@ mod tests { fn wrapperless_claims_are_atomic_and_never_supersede_a_live_wrapper() { let tmp = tempfile::tempdir().unwrap(); let wl = - |token: &str| claim_wrapperless(tmp.path(), "hetz.worker", "claude", token).unwrap(); + |token: &str| legacy_claim_wrapperless(tmp.path(), "hetz.worker", "claude", token).unwrap(); assert!(wl("claude-session-a").is_some(), "virgin dir"); // A wrapper's FRESH claim placeholder is a session mid-startup, not an ended one: the // check-and-write is one act under the lock, so the racing hooks-only SessionStart // cannot steal the sequence between the wrapper's read and its write. let wrapper_token = session_token(); - let wrapper_seq = claim(tmp.path(), "hetz.worker", "claude", &wrapper_token).unwrap(); + let wrapper_seq = legacy_claim(tmp.path(), "hetz.worker", "claude", &wrapper_token).unwrap(); assert!( wl("claude-session-b").is_none(), "fresh placeholder is owned" ); // A live wrapper record stays off limits; a REAL terminal record is claimable. - let mut wrapper = Writer::new( + let mut wrapper = legacy_new( tmp.path(), "hetz.worker", "claude", @@ -4447,9 +4614,9 @@ mod tests { fn an_abandoned_wrapper_placeholder_is_claimable_once_stale() { let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); - claim(tmp.path(), "hetz.worker", "claude", &session_token()).unwrap(); + legacy_claim(tmp.path(), "hetz.worker", "claude", &session_token()).unwrap(); assert!( - claim_wrapperless(tmp.path(), "hetz.worker", "claude", "claude-session-x") + legacy_claim_wrapperless(tmp.path(), "hetz.worker", "claude", "claude-session-x") .unwrap() .is_none() ); @@ -4458,7 +4625,7 @@ mod tests { aged.written_at_ms = crate::message::now_ms() - duration_ms(HARNESS_STATE_STALE) - 1; write_record(&path, &aged).unwrap(); assert!( - claim_wrapperless(tmp.path(), "hetz.worker", "claude", "claude-session-x") + legacy_claim_wrapperless(tmp.path(), "hetz.worker", "claude", "claude-session-x") .unwrap() .is_some() ); @@ -4475,7 +4642,7 @@ mod tests { u64::MAX ); fs::write(&path, saturated).unwrap(); - assert!(claim(tmp.path(), "hetz.worker", "codex", "t").is_err()); + assert!(legacy_claim(tmp.path(), "hetz.worker", "codex", "t").is_err()); fs::write(&path, b"{not json").unwrap(); let before = fs::read(&path).unwrap(); @@ -4486,7 +4653,7 @@ mod tests { before, "non-claiming writers refuse" ); - let mut adopted = Writer::new(tmp.path(), "hetz.worker", "codex", Some("worker".into())) + let mut adopted = legacy_new(tmp.path(), "hetz.worker", "codex", Some("worker".into())) .with_ownership(session_token(), 7); adopted.observe(active()).unwrap(); assert_eq!( @@ -4497,7 +4664,7 @@ mod tests { // The written claim supersedes even bytes it cannot parse; sequence and counter restart. let token = session_token(); - let seq = claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); + let seq = legacy_claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); assert_eq!(seq, 1); let record: Record = serde_json::from_slice(&fs::read(&path).unwrap()).unwrap(); assert_eq!(record.transitions, 0); @@ -4516,7 +4683,7 @@ mod tests { fs::write(&path, b"{corrupted").unwrap(); let token = session_token(); - let seq = claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); + let seq = legacy_claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); assert!( seq > damaged_seq.seq, "the floor carries the sequence past the damage ({seq} vs {})", @@ -4552,7 +4719,7 @@ mod tests { // lingering token-only predecessor stays fenced out. fs::write(&path, b"{corrupted").unwrap(); let token = session_token(); - let seq = claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); + let seq = legacy_claim(tmp.path(), "hetz.worker", "codex", &token).unwrap(); assert!( seq > 1, "the persisted floor carries the claim past sequence one ({seq})" @@ -4582,7 +4749,7 @@ mod tests { )) .unwrap(); assert!( - claim_wrapperless(tmp.path(), "hetz.worker", "claude", "claude-session-x") + legacy_claim_wrapperless(tmp.path(), "hetz.worker", "claude", "claude-session-x") .unwrap() .is_none(), "wrapperless claims refuse unreadable records" @@ -5222,13 +5389,18 @@ mod tests { ) } - /// Reader-first means exactly this: this build READS a version 3 record and refuses to touch - /// it. The defect being fenced is subtle — a v3 record does not decode as this build's record - /// shape at all, so without the version-independent envelope the claim path would see "no - /// record on disk" and rename a version 2 claim over a live migrated record, destroying its - /// fault axis with no trace that it happened. + /// The one-way fence, now driven from the side activation left behind: a build whose writer + /// emits version 2 READS a version 3 record and refuses to touch it, by every path. This is + /// the rollout hazard in a mixed fleet — a not-yet-replaced peer, or a rolled-back binary, + /// running beside an activated one — and the defect it fences is subtle: a v3 record does not + /// decode as the legacy record shape at all, so without the version-independent envelope the + /// claim path would see "no record on disk" and rename a version 2 claim over a live migrated + /// record, destroying its fault axis with no trace that it happened. + /// + /// The version 2 writer is reached through the emitted-schema seam because this build's own + /// production writer is on version 3, where this direction is unreachable by construction. #[test] - fn a_version_three_record_is_read_but_never_written_over_by_this_builds_writer() { + fn a_version_two_writer_never_takes_over_a_version_three_record() { let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); let planted = format!( @@ -5237,7 +5409,7 @@ mod tests { ); fs::write(&path, &planted).unwrap(); - // Reading works — that is the whole point of shipping the reader first. + // Reading works from either side — that is what makes a staged rollout safe at all. let observed = read(&path, None).unwrap(); assert_eq!(observed.condition, ConditionView::Clear); assert_eq!(observed.schema.as_deref(), Some(SCHEMA_V3)); @@ -5247,19 +5419,35 @@ mod tests { let mut token_only = writer(tmp.path()); assert!(!token_only.observe_unless_ended(active()).unwrap()); assert!( - claim_wrapperless(tmp.path(), "hetz.worker", "codex", "claude-session-x") - .unwrap() - .is_none() + claim_wrapperless_emitting( + tmp.path(), + "hetz.worker", + "codex", + "claude-session-x", + SCHEMA_V2 + ) + .unwrap() + .is_none() ); - let error = claim(tmp.path(), "hetz.worker", "codex", &session_token()) - .unwrap_err() - .to_string(); + let error = claim_emitting( + tmp.path(), + "hetz.worker", + "codex", + &session_token(), + SCHEMA_V2, + ) + .unwrap_err() + .to_string(); assert!(error.contains(SCHEMA_V3), "{error}"); assert_eq!( fs::read(&path).unwrap(), planted.as_bytes(), "the version 3 record is left byte-identical by every path" ); + + // And the other direction is open, which is what makes the fence one-way rather than a + // deadlock: this build's own production claim supersedes it. + assert!(claim(tmp.path(), FIXTURE_AGENT_ID, "codex", &session_token()).is_ok()); } /// The other half of the envelope's job: bytes this build cannot decode as a record still @@ -5278,7 +5466,7 @@ mod tests { ) .unwrap(); - let seq = claim(tmp.path(), "hetz.worker", "codex", &session_token()).unwrap(); + let seq = legacy_claim(tmp.path(), "hetz.worker", "codex", &session_token()).unwrap(); assert_eq!( seq, 10, "the ownership sequence continues past bytes this build cannot read" @@ -5291,11 +5479,12 @@ mod tests { } // ----------------------------------------------------------------------- - // The version 3 producer surface. Every one of these runs the REAL locked - // writer path — same lock, same fences, same atomic rename — and the ones - // that need version 3 bytes get them through the test-only emitted-schema - // seam, because this build's production writer is still version 2 (proved - // by `the_writer_declares_version_two_so_its_agent_field_means_an_immutable_id`). + // The version 3 producer surface — this build's PRODUCTION surface since + // activation. Every one of these runs the REAL locked writer path: same + // lock, same fences, same atomic rename. `v3_writer` names the version + // explicitly rather than relying on `SCHEMA`, so these keep proving the + // version 3 semantics even from a build that has been rolled back (proved + // current by `the_writer_declares_version_three_and_still_means_an_immutable_id`). // ----------------------------------------------------------------------- fn v3_writer(dir: &Path) -> Writer { @@ -5328,12 +5517,13 @@ mod tests { ) } - /// The rollout guarantee: while `EMIT_SCHEMA_V2` is true the producer API is a projection and - /// nothing more. The same tuple stated through the legacy API and through the version 3 API - /// produces the same bytes — the projection coalesces against the legacy record rather than - /// writing — the condition and conversation axes never reach the version 2 wire, and the - /// condition operations refuse as a VALUE rather than pretending to have stored an axis the - /// wire cannot carry. + /// The COMPATIBILITY guarantee, kept after activation: a writer emitting version 2 makes the + /// producer API a projection and nothing more. The same tuple stated through the legacy API + /// and through the version 3 API produces the same bytes — the projection coalesces against + /// the legacy record rather than writing — the condition and conversation axes never reach + /// the version 2 wire, and the condition operations refuse as a VALUE rather than pretending + /// to have stored an axis the wire cannot carry. This is what a rollback gets, and what every + /// version 2 record still on disk was written by. #[test] fn the_version_two_projection_is_byte_identical_and_carries_no_condition_axis() { let tmp = tempfile::tempdir().unwrap(); @@ -5840,8 +6030,8 @@ mod tests { // The claim fence: a session that has claimed the seat and observed nothing. Its required // condition axis is not a statement, so nothing attaches to it and nothing is carried out - // of it — the fence is planted here in version 3 shape because this build's own [`claim`] - // writes the version it emits, which is still 2. + // of it — planted rather than claimed so the fence's exact bytes are the subject, not + // whichever version [`claim`] happens to emit. let claimed = tempfile::tempdir().unwrap(); let fence_path = harness_state_path(claimed.path()); let token = session_token(); @@ -6045,8 +6235,8 @@ mod tests { /// Wrapperless eligibility decodes BOTH writable shapes. A hooks-only session must be able to /// claim a version 3 record for exactly the reasons it can claim a version 2 one — otherwise, /// after the writer cutover, every record would look like undecodable bytes and no wrapperless - /// session could ever claim again. Decided as a pure function so the version 3 arm is provable - /// while this build's own writer is still version 2. + /// session could ever claim again. Decided as a pure function so BOTH arms stay provable from + /// one build whichever version its own writer emits. #[test] fn wrapperless_eligibility_decodes_version_three_records_too() { let now_ms = crate::message::now_ms(); @@ -6098,8 +6288,9 @@ mod tests { ), now_ms )); - // The one-way schema fence still holds: this build's version 2 writer has no eligible - // takeover of a version 3 record, however orphaned it looks. + // The one-way schema fence still holds from the other side: a version 2 writer — a peer + // not yet replaced, or a rollback — has no eligible takeover of a version 3 record, + // however orphaned it looks. assert!(!wrapperless_eligible( SCHEMA_V2, &v3( @@ -6121,18 +6312,181 @@ mod tests { &CurrentRecord::Unreadable, now_ms )); - // And the legacy path is unchanged: the live wrapper record this build writes stays - // unclaimable and its terminal one stays claimable. + // And the legacy path is unchanged: a live wrapper record written by a version 2 writer + // stays unclaimable to a version 2 claimer, and its terminal one stays claimable. let tmp = tempfile::tempdir().unwrap(); let path = harness_state_path(tmp.path()); let mut wrapper = writer(tmp.path()); wrapper.observe(active()).unwrap(); assert!(!wrapperless_eligible( - SCHEMA, + SCHEMA_V2, &read_current(&path), now_ms )); wrapper.ended("exit 0").unwrap(); - assert!(wrapperless_eligible(SCHEMA, &read_current(&path), now_ms)); + assert!(wrapperless_eligible( + SCHEMA_V2, + &read_current(&path), + now_ms + )); + } + + // ----------------------------------------------------------------------- + // Activation proofs: what changed at the flip, and what must not have. + // ----------------------------------------------------------------------- + + /// A legacy `observe` call still works after the cutover — but only on top of a stated axis, + /// and it projects into the version 3 wire rather than into legacy bytes. This is what every + /// unmigrated legacy call site now does: the tagged ask is derived from the legacy pair, the + /// standing condition is carried forward untouched, and `blockedOn` is gone from the bytes. + #[test] + fn a_legacy_observation_projects_into_the_version_three_wire() { + let tmp = tempfile::tempdir().unwrap(); + let path = harness_state_path(tmp.path()); + let token = session_token(); + let mut producer = v3_writer(tmp.path()).with_session(token.clone()); + let fault = quota_fault(crate::message::now_ms() - 5_000); + assert!( + producer + .publish(v3_active(ConditionReport::Fault(fault.clone()))) + .unwrap() + .landed() + ); + + let mut legacy = v3_writer(tmp.path()).with_session(token); + assert!( + legacy + .observe_unless_ended( + Observation::new(Activity::Idle, BlockedOn::Human, InputBuffer::Empty) + .with_ask(Ask::Permission) + ) + .unwrap() + ); + let record = record_json(&path); + assert_eq!(record["schema"], SCHEMA_V3); + assert!( + record.get("blockedOn").is_none(), + "the legacy pair is projected, not appended: {record}" + ); + assert_eq!(record["ask"]["kind"], "pending"); + assert_eq!(record["ask"]["ask"], "permission"); + assert_eq!( + record["condition"]["kind"], "fault", + "an activity edge carries a standing fault forward and never clears it" + ); + let observed = read(&path, None).unwrap(); + assert_eq!(observed.state, Activity::Idle); + assert_eq!(observed.human_ask, HumanAsk::Pending(AskKind::Permission)); + assert!(matches!(observed.condition, ConditionView::Fault(_))); + } + + /// THE activation test. A wrapper is the one process that sees the provider die, and on many + /// incarnations its terminal write is the session's FIRST write: the claim fence states + /// nothing and is excluded from carry-forward, so version 3 refuses an unstated terminal + /// frame. The shared wrapper path states the axis once and the record lands anyway — without + /// it, wrapper-only `ended` would silently stop working and every such seat would age into + /// `unknown` instead of reporting how it ended. + #[test] + fn a_terminal_record_lands_even_when_no_producer_ever_stated_the_condition_axis() { + let tmp = tempfile::tempdir().unwrap(); + let path = harness_state_path(tmp.path()); + let token = session_token(); + let seq = claim(tmp.path(), FIXTURE_AGENT_ID, "pi", &token).unwrap(); + assert_eq!(record_json(&path)["schema"], SCHEMA_V3); + assert!( + read(&path, None).unwrap().state != Activity::Ended, + "the fence is indeterminate, not a terminal record" + ); + + let mut wrapper = + Writer::new(tmp.path(), FIXTURE_AGENT_ID, "pi", Some("worker".to_string())) + .with_ownership(token, seq); + // The legacy terminal call is exactly what refuses, which is why the wrappers moved. + assert!( + matches!( + wrapper + .publish( + Frame::new( + Activity::Ended, + InputBuffer::Unknown, + ConditionReport::Unchanged, + HumanAsk::None + ) + .with_exit("signal 9") + ) + .unwrap(), + WriteOutcome::Refused(Refusal::Unstated) + ), + "an unstated axis over a fence is refused, and that refusal is the retry's premise" + ); + crate::provider_session::write_terminal( + &mut wrapper, + "signal 9", + None, + ConditionReport::Clear, + ) + .unwrap(); + + let observed = read(&path, None).unwrap(); + assert_eq!(observed.state, Activity::Ended); + assert_eq!(observed.exit.as_deref(), Some("signal 9")); + assert_eq!( + observed.condition, + ConditionView::Clear, + "the wrapper states the axis once rather than fabricating a fault" + ); + } + + /// The fence from the ACTIVATED side: this build's claim supersedes every supported version, + /// including its own, and no supported version supersedes it. Both halves matter during a + /// staged rollout — the first is why an activated binary can adopt any seat it finds, the + /// second is why a peer that has not been replaced cannot undo that. + #[test] + fn a_version_three_claim_supersedes_every_supported_version_and_is_never_superseded() { + for found in [SCHEMA_V1, SCHEMA_V2, SCHEMA_V3] { + assert!( + claim_may_supersede(SCHEMA_V3, Some(found)), + "a version 3 claim must supersede {found}" + ); + } + for ours in [SCHEMA_V1, SCHEMA_V2] { + assert!( + !claim_may_supersede(ours, Some(SCHEMA_V3)), + "{ours} must never supersede a version 3 record" + ); + } + + // End to end, through the real locked claim: a version 2 record on disk is superseded by + // this build and the sequence continues across the version change. + let tmp = tempfile::tempdir().unwrap(); + let path = planted(tmp.path(), SCHEMA_V2, FIXTURE_AGENT_ID, 5); + let seq = claim(tmp.path(), FIXTURE_AGENT_ID, "codex", &session_token()).unwrap(); + assert_eq!(seq, 6); + assert_eq!(record_json(&path)["schema"], SCHEMA_V3); + } + + /// Legacy readability, after the writer moved: the frozen version 1 and version 2 fixtures + /// still project every axis a consumer joins to, and their condition reads as explicitly + /// ABSENT rather than as health. Nothing about the cutover may turn a record written by an + /// older binary into a claim that its harness was fine. + #[test] + fn a_legacy_record_is_still_read_and_projected_after_the_writer_cutover() { + assert_eq!(SCHEMA, SCHEMA_V3, "this is the post-cutover statement"); + for raw in [ + include_str!("../tests/fixtures/harness-state/v1-active.json"), + include_str!("../tests/fixtures/harness-state/v2-blocked.json"), + ] { + let observed = fixture(raw); + assert!( + observed.state != Activity::Unknown, + "a legacy record is a definite observation: {observed:?}" + ); + assert!(observed.subject.is_some()); + assert_eq!( + observed.condition, + ConditionView::Absent, + "a version without the axis states nothing on it: {observed:?}" + ); + } } } diff --git a/src/lib.rs b/src/lib.rs index a7da54e0..dced4486 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -18,6 +18,7 @@ pub mod claude_mcp; pub mod claude_session; pub mod codex_app_server; pub mod context; +pub mod delivery_ledger; pub mod ding; pub mod driver; pub mod driver_diagnostic; diff --git a/src/omp_session.rs b/src/omp_session.rs index fd494a67..4e3b88d5 100644 --- a/src/omp_session.rs +++ b/src/omp_session.rs @@ -18,7 +18,8 @@ use std::process::ExitStatus; use anyhow::{Context as _, Result}; use crate::provider_session::{ - install_signal_handler, run_provider_observed, ProviderOutcome, PROVIDER_POLL, STOP, + PROVIDER_POLL, ProviderOutcome, STOP, install_signal_handler, run_provider_observed, + write_terminal, }; use crate::{harness_state, harness_version, hooks, message, status}; @@ -121,14 +122,15 @@ pub fn run( Some(runtime_id.clone()), ) .with_ownership(session.clone(), seq); - let _ = writer.observe( - harness_state::Observation::new( - harness_state::Activity::Ended, - harness_state::BlockedOn::None, - harness_state::InputBuffer::Unknown, - ) - .with_reason("launch-error") - .with_exit("exit unknown"), + // The launch error is this incarnation's FIRST write: the claim fence above states + // no condition and fences are never carried forward, so the version 3 write needs + // the one bootstrap retry [`write_terminal`] owns or the seat would keep reading a + // takeover placeholder instead of a launch that never ran. + let _ = write_terminal( + &mut writer, + "exit unknown", + Some("launch-error"), + harness_state::ConditionReport::Clear, ); return Err(error); } @@ -208,7 +210,15 @@ fn record_session_end( let mut writer = harness_state::Writer::new(agent_dir, identity, "omp", Some(runtime_id.to_string())) .with_ownership(session, seq); - if let Err(error) = writer.ended(label) { + // The channel may have published nothing at all — an omp that died before loading its + // extension is exactly that — so this terminal write states the condition axis itself when + // nothing else did, and otherwise carries the channel's standing condition forward. + if let Err(error) = write_terminal( + &mut writer, + &label, + None, + harness_state::ConditionReport::Clear, + ) { eprintln!("st2 omp driver: recording session end failed: {error}"); } } diff --git a/src/opencode_session.rs b/src/opencode_session.rs index 056c18f8..4c817777 100644 --- a/src/opencode_session.rs +++ b/src/opencode_session.rs @@ -6,16 +6,19 @@ //! stream and projects session status, permission asks, and questions into the generic //! `harness-state` record — evidence-gated, so a dropped stream stops the heartbeat and the record //! ages out rather than restating a state nobody is watching. The delivery pump mirrors the Codex -//! FIFO discipline: an `Attempted` receipt is persisted before transport, the transport is -//! `POST /session//prompt_async` with a caller-derived stable `messageID`, and the only -//! accepted receipt is the message read back from the server — never the `/tui/*` endpoints, which -//! acknowledge input even when no TUI is attached. +//! FIFO discipline through the shared [`crate::delivery_ledger`]: an `attempted` entry is durable +//! before transport, the transport is `POST /session//prompt_async` with a caller-derived +//! stable `messageID`, and the only receipt this transport can produce is the message read back +//! from the server — never the `/tui/*` endpoints, which acknowledge input even when no TUI is +//! attached. That read-back is graded as `persisted`, because it proves the server STORED the +//! message and says nothing about the session scheduler admitting it; persistence therefore holds +//! the inbox entry rather than releasing it, and is never mistaken for consumption. //! //! Fail-closed gate: delivery requires both a supported `opencode --version` and a live `/doc` //! subset check proving the exact API arms st2 depends on. Observation requires only the `/doc` //! check — its vocabulary already degrades to indeterminate on anything unrecognized. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::io::{BufRead as _, BufReader, Read as _, Write as _}; use std::net::{TcpListener, TcpStream}; use std::os::unix::process::ExitStatusExt as _; @@ -27,7 +30,6 @@ use std::thread; use std::time::{Duration, Instant}; use anyhow::{Context as _, Result}; -use serde::{Deserialize, Serialize}; use serde_json::{Value, json}; use sha2::{Digest as _, Sha256}; @@ -35,9 +37,13 @@ use crate::driver_diagnostic::{ Driver as DiagnosticDriver, Publisher as DiagnosticPublisher, Reason as DiagnosticReason, Source as DiagnosticSource, Stage as DiagnosticStage, Support as DiagnosticSupport, }; -use crate::harness_state::{self, Activity, Ask, BlockedOn, InputBuffer, Observation, Writer}; +use crate::harness_state::{ + self, Activity, Ask, AskKind, BlockedOn, CapabilityEvidence, ConditionReport, ConversationClaim, + ConversationState, FaultCategory, FaultKey, FaultReport, Frame, HistoryMutability, HumanAsk, + InputBuffer, Observation, ProgressProof, Recovery, Refusal, WriteOutcome, Writer, +}; use crate::provider_session::{PROVIDER_POLL, STOP, install_signal_handler}; -use crate::{ding, harness_context, harness_version, message, status}; +use crate::{delivery_ledger, ding, harness_context, harness_version, message, status}; /// OpenCode MINORS whose `/event`, `/session`, and `prompt_async` surfaces were verified /// (1.18, measured at 1.18.19). The live `/doc` check below guards the shape; this list guards @@ -50,7 +56,6 @@ use crate::{ding, harness_context, harness_version, message, status}; /// risk this widening accepts. A new MINOR still needs the surfaces re-verified. const SUPPORTED_OPENCODE_MINORS: [(u32, u32); 1] = [(1, 18)]; -const DELIVERY_STATE_SCHEMA: &str = "st2.opencode-delivery-state.v1"; const STOP_GRACE: Duration = Duration::from_secs(5); const INBOX_REFRESH_FALLBACK: Duration = Duration::from_secs(2); const DELIVERY_RETRY: Duration = Duration::from_secs(2); @@ -188,10 +193,14 @@ pub fn run( // would leave the exitless `ended (superseded)` placeholder standing as a false // takeover. The launch failed under THIS session's ownership, so the record ends // honestly here instead (the same contract pi's launch path keeps). - let _ = session.writer.observe( - Observation::new(Activity::Ended, BlockedOn::None, InputBuffer::Unknown) - .with_reason("launch-error") - .with_exit("exit unknown"), + // No projection exists yet — the provider never started — so the axis this states, + // if the record still needs it stated, is the empty machine's: no standing fault. + // The launch failure itself rides the record's `reason` and `exit`. + write_terminal( + &mut session.writer, + "exit unknown", + Some("launch-error"), + EventMachine::default().bootstrap_condition(), ); return Err(error); } @@ -241,16 +250,31 @@ fn run_session(mut session: Session, child: &mut Child, agent_dir: &Path) -> Res // the group yields inside the grace window the wrapper survives, so the record is // rewritten with the exit the reap actually observed — "stopped" remains only as // escalation cover. - let _ = session.writer.ended("stopped"); + write_terminal( + &mut session.writer, + "stopped", + None, + machine.bootstrap_condition(), + ); let reaped = stop_provider_group(child); if let Ok(Some(exit)) = &reaped { - let _ = session.writer.ended(describe_exit(*exit)); + write_terminal( + &mut session.writer, + &describe_exit(*exit), + None, + machine.bootstrap_condition(), + ); } break reaped.map(|_| ()); } match child.try_wait() { Ok(Some(exit)) => { - let _ = session.writer.ended(describe_exit(exit)); + write_terminal( + &mut session.writer, + &describe_exit(exit), + None, + machine.bootstrap_condition(), + ); break completed(exit); } Ok(None) => {} @@ -258,10 +282,11 @@ fn run_session(mut session: Session, child: &mut Child, agent_dir: &Path) -> Res // The liveness check failing is a terminal outcome too: without this write the // claim placeholder stands as the visible state (the self-review's error-arm // class). - let _ = session.writer.observe( - Observation::new(Activity::Ended, BlockedOn::None, InputBuffer::Unknown) - .with_reason("launch-error") - .with_exit("exit unknown"), + write_terminal( + &mut session.writer, + "exit unknown", + Some("launch-error"), + machine.bootstrap_condition(), ); return Err(error).context("checking opencode provider"); } @@ -309,7 +334,10 @@ fn run_session(mut session: Session, child: &mut Child, agent_dir: &Path) -> Res while let Ok(event) = event_rx.try_recv() { match event { SseMessage::Connected => { - machine = EventMachine::default(); + // The ACTIVITY axis restarts from the level seed below; the standing faults + // and the conversation identity ride across, because the level surface + // states neither and a blip must not silence a live fault. + machine = machine.reconnected(); sse_connected = true; session.diagnostics.clear(DiagnosticStage::Sse); // Evidence turns on only once the level seed succeeds: resuming heartbeats @@ -354,7 +382,12 @@ fn run_session(mut session: Session, child: &mut Child, agent_dir: &Path) -> Res context.publish(); } } - if machine.poisoned && machine.ended.is_none() && evidence { + // Which projection a poisoned status word withholds depends on the emitted version: the + // legacy one keeps a sticky terminal that outranks poison, the version 3 one has no + // terminal arm at all, so it is withheld outright until a seed rebuilds the activity. + let withheld = + machine.poisoned && (session.writer.writes_condition_axis() || machine.ended.is_none()); + if withheld && evidence { // The projection went untrustworthy mid-stream: stop heartbeating over it and let // the level seed rebuild the whole picture from the server's own truth. evidence = false; @@ -373,8 +406,44 @@ fn run_session(mut session: Session, child: &mut Child, agent_dir: &Path) -> Res &mut session.diagnostics, ); } - if evidence && let Some(observation) = machine.observation() { - let _ = session.writer.observe(observation); + if session.writer.writes_condition_axis() { + // ORDER MATTERS. A condition operation attaches to an OBSERVATION: against a claim + // fence — which has observed nothing — it is refused as `Unobserved`, and a frame + // whose condition is `Unchanged` over that same fence has no axis to inherit and is + // refused as `Unstated`. So the frame goes first and states the axis when the record + // needs it stated, and only then do the queued edges move it. + // + // While there is no evidence nothing is written at all AND the queue is HELD: the + // faults were observed on frames that did arrive, and a lost stream must not silence + // them — it only delays the write until the record has an observation to carry it. + if evidence { + if let Some(frame) = machine.frame() { + let outcome = session.writer.publish(frame); + if matches!(&outcome, Ok(WriteOutcome::Refused(Refusal::Unstated))) { + // The one retry: this session's first frame, stating the axis the fence + // left absent as whatever the reducer projects right now. + if let Some(stated) = machine.frame_with(machine.bootstrap_condition()) { + note_write(session.writer.publish(stated)); + } + } else { + note_write(outcome); + } + } + for edge in machine.take_edges() { + note_write(match edge { + ConditionEdge::Raise(fault) => session.writer.raise_fault(fault), + ConditionEdge::ClearPaired(key) => session.writer.clear_fault(key), + ConditionEdge::ClearAll(proof) => session.writer.clear_all(proof), + }); + } + } + } else { + // The version 2 wire has nowhere to carry a condition, so the queue is drained and + // dropped rather than growing for a writer that can never state it. + let _ = machine.take_edges(); + if evidence && let Some(observation) = machine.observation() { + let _ = session.writer.observe(observation); + } } let now = Instant::now(); @@ -405,6 +474,47 @@ fn run_session(mut session: Session, child: &mut Child, agent_dir: &Path) -> Res outcome } +/// A version 3 write's outcome, logged rather than dropped. Landing and coalescing are both +/// success; a refusal — a foreign schema, a later session's claim, a paired clear that matched +/// nothing — is a value, and this record family has exactly one store and no second place to +/// report it through. +fn note_write(outcome: Result) { + match outcome { + Ok(outcome) => { + if let Some(refusal) = outcome.refusal() { + tracing::warn!("st2 opencode-session: harness-state write refused: {refusal:?}"); + } + } + Err(error) => { + tracing::warn!("st2 opencode-session: harness-state write failed: {error:#}"); + } + } +} + +/// THE terminal record for this driver's four process-exit owners — the launch failure, the STOP +/// path and its reap, the ordinary child exit, and the failed liveness check. +/// +/// The vocabulary decision, the `Unchanged`-first attempt, and the one-shot `Unstated` retry are +/// [`crate::provider_session::write_terminal`]'s: every terminal owner in the crate must write +/// the same shape, so that rule lives in one place. What stays here is what is OpenCode's alone — +/// the `bootstrap` this driver hands it is the axis its own reducer projects for the winning +/// session rather than a blanket `clear`, and a refusal is reported through this module's +/// [`note_write`] rather than swallowed. +fn write_terminal( + writer: &mut Writer, + exit: &str, + reason: Option<&str>, + bootstrap: ConditionReport, +) { + match crate::provider_session::write_terminal(writer, exit, reason, bootstrap) { + // The legacy surface has no typed outcome: it made the exact statement `Writer::ended` + // makes and there is nothing to report. + Ok(None) => {} + Ok(Some(outcome)) => note_write(Ok(outcome)), + Err(error) => note_write(Err(error)), + } +} + fn spawn_provider(argv: &[String], password: &str) -> Result { use std::os::unix::process::CommandExt as _; let (program, args) = argv @@ -740,15 +850,21 @@ fn seed_from_server( DiagnosticReason::MalformedStatus, DiagnosticSource::StatusSnapshot, ))?; - let mut seeded = EventMachine::default(); + // The ACTIVITY axis is what the level surface owns; the standing faults and the conversation + // identity ride across the swap, because `/session/status` states neither. + let mut seeded = machine.reconnected(); seeded.seed_idle(); + let mut retrying_now = BTreeSet::new(); for (session_id, status) in map { // Exactly the pinned vocabulary: an unknown future word is not "busy" — it is surface // drift the /doc gate vocabulary did not cover. match status.get("type").and_then(Value::as_str) { Some("idle") => {} Some("busy") => seeded.seed_busy(session_id.clone(), false), - Some("retry") => seeded.seed_busy(session_id.clone(), true), + Some("retry") => { + seeded.seed_busy(session_id.clone(), true); + retrying_now.insert(session_id.clone()); + } _ => { return Err(( DiagnosticReason::UnknownStatus, @@ -757,6 +873,11 @@ fn seed_from_server( } } } + // This map is the authoritative statement of who is retrying, so a retry fault carried + // across the swap whose session is not in it has ended: its exit arm passed unobserved while + // the stream was down, and leaving it standing would wedge an automatic recovery that is + // already over. Each is retired through its own exact key. + seeded.retire_retries(&retrying_now); // Both pending-ask listings must succeed for the seed to count: a transient failure must not // restore evidence on an unblocked picture and silently wedge an ask opened during the outage. for (endpoint, kind, unavailable, malformed, source) in [ @@ -787,6 +908,10 @@ fn seed_from_server( seeded.seed_ask(id.to_string(), kind); } } + // Every read above answered 200 against this seat's own server, which is what makes the + // conversation link's capability PROBED rather than declared, and its verification bound + // finite. The identity itself comes only from OpenCode's typed `sessionID`. + seeded.verified_through_ms = message::now_ms(); *machine = seeded; Ok(()) } @@ -811,9 +936,212 @@ fn seed_with_diagnostics( // ---- event projection ------------------------------------------------------------------------ +/// The floor at which a millisecond number is a unix INSTANT rather than a duration: `1e12` ms is +/// September 2001, and no retry delay is 30 years. Used to read `session.status{retry}.next` +/// without guessing which of the two the unverified spelling means. +const EPOCH_MS_FLOOR: u64 = 1_000_000_000_000; + +/// How many condition edges are held while the producer has no evidence to write them against. +/// See [`EventMachine::push_edge`] for why trimming the oldest surplus is lossless. +const CONDITION_EDGE_QUEUE: usize = 64; + +/// One condition-axis edge this reducer decided, replayed onto the record by the wrapper loop. +/// Edges rather than a level restatement because the shared writer carries the rest of the tuple +/// across a condition mutation verbatim: a producer that learned its provider is throttled has +/// learned nothing new about whether the model is working, and making it restate an activity it +/// did not observe is how a stale activity gets refreshed by a fault. +#[derive(Debug, Clone, PartialEq, Eq)] +enum ConditionEdge { + Raise(FaultReport), + ClearPaired(FaultKey), + ClearAll(ProgressProof), +} + +/// The pairing identity a clear must name EXACTLY: category plus the full code, never a prefix. +type ConditionKey = (&'static str, Option); + +fn condition_key(fault: &FaultReport) -> ConditionKey { + (fault.category.as_str(), fault.code.clone()) +} + +/// The single slot's total order, so a record carrying one condition while several faults stand +/// never flaps between two equally-ranked ones: (1) a recovery a person owns outranks one the +/// harness owns, (2) category severity, (3) the earlier observation, (4) the code. Total and +/// antisymmetric — the last component is unique inside a set keyed by `(category, code)`. +fn fault_rank(fault: &FaultReport) -> (u8, u8, u64, &str) { + let recovery = match fault.recovery { + Recovery::Human | Recovery::Terminal | Recovery::Unknown => 0, + Recovery::Automatic => 1, + }; + let category = match fault.category { + FaultCategory::Authentication => 0, + FaultCategory::Account => 1, + FaultCategory::Quota => 2, + FaultCategory::RateLimit => 3, + FaultCategory::Policy => 4, + FaultCategory::Context => 5, + FaultCategory::Configuration => 6, + FaultCategory::Provider => 7, + FaultCategory::Harness => 8, + }; + ( + recovery, + category, + fault.observed_at_ms, + fault.code.as_deref().unwrap_or(""), + ) +} + +/// Whether two reports are the SAME fault, ignoring when it was observed — everything a consumer +/// would route, time, or read differently counts, and `observedAtMs` is precisely the field a +/// restatement must not move. +fn same_fault(left: &FaultReport, right: &FaultReport) -> bool { + left.category == right.category + && left.code == right.code + && left.recovery == right.recovery + && left.next_observation_due_ms == right.next_observation_due_ms + && left.detail == right.detail +} + +/// A fault code is `provider/code` and diagnostic underneath a routable category, so a name st2 +/// could not classify rides it verbatim except for the characters that would break that grammar. +fn code_word(name: &str) -> String { + let word: String = name + .chars() + .map(|character| { + if character.is_ascii_alphanumeric() || matches!(character, '_' | '-' | '.') { + character + } else { + '_' + } + }) + .collect(); + if word.is_empty() { + "unknown".to_string() + } else { + word + } +} + +/// `session.error` by its typed `error.name` — the eight-arm union measured on 1.18.19, matched +/// EXHAUSTIVELY so "maps to nothing" and "unknown to me" stay different answers rather than one +/// `unwrap_or` default. +/// +/// `MessageAbortedError` is the one deliberate null: an interruption is not a fault and the closed +/// category set has no word for one, so it never raises and never clears. An unrecognized name is +/// NOT a null — it is a failure the harness itself reported and st2 could not classify, so it +/// stays visible under the most conservative truthful category (`harness`, whose plumbing +/// produced an unreadable verdict) with its recovery left unclaimed, carrying the name it could +/// not classify as diagnostic granularity. The name also rides `reason`; nothing branches on it. +fn session_error_fault(name: &str, payload: &Value, now_ms: u64) -> Option { + Some(match name { + "ProviderAuthError" => { + FaultReport::new(FaultCategory::Authentication, Recovery::Human, now_ms) + .with_code("opencode/ProviderAuthError") + } + "ContextOverflowError" => { + FaultReport::new(FaultCategory::Context, Recovery::Human, now_ms) + .with_code("opencode/ContextOverflowError") + } + "MessageOutputLengthError" => { + FaultReport::new(FaultCategory::Context, Recovery::Human, now_ms) + .with_code("opencode/MessageOutputLengthError") + } + "ContentFilterError" => FaultReport::new(FaultCategory::Policy, Recovery::Human, now_ms) + .with_code("opencode/ContentFilterError"), + "StructuredOutputError" => { + FaultReport::new(FaultCategory::Harness, Recovery::Human, now_ms) + .with_code("opencode/StructuredOutputError") + } + "MessageAbortedError" => return None, + "APIError" => api_error_fault(payload, now_ms), + unclassified => FaultReport::new(FaultCategory::Harness, Recovery::Unknown, now_ms) + .with_code(format!("opencode/session_error.{}", code_word(unclassified))), + }) +} + +/// `APIError` by the HTTP status the provider actually returned. The split the closed vocabulary +/// forces: 402 is an allowance/billing wall (`account`), 429 is a throttle with the allowance +/// intact (`rateLimit`), and 5xx is the provider failing on its own side (`provider`) — there is +/// no `network` word, and SSE transport loss is not a fault at all. +/// +/// `data.isRetryable`, when the harness states it, overrides the recovery class in both +/// directions: it is OpenCode's own verdict on whether it will keep trying. +fn api_error_fault(payload: &Value, now_ms: u64) -> FaultReport { + let (category, recovery, code) = match payload + .pointer("/error/data/status") + .and_then(Value::as_u64) + { + Some(status @ (401 | 403)) => ( + FaultCategory::Authentication, + Recovery::Human, + format!("opencode/APIError.{status}"), + ), + Some(402) => ( + FaultCategory::Account, + Recovery::Human, + "opencode/APIError.402".to_string(), + ), + Some(429) => ( + FaultCategory::RateLimit, + Recovery::Automatic, + "opencode/APIError.429".to_string(), + ), + Some(status) if status >= 500 => ( + FaultCategory::Provider, + Recovery::Automatic, + format!("opencode/APIError.{status}"), + ), + Some(status) if status >= 400 => ( + FaultCategory::Configuration, + Recovery::Human, + format!("opencode/APIError.{status}"), + ), + // A status this version cannot read still happened on the provider's own API: `provider` + // is the truthful floor for it and the recovery stays unclaimed rather than guessed. + _ => ( + FaultCategory::Provider, + Recovery::Unknown, + "opencode/APIError".to_string(), + ), + }; + let recovery = match payload + .pointer("/error/data/isRetryable") + .and_then(Value::as_bool) + { + Some(true) => Recovery::Automatic, + Some(false) => Recovery::Human, + None => recovery, + }; + FaultReport::new(category, recovery, now_ms).with_code(code) +} + +/// The ONE positive clearAll edge: a completed assistant turn that carried no error. The `summary` +/// exclusion reuses the measured trap the numeric axis already encodes — `summary` is a boolean on +/// assistant messages and an object on user ones, and the compaction summarizer's own message is +/// an assistant message. `session.idle`, `session.status{idle}`, a delivery read-back 200 (which +/// proves persistence, not turn success) and a level reseed are deliberately NOT this edge. +fn turn_completed(payload: &Value) -> bool { + let info = payload.get("info").unwrap_or(&Value::Null); + info.get("role").and_then(Value::as_str) == Some("assistant") + && info.get("summary").and_then(Value::as_bool) != Some(true) + && info + .pointer("/time/completed") + .is_some_and(|completed| !completed.is_null()) + && info.get("error").is_none_or(Value::is_null) +} + /// The pure projection from OpenCode's event stream to one seat-level observation. A dedicated /// seat aggregates across the server's sessions: any busy session is activity, any open /// permission or question is a human block, and idle is only derived from positive level evidence. +/// +/// The version 3 fault axis is folded in here and nowhere else, shaped by what OpenCode actually +/// emits: a `session.error` arrives BEFORE `session.status{idle}` and a second one can arrive +/// after it. So no activity edge may clear a condition, `session.idle` is not a success edge, and +/// the standing faults are independent of the activity map — they survive an idle, an abort, a +/// poisoned status word, and an SSE reconnect. The record carries one condition while several +/// faults can stand, so [`EventMachine::winner`] projects the set through [`fault_rank`] and +/// [`EventMachine::edges`] carries only the transitions of that one slot. #[derive(Default)] struct EventMachine { /// sessionID → currently retrying (vs plainly busy). @@ -822,16 +1150,76 @@ struct EventMachine { blocked: BTreeMap, /// Level evidence seen: idle is a proof, never a default. seen_level: bool, - /// A tracked-busy session moved to a status word this version cannot read: every projection - /// is withheld until a fresh level seed replaces this machine. + /// A tracked-busy session moved to a status word this version cannot read: every ACTIVITY + /// projection is withheld until a fresh level seed replaces this machine. A standing fault + /// still reads: the unknown word made the busy map untrustworthy, not the fault set. poisoned: bool, - /// Terminal reason, once observed. + /// Terminal reason, once observed. LEGACY (schema 2) ONLY: the version 2 wire has no + /// condition axis, so a rejected provider credential can only be spelled as this record's + /// last word there, and it stays byte-for-byte what the shipped adapter writes. The version 3 + /// projection has no `ended` arm at all — see [`EventMachine::frame`]. ended: Option<&'static str>, /// The most recent non-terminal session error, surfaced as the idle reason once. last_error: Option, + /// The standing faults, keyed exactly as a paired clear must name them. + conditions: BTreeMap, + /// sessionID → the retry fault raised for it. `session.status{retry}` does not repeat + /// `action.reason` on its exit arm, so the exact code a clear must name is remembered here. + retrying: BTreeMap, + /// The fault on the record's single slot, as this reducer last stated it. + published: Option, + /// Condition-axis transitions awaiting the writer, in the order they were observed. + edges: Vec, + /// The provider's own conversation identity, from OpenCode's typed `sessionID`. + conversation: Option, + /// When the wrapper last had that session named to it by the server it is talking to. + verified_through_ms: u64, } impl EventMachine { + /// The machine a reconnect or a fresh level seed starts from. The ACTIVITY axis is rebuilt + /// from the level surface — that is the whole point of the seed — while the standing faults + /// are carried across: OpenCode's level surface says nothing about them, so dropping them on + /// a stream blip would silence a live auth wall. + /// + /// The queued EDGES ride across too. A queued edge is an observation that already happened on + /// a frame the stream did deliver; a reconnect is not evidence against it, and the producer + /// may well have been unable to write it yet (a lost stream writes nothing). The carried slot + /// is then re-stated, because a raise of the same fault preserves its original `observedAtMs` + /// and the record is the only other place that slot lives. + fn reconnected(&self) -> Self { + let mut fresh = Self { + conditions: self.conditions.clone(), + retrying: self.retrying.clone(), + published: self.published.clone(), + conversation: self.conversation.clone(), + verified_through_ms: self.verified_through_ms, + edges: self.edges.clone(), + ..Self::default() + }; + if let Some(fault) = fresh.published.clone() { + fresh.push_edge(ConditionEdge::Raise(fault)); + } + fresh + } + + /// Retire the retry faults the AUTHORITATIVE level surface no longer reports as retrying. + /// A retry carried across a reseed whose session is absent from the status map (idle sessions + /// are omitted) or reports any other word has ended: its own end event passed while the + /// stream was down, and the level map at seed time is the whole truth about who is retrying. + /// Each one is retired through its exact `(category, code)` key, like any other paired clear. + fn retire_retries(&mut self, retrying_now: &BTreeSet) { + let settled: Vec = self + .retrying + .keys() + .filter(|session| !retrying_now.contains(*session)) + .cloned() + .collect(); + for session in settled { + self.retry_ended(&session); + } + } + fn seed_idle(&mut self) { self.seen_level = true; } @@ -848,38 +1236,59 @@ impl EventMachine { } fn apply(&mut self, event: &Value) { + self.apply_at(event, message::now_ms()); + } + + /// The reducer, with the observation instant passed in: `observedAtMs` is the SEMANTIC clock + /// of a fault, so the one place it is minted is the edge that observed the fault. + fn apply_at(&mut self, event: &Value, now_ms: u64) { let Some(kind) = event.get("type").and_then(Value::as_str) else { return; }; - let properties = event.get("properties").unwrap_or(&Value::Null); + // The `.v2.` spellings move the payload from `properties` to `data` — OpenCode's own SDK + // shim performs that downgrade, which is exactly why the `/doc` marker gate cannot see + // the switch. Read whichever root this frame actually carries. + let payload = [event.get("properties"), event.get("data")] + .into_iter() + .flatten() + .find(|value| value.is_object()) + .unwrap_or(&Value::Null); let session_id = || { - properties + payload .get("sessionID") .and_then(Value::as_str) .map(str::to_string) }; + if kind.starts_with("session.") + && let Some(id) = payload.get("sessionID").and_then(Value::as_str) + { + self.saw_conversation(id, now_ms); + } match kind { "session.status" => { let Some(session_id) = session_id() else { return; }; - match properties + match payload .pointer("/status/type") .and_then(Value::as_str) .unwrap_or("") { "busy" => { self.seen_level = true; - self.busy.insert(session_id, false); + self.busy.insert(session_id.clone(), false); self.last_error = None; + self.retry_ended(&session_id); } "retry" => { self.seen_level = true; - self.busy.insert(session_id, true); + self.busy.insert(session_id.clone(), true); + self.retry_began(&session_id, payload, now_ms); } "idle" => { self.seen_level = true; self.busy.remove(&session_id); + self.retry_ended(&session_id); } // A future status arm is not evidence of anything — not even level evidence: // counting it would let an unrecognized word prove `idle` on a quiet server. @@ -897,14 +1306,22 @@ impl EventMachine { if let Some(session_id) = session_id() { self.seen_level = true; self.busy.remove(&session_id); + // The retry signal's other end event: this session has settled, so the retry + // it was in is over. Still the SAME session key and still the exact code — + // no other fault is touched, and no other session's retry is. + self.retry_ended(&session_id); } } "session.error" => { - let name = properties + let name = payload .pointer("/error/name") .and_then(Value::as_str) .unwrap_or("unknown"); if name == "ProviderAuthError" { + // The LEGACY word, unchanged (see the field's own note). The version 3 + // projection reads the authentication fault raised below instead and leaves + // the seat live: OpenCode keeps serving, and only the process-exit owner + // writes a terminal record. self.ended = Some("providerAuth"); } else { if let Some(session_id) = session_id() { @@ -913,32 +1330,249 @@ impl EventMachine { self.seen_level = true; self.last_error = Some(format!("error:{name}")); } + if let Some(fault) = session_error_fault(name, payload, now_ms) { + self.raise(fault); + } } - "permission.asked" => { - if let Some(id) = ask_id(properties) { + "permission.asked" | "permission.v2.asked" => { + if let Some(id) = ask_id(payload) { self.blocked.insert(id, "permission"); } } - "permission.replied" => { - if let Some(id) = ask_id(properties) { + "permission.replied" | "permission.v2.replied" => { + if let Some(id) = ask_id(payload) { self.blocked.remove(&id); } } - "question.asked" => { - if let Some(id) = ask_id(properties) { + "question.asked" | "question.v2.asked" => { + if let Some(id) = ask_id(payload) { self.blocked.insert(id, "question"); } } - "question.replied" | "question.rejected" => { - if let Some(id) = ask_id(properties) { + "question.replied" + | "question.rejected" + | "question.v2.replied" + | "question.v2.rejected" => { + if let Some(id) = ask_id(payload) { self.blocked.remove(&id); } } - // server.connected, server.heartbeat, plugin.added replay, message.*, … — not state. + // The turn's own success edge, and the only thing that clears the whole axis. + "message.updated" => { + if turn_completed(payload) { + self.clear_everything(ProgressProof::TurnCompleted); + } + } + // The server threw its own instance away. NOT a terminal record: the child may still + // be reaped normally afterwards, and its exit is the process-exit owner's word. + "global.disposed" | "server.instance.disposed" => { + self.raise( + FaultReport::new(FaultCategory::Harness, Recovery::Human, now_ms) + .with_code("opencode/disposed"), + ); + } + // server.connected, server.heartbeat, plugin.added replay, session.compacted (normal + // compaction, numeric axis only), … — not state. _ => {} } } + /// `session.status{retry}` is the only deadline OpenCode declares for itself (DQ-H8): the + /// harness says when it will next act, and the record carries THAT instant rather than a + /// number st2 invented. An automatic recovery with no declared deadline carries none. + fn retry_began(&mut self, session_id: &str, payload: &Value, now_ms: u64) { + let (category, word) = match payload + .pointer("/status/action/reason") + .and_then(Value::as_str) + { + Some("provider_auth") => (FaultCategory::Authentication, "provider_auth"), + Some("rate_limit") => (FaultCategory::RateLimit, "rate_limit"), + Some("provider_error") => (FaultCategory::Provider, "provider_error"), + // A retry whose cause the harness did not name: `provider` is the truthful floor — + // something on the provider side made OpenCode retry — and no narrower claim is + // evidenced. The code says `unknown` so the two cases stay distinguishable. + _ => (FaultCategory::Provider, "unknown"), + }; + let mut fault = FaultReport::new(category, Recovery::Automatic, now_ms) + .with_code(format!("opencode/retry.{word}")); + if let Some(next) = payload + .pointer("/status/next") + .and_then(Value::as_u64) + .filter(|next| *next > 0) + { + // `next` is measured as the harness's delay in milliseconds, but the spelling is + // unverified at the pinned build, so it is disambiguated by MAGNITUDE rather than by + // guessing: anything at or above the epoch floor is a unix-millisecond instant (no + // retry waits 30+ years), everything below it is a delay from the observation. An + // absolute instant already in the past is due NOW — the deadline cannot precede the + // observation that carries it, and a past deadline is exactly what "overdue" means. + let due = if next >= EPOCH_MS_FLOOR { + next.max(now_ms) + } else { + now_ms.saturating_add(next) + }; + fault = fault.with_observation_due(due); + } + if let Some(attempt) = payload.pointer("/status/attempt").and_then(Value::as_u64) { + fault = fault.with_detail(format!("retry attempt {attempt}")); + } + self.retrying.insert(session_id.to_string(), fault.key()); + self.raise(fault); + } + + /// The retry signal's OWN end event: the same session reporting a status word that is not + /// `retry`. This is the same event family and the same session key as the raise, not a + /// foreign activity edge — and it clears its exact code, because a `429` rateLimit fault and + /// a `retry.rate_limit` fault share a category and differ only by code. + /// + /// The record is SEAT-level, though, and several sessions of one server can be retrying for + /// the same reason under one code. So the condition holds until the LAST of them settles: + /// clearing on the first exit would silence a retry still in flight on a sibling session, + /// while a per-session code would fragment a seat-level axis into unbounded cardinality. + fn retry_ended(&mut self, session_id: &str) { + let Some(key) = self.retrying.remove(session_id) else { + return; + }; + if self.retrying.values().any(|standing| *standing == key) { + return; + } + self.clear_paired(key); + } + + /// Raise a fault, or restate one that already stands. The semantic clock is minted once per + /// key: a restatement — a further retry attempt included — never postpones the instant the + /// condition was first observed (OHS-R20). + fn raise(&mut self, mut fault: FaultReport) { + let key = condition_key(&fault); + if let Some(standing) = self.conditions.get(&key) { + fault.observed_at_ms = standing.observed_at_ms; + // The carried-back instant can predate a deadline computed from a later clock, and a + // deadline may never precede the observation it follows. + if let Some(due) = fault.next_observation_due_ms { + fault.next_observation_due_ms = Some(due.max(fault.observed_at_ms)); + } + if same_fault(standing, &fault) { + return; + } + } + self.conditions.insert(key, fault); + self.settle(); + } + + /// Clear the fault this key names and ONLY that one. A key naming a fault that does not stand + /// clears nothing, and clearing a fault a higher-ranked one was masking moves no slot. + fn clear_paired(&mut self, key: FaultKey) { + if self + .conditions + .remove(&(key.category.as_str(), key.code.clone())) + .is_none() + { + return; + } + if self + .published + .as_ref() + .is_some_and(|standing| standing.key() == key) + { + self.published = None; + self.push_edge(ConditionEdge::ClearPaired(key)); + } + self.settle(); + } + + /// The only blanket clear, and only behind a positive typed proof of progress the standing + /// fault would have prevented. It is stated even over an empty set: a completed turn is a + /// positive health observation, and `clear` is a different answer from `absent`. + fn clear_everything(&mut self, proof: ProgressProof) { + self.conditions.clear(); + self.retrying.clear(); + self.published = None; + self.push_edge(ConditionEdge::ClearAll(proof)); + } + + /// Project the fault set onto the record's single slot and queue the transition, if it moved. + fn settle(&mut self) { + let winner = self.winner().cloned(); + if winner != self.published { + if let Some(fault) = winner.clone() { + self.push_edge(ConditionEdge::Raise(fault)); + } + self.published = winner; + } + } + + fn winner(&self) -> Option<&FaultReport> { + self.conditions + .values() + .min_by(|left, right| fault_rank(left).cmp(&fault_rank(right))) + } + + fn take_edges(&mut self) -> Vec { + std::mem::take(&mut self.edges) + } + + /// Queue one condition edge. The queue is HELD, not dropped, while the producer has no + /// evidence to hang a write on: the fault was observed on a frame that did arrive. It is + /// bounded because the record shows exactly one slot and the newest edge always states the + /// slot the reducer projects now, so trimming the oldest surplus loses only intermediate + /// history no reader could ever have observed. + fn push_edge(&mut self, edge: ConditionEdge) { + if self.edges.last() == Some(&edge) { + return; + } + self.edges.push(edge); + if self.edges.len() > CONDITION_EDGE_QUEUE { + self.edges.drain(..self.edges.len() - CONDITION_EDGE_QUEUE); + } + } + + /// The axis a version 3 record must have STATED once before an activity-only frame can ride + /// on top of it. A claim fence has observed nothing, so it carries no condition to inherit + /// and `Unchanged` over it is refused as [`harness_state::Refusal::Unstated`]; this is what + /// the first frame of a session says instead. `Clear` over an empty fault set is a positive + /// health claim the producer has earned: nothing is published until the level seed succeeds. + fn bootstrap_condition(&self) -> ConditionReport { + match self.winner() { + Some(fault) => ConditionReport::Fault(fault.clone()), + None => ConditionReport::Clear, + } + } + + /// The seat's conversation identity, from OpenCode's own typed `sessionID`. A newer id + /// replaces the old one, matching the delivery pump's last-writer-wins target (DQ-C10), and + /// the verification bound only ever moves forward. + fn saw_conversation(&mut self, session_id: &str, at_ms: u64) { + if self.conversation.as_deref() != Some(session_id) { + self.conversation = Some(session_id.to_string()); + } + self.verified_through_ms = self.verified_through_ms.max(at_ms); + } + + fn conversation_state(&self) -> ConversationState { + match &self.conversation { + Some(conversation) if self.verified_through_ms > 0 => { + ConversationState::Linked(ConversationClaim { + driver: "opencode".to_string(), + conversation: conversation.clone(), + // Positively evidenced, not declared: this adapter handles `session.compacted` + // and filters the summarizer's own assistant message, i.e. OpenCode + // demonstrably rewrites history underneath a reader. + history_mutability: HistoryMutability::Rewritable, + // The wrapper exercises this session over HTTP — the level seed's own 200s and + // the frames the server names it on — so the capability is probed, never + // declared from pinned knowledge. + capability_evidence: CapabilityEvidence::Probed, + verified_through_ms: self.verified_through_ms, + }) + } + // Never `Unsupported`: OpenCode HAS conversation identity and the seat's TUI has + // simply not created a session yet. Denying the capability would be a false claim. + _ => ConversationState::Unavailable(Some("no-session".to_string())), + } + } + + /// The LEGACY (schema 2) projection, unchanged: this is what the shipped adapter writes while + /// the emitted version has no condition axis to carry a fault on. fn observation(&self) -> Option { // A sticky terminal outranks poison: `ended` does not depend on the busy map the // unknown word made untrustworthy, and withholding it would lose the terminal to the @@ -982,6 +1616,55 @@ impl EventMachine { } None } + + /// The version 3 resolved tuple, with NO terminal arm: only the process-exit owner writes + /// `ended`, so a rejected credential reads here as a live seat carrying an authentication + /// fault. The condition rides `Unchanged` on every frame — an activity edge is not evidence + /// about a fault in either direction, and OpenCode proves it by emitting the error before the + /// idle — so the queued condition edges are the only writes that move that axis. + /// + /// `HumanAsk::None` over an empty ask map is a positive claim this producer can make: the + /// level seed that gates every publish reads `/permission` and `/question`, so absence is + /// listed rather than assumed. + fn frame(&self) -> Option { + self.frame_with(ConditionReport::Unchanged) + } + + /// [`EventMachine::frame`] with the condition axis stated explicitly, for the one write that + /// has to state it: the session's first frame, over a claim fence that carries no axis to + /// leave unchanged. + fn frame_with(&self, condition: ConditionReport) -> Option { + if self.poisoned { + return None; + } + let frame = |state: Activity, ask: HumanAsk| { + Frame::new(state, InputBuffer::Unknown, condition.clone(), ask) + .with_conversation(self.conversation_state()) + }; + if let Some(kind) = self.blocked.values().next() { + let ask = match *kind { + "question" => AskKind::Question, + _ => AskKind::Permission, + }; + return Some(frame(Activity::Active, HumanAsk::Pending(ask)).with_reason(*kind)); + } + if !self.busy.is_empty() { + let active = frame(Activity::Active, HumanAsk::None); + return Some(if self.busy.values().all(|retry| *retry) { + active.with_reason("retry") + } else { + active + }); + } + if self.seen_level { + let idle = frame(Activity::Idle, HumanAsk::None); + return Some(match &self.last_error { + Some(reason) => idle.with_reason(reason.clone()), + None => idle, + }); + } + None + } } /// The most recently updated session id from `GET /session`, or the last listed when the entries @@ -1304,6 +1987,9 @@ fn cumulative_tokens(tokens: &Value) -> Option { // ---- native delivery ------------------------------------------------------------------------- +/// What `GET /session/{session}/message/{message}` proved. A storage receipt, graded to +/// [`delivery_ledger::Phase::Persisted`]: it says the server holds the exact client message, never +/// that anything read it. #[derive(Clone, Copy)] enum ReadBack { Durable, @@ -1311,28 +1997,6 @@ enum ReadBack { Indeterminate, } -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "camelCase")] -enum DeliveryPhase { - Attempted, - Accepted, -} - -/// One durable FIFO delivery attempt, written before transport (the Codex discipline). The stable -/// `messageID` makes a replayed attempt reconcilable: the server either shows the message durably -/// (accepted) or does not (retry the same identity, never a second one). -#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "camelCase", deny_unknown_fields)] -struct DeliveryState { - schema: String, - agent: String, - runtime_id: String, - session_id: String, - filename: String, - message_id: String, - phase: DeliveryPhase, -} - struct Delivery { catalog_root: PathBuf, inbox: PathBuf, @@ -1340,8 +2004,7 @@ struct Delivery { this_host: String, identity: String, runtime_id: String, - state_path: PathBuf, - state: Option, + ledger: delivery_ledger::Ledger, /// The session a new delivery binds to: the most recently observed one. target_session: Option, next_attempt: Instant, @@ -1355,35 +2018,37 @@ impl Delivery { identity: &str, runtime_id: &str, ) -> Self { - let state_path = state_dir(catalog_root, identity).join("delivery-state.json"); + let legacy_path = state_dir(catalog_root, identity).join(delivery_ledger::LEGACY_FILE); Self::with_state_path( catalog_root, agent_dir, this_host, identity, runtime_id, - state_path, + legacy_path, ) } + /// `legacy_path` is the v1 `delivery-state.json`: the one-shot adoption source and the + /// rollback floor. The ledger itself lives beside it under [`delivery_ledger::LEDGER_FILE`]. fn with_state_path( catalog_root: &Path, agent_dir: &Path, this_host: &str, identity: &str, runtime_id: &str, - state_path: PathBuf, + legacy_path: PathBuf, ) -> Self { - let state = std::fs::read(&state_path) - .ok() - .and_then(|bytes| serde_json::from_slice::(&bytes).ok()) - .filter(|state| { - state.schema == DELIVERY_STATE_SCHEMA - && state.agent == identity - && message::is_message_filename(&state.filename) - && state.message_id - == stable_message_id(identity, &state.session_id, &state.filename) - }); + let owner = identity.to_string(); + // The same derivation the transport uses, so a record whose messageID does not match its + // own session and filename is provably not this agent's and fails closed. + let ledger = delivery_ledger::Ledger::open( + &legacy_path, + delivery_ledger::Harness::OpenCode.profile(), + identity, + runtime_id, + |session, filename| stable_message_id(&owner, session, filename), + ); Self { catalog_root: catalog_root.to_path_buf(), inbox: message::inbox_dir(agent_dir), @@ -1391,8 +2056,7 @@ impl Delivery { this_host: this_host.to_string(), identity: identity.to_string(), runtime_id: runtime_id.to_string(), - state_path, - state, + ledger, target_session: None, next_attempt: Instant::now(), } @@ -1427,13 +2091,10 @@ impl Delivery { mut diagnostics: Option<&mut DiagnosticPublisher>, ) -> Result<()> { let unread = message::list_inbox(&self.inbox)?; - if let Some(state) = self.state.as_ref() - && unread - .iter() - .all(|message| message.filename != state.filename) - { - self.clear_state()?; - } + // Archive is the recipient agent's act and the only settlement authority. An entry whose + // file left the inbox releases ownership here; this pump never moves a file itself. + self.ledger + .prune(|filename| unread.iter().any(|entry| entry.filename == filename))?; if status::read_state(&self.status_path) == status::State::Dnd { return Ok(()); } @@ -1458,65 +2119,93 @@ impl Delivery { recovered } }; - if let Some(state) = self.state.as_ref() - && state.session_id != target - { - // A newly selected session is a different delivery binding (the Codex thread rule). - self.clear_state()?; + // A newly selected session is a different delivery binding (the Codex thread rule): the + // old binding's receipt may neither suppress nor acknowledge delivery to this one. + if self.ledger.binding().is_some_and(|binding| binding != target) { + self.ledger.rebind(&target)?; } - - if let Some(state) = self.state.clone() { - if state.filename != head.filename { - return Ok(()); // The bound message is behind the head; archive precedence resolves it. + // Fail closed. An unreadable ledger holds and surfaces instead of guessing, and it never + // refused to start: a driver that will not start delivers nothing at all. The + // operator-visible surface is the existing typed boundary — the transport is unavailable — + // and the raw reason stays in tracing, so no unbounded prose reaches the record. + if let Some(reason) = self.ledger.quarantined() { + tracing::warn!("st2 opencode-session: delivery ledger is quarantined: {reason}"); + if let Some(diagnostics) = diagnostics.as_deref_mut() { + diagnostics.publish( + DiagnosticStage::Delivery, + DiagnosticReason::DeliveryUnavailable, + DiagnosticSource::PromptTransport, + ); } - match state.phase { - DeliveryPhase::Accepted => { - if let Some(diagnostics) = diagnostics.as_deref_mut() { - diagnostics.clear(DiagnosticStage::Delivery); - diagnostics.clear(DiagnosticStage::ReadBack); - } - return Ok(()); - } - DeliveryPhase::Attempted => { - return self.reconcile_or_retry(client, state, diagnostics); + return Ok(()); + } + // Re-asserted on every pass while an entry is outstanding: a crash exactly at the floor + // write would otherwise leave a landing with no v1-readable lower bound, and a rolled-back + // binary would take its state=None path and POST the same messageID again. + self.ledger.reassert_floor()?; + if let Some(entry) = self.ledger.entry(&head.filename).cloned() { + // Storage is a receipt, not consumption: it stops the POST loop and holds the inbox + // entry, and only prompt admission would release ownership. + if entry.phase >= delivery_ledger::Phase::Persisted { + if let Some(diagnostics) = diagnostics.as_deref_mut() { + diagnostics.clear(DiagnosticStage::Delivery); + diagnostics.clear(DiagnosticStage::ReadBack); } + return Ok(()); } + return self.reconcile_or_retry(client, entry, diagnostics); + } + if !self.ledger.entries().is_empty() { + return Ok(()); // The bound message is behind the head; archive precedence resolves it. } let message_id = stable_message_id(&self.identity, &target, &head.filename); - let state = DeliveryState { - schema: DELIVERY_STATE_SCHEMA.to_string(), - agent: self.identity.clone(), - runtime_id: self.runtime_id.clone(), - session_id: target, + let entry = self.ledger.begin(delivery_ledger::Begin { filename: head.filename.clone(), - message_id, - phase: DeliveryPhase::Attempted, - }; - self.write_state(state.clone())?; + binding: target.clone(), + correlation: delivery_ledger::Correlation::native(message_id.clone()), + // OpenCode's v1 record carried no incarnation and its read-back is a durable query, + // not a live frame, so a pre-crash attempt is reconcilable without one. + incarnation: None, + legacy_floor: delivery_ledger::opencode_floor( + &self.identity, + &self.runtime_id, + &target, + &head.filename, + &message_id, + ), + })?; let text = ding::poke_text(&self.catalog_root, &self.this_host, &self.identity, &head); - self.send(client, &state, &text, diagnostics) + self.send(client, &entry, &text, diagnostics) } fn reconcile_or_retry( &mut self, client: &Client, - state: DeliveryState, + entry: delivery_ledger::Entry, mut diagnostics: Option<&mut DiagnosticPublisher>, ) -> Result<()> { - let read_back = self.read_back(client, &state); + let read_back = self.read_back(client, &entry); report_read_back(read_back, &mut diagnostics); match read_back { ReadBack::Durable => { - let mut accepted = state; - accepted.phase = DeliveryPhase::Accepted; - return self.write_state(accepted); + self.ledger + .record(&entry.filename, delivery_ledger::Evidence::Persisted)?; + return Ok(()); } // Measured on 1.18.19: a second POST with the same messageID appends its parts again // into the same message, so an indeterminate read-back must never trigger a resend — // the read-back itself is retried on a later pass. ReadBack::Indeterminate => return Ok(()), - ReadBack::Absent => {} + // A 404 for the exact client message is an authoritative absence: the only receipt + // that may authorize another POST of the same identity. + ReadBack::Absent => { + self.ledger + .negative(&entry.filename, delivery_ledger::NegativeReceipt::Absent)?; + } + } + if self.ledger.retry(&entry.filename) != delivery_ledger::RetryDecision::Retry { + return Ok(()); } if Instant::now() < self.next_attempt { return Ok(()); @@ -1524,27 +2213,27 @@ impl Delivery { let unread = message::list_inbox(&self.inbox)?; let Some(head) = unread .into_iter() - .find(|message| message.filename == state.filename) + .find(|message| message.filename == entry.filename) else { return Ok(()); }; let text = ding::poke_text(&self.catalog_root, &self.this_host, &self.identity, &head); - self.send(client, &state, &text, diagnostics) + self.send(client, &entry, &text, diagnostics) } fn send( &mut self, client: &Client, - state: &DeliveryState, + entry: &delivery_ledger::Entry, text: &str, mut diagnostics: Option<&mut DiagnosticPublisher>, ) -> Result<()> { self.next_attempt = Instant::now() + DELIVERY_RETRY; let payload = json!({ - "messageID": state.message_id, + "messageID": entry.correlation.value, "parts": [{ "type": "text", "text": text }], }); - let path = format!("/session/{}/prompt_async", state.session_id); + let path = format!("/session/{}/prompt_async", entry.binding); let status = match client.post_json(&path, &payload) { Ok(status) => status, Err(error) => { @@ -1568,22 +2257,28 @@ impl Delivery { } anyhow::bail!("POST {path} returned {status}"); } + // The transport call succeeded. That is a fact about the call, not about the server's + // state, so it grades no higher than `transportAccepted`. + self.ledger + .record(&entry.filename, delivery_ledger::Evidence::TransportAccepted)?; if let Some(diagnostics) = diagnostics.as_deref_mut() { diagnostics.clear(DiagnosticStage::Delivery); } - let read_back = self.read_back(client, state); + let read_back = self.read_back(client, entry); report_read_back(read_back, &mut diagnostics); if matches!(read_back, ReadBack::Durable) { - let mut accepted = state.clone(); - accepted.phase = DeliveryPhase::Accepted; - self.write_state(accepted)?; + self.ledger + .record(&entry.filename, delivery_ledger::Evidence::Persisted)?; } Ok(()) } /// The only receipt this transport accepts: the exact client message read back durably. - fn read_back(&self, client: &Client, state: &DeliveryState) -> ReadBack { - let path = format!("/session/{}/message/{}", state.session_id, state.message_id); + fn read_back(&self, client: &Client, entry: &delivery_ledger::Entry) -> ReadBack { + let path = format!( + "/session/{}/message/{}", + entry.binding, entry.correlation.value + ); match client.status_of_get(&path) { Ok(200) => ReadBack::Durable, Ok(404) => ReadBack::Absent, @@ -1591,22 +2286,6 @@ impl Delivery { Ok(_) | Err(_) => ReadBack::Indeterminate, } } - - fn write_state(&mut self, state: DeliveryState) -> Result<()> { - atomic_json(&self.state_path, &state)?; - self.state = Some(state); - Ok(()) - } - - fn clear_state(&mut self) -> Result<()> { - match std::fs::remove_file(&self.state_path) { - Ok(()) => {} - Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} - Err(error) => return Err(error.into()), - } - self.state = None; - Ok(()) - } } fn report_read_back( @@ -1666,28 +2345,6 @@ fn state_dir(catalog_root: &Path, identity: &str) -> PathBuf { base.join("st2").join("opencode").join(&digest[..24]) } -fn atomic_json(path: &Path, value: &impl Serialize) -> Result<()> { - let parent = path.parent().context("state file has no parent")?; - std::fs::create_dir_all(parent)?; - let temp = parent.join(format!(".{}.tmp", std::process::id())); - // Durability, not just atomicity: a crash between the rename and OpenCode's acceptance of - // the prompt_async would lose the Attempted receipt and make the pump re-POST duplicate - // parts. The bytes reach disk before the rename and the directory entry afterwards, so - // once delivery proceeds the receipt survives. - let mut file = std::fs::File::create(&temp)?; - file.write_all(&serde_json::to_vec(value)?)?; - file.sync_all()?; - drop(file); - if let Err(error) = std::fs::rename(&temp, path) { - let _ = std::fs::remove_file(&temp); - return Err(error.into()); - } - if let Ok(dir) = std::fs::File::open(parent) { - let _ = dir.sync_all(); - } - Ok(()) -} - #[cfg(test)] mod tests { @@ -1734,6 +2391,18 @@ mod tests { machine.observation().expect("an observation") } + /// The version 3 resolved tuple. Every assertion on it is reducer-level on purpose: the + /// emitted version is still 2, so a condition-only change is projected away on the wire and + /// only the pure native-event→typed mapping is observable here. + fn framed(machine: &EventMachine) -> Frame { + machine.frame().expect("a frame") + } + + /// The one fault the record's single slot would carry. + fn standing(machine: &EventMachine) -> FaultReport { + machine.winner().cloned().expect("a standing fault") + } + #[test] fn no_evidence_yields_no_observation_so_nothing_is_written() { let mut machine = EventMachine::default(); @@ -1872,8 +2541,14 @@ mod tests { assert_eq!(observed(&machine).blocked_on, BlockedOn::None); } + /// The LEGACY (schema 2) projection, pinned unchanged. Production now emits version 3, so + /// this is no longer the live shape — which is exactly why it is pinned: a version 1 or 2 + /// record still on disk, and any seat a rollback puts back on version 2, must keep reading + /// the same way, including the terminal word version 2 has to spell a rejected credential + /// with, having no condition axis to put it on. The version 3 projection of the same replay + /// is the test below. #[test] - fn provider_auth_error_is_terminal_and_other_errors_settle_to_idle_with_a_reason() { + fn the_legacy_projection_keeps_its_terminal_auth_word_and_idle_reasons() { let mut machine = EventMachine::default(); machine.apply(&event( r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"busy"}}}"#, @@ -1893,46 +2568,971 @@ mod tests { assert_eq!(ended.reason.as_deref(), Some("providerAuth")); } + /// Spike I10, the one live defect in the shipped adapter: a rejected provider credential is a + /// live authentication fault, not this record's last word. The version 3 projection has no + /// terminal arm at all — the wrapper's own exit path is the sole terminal writer — and the + /// seat stays observable so an operator can see the wall while the TUI keeps answering. #[test] - fn openapi_gate_names_every_missing_marker() { - let complete: Value = serde_json::from_str(&format!( - r#"{{"paths":{{}},"markers":{:?}}}"#, - REQUIRED_API_MARKERS - )) - .unwrap(); - check_openapi_subset(&complete).unwrap(); + fn provider_auth_error_is_a_live_authentication_fault_never_a_terminal() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"busy"}}}"#, + ), + 1_000, + ); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ProviderAuthError"}}}"#, + ), + 2_000, + ); - let error = check_openapi_subset(&serde_json::json!({"paths": {"/session": {}}})) - .unwrap_err() - .to_string(); - assert!(error.contains("prompt_async"), "{error}"); - assert!(error.contains("permission.asked"), "{error}"); + let fault = standing(&machine); + assert_eq!(fault.category, FaultCategory::Authentication); + assert_eq!(fault.code.as_deref(), Some("opencode/ProviderAuthError")); + assert_eq!(fault.recovery, Recovery::Human); + assert_eq!(fault.observed_at_ms, 2_000); + assert_eq!( + fault.next_observation_due_ms, None, + "a wall a person must repair has no automatic deadline" + ); + let live = framed(&machine); + assert_eq!(live.state, Activity::Active, "the seat keeps serving"); + assert_eq!(live.condition, ConditionReport::Unchanged); + assert_eq!(live.exit, None); + assert_eq!(machine.take_edges(), vec![ConditionEdge::Raise(fault)]); + + // OpenCode's own ordering: the idle follows the error. It leaves the wall standing and + // still never produces a terminal. + machine.apply_at( + &event(r#"{"type":"session.idle","properties":{"sessionID":"ses_a"}}"#), + 3_000, + ); + assert_eq!(framed(&machine).state, Activity::Idle); + assert_eq!( + standing(&machine).code.as_deref(), + Some("opencode/ProviderAuthError") + ); + assert_eq!( + machine.take_edges(), + vec![], + "an activity edge never writes the condition axis" + ); } + /// OpenCode's measured ordering: the error arrives BEFORE `session.status{idle}` and a second + /// one can arrive after it. So the idle clears nothing, and restating the same condition does + /// not re-mint the instant it was first observed (OHS-R20). #[test] - fn stable_message_ids_are_grammatical_and_distinct_per_binding() { - let id = stable_message_id("h.worker", "ses_a", "1786380000000-abc123.md"); - assert!(id.starts_with("msg")); + fn an_error_before_idle_and_a_second_error_after_keep_one_standing_fault() { + let overflow = r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ContextOverflowError"}}}"#; + let mut machine = EventMachine::default(); + machine.apply_at(&event(overflow), 1_000); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"idle"}}}"#, + ), + 2_000, + ); + assert_eq!(framed(&machine).state, Activity::Idle); assert_eq!( - id, - stable_message_id("h.worker", "ses_a", "1786380000000-abc123.md") + standing(&machine).code.as_deref(), + Some("opencode/ContextOverflowError"), + "`session.idle` is not a success edge on this harness" + ); + let _ = machine.take_edges(); + + machine.apply_at(&event(overflow), 3_000); + assert_eq!( + standing(&machine).observed_at_ms, 1_000, + "the semantic clock is minted once per condition" + ); + assert_eq!( + machine.take_edges(), + vec![], + "a restatement of the same fault moves no slot" ); - for other in [ - stable_message_id("h.other", "ses_a", "1786380000000-abc123.md"), - stable_message_id("h.worker", "ses_b", "1786380000000-abc123.md"), - stable_message_id("h.worker", "ses_a", "1786380000000-def456.md"), - ] { - assert_ne!(id, other); - } } + /// DQ-H8, answered for OpenCode by OpenCode: `session.status{retry}` declares its own next + /// attempt, so the deadline on the record is the harness's `next` and never a number st2 + /// invented. Its clear is the same signal's own end event, on the same session key. #[test] - fn base64_matches_known_vectors() { - for (input, expected) in [ - (&b""[..], ""), - (b"f", "Zg=="), - (b"fo", "Zm8="), - (b"foo", "Zm9v"), + fn retry_carries_the_harness_declared_deadline_and_clears_its_own_exact_code() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"retry","attempt":2,"next":30000,"action":{"reason":"rate_limit"}}}}"#, + ), + 1_000_000, + ); + let fault = standing(&machine); + assert_eq!(fault.category, FaultCategory::RateLimit); + assert_eq!(fault.code.as_deref(), Some("opencode/retry.rate_limit")); + assert_eq!(fault.recovery, Recovery::Automatic); + assert_eq!( + fault.next_observation_due_ms, + Some(1_030_000), + "the deadline is the harness's own `next`, offset from the observation" + ); + assert_eq!(fault.detail.as_deref(), Some("retry attempt 2")); + assert_eq!(framed(&machine).reason.as_deref(), Some("retry")); + let _ = machine.take_edges(); + + // A retry on ANOTHER session is its own fault, under its own reason's code… + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_b","status":{"type":"retry","action":{"reason":"provider_error"}}}}"#, + ), + 1_000_100, + ); + // …and this session's own end event clears exactly the code it raised. + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"busy"}}}"#, + ), + 1_000_200, + ); + let remaining = standing(&machine); + assert_eq!( + remaining.code.as_deref(), + Some("opencode/retry.provider_error"), + "the other session's retry is untouched" + ); + assert_eq!( + remaining.next_observation_due_ms, None, + "an automatic recovery the harness gave no deadline for carries none" + ); + assert_eq!( + machine.take_edges(), + vec![ + ConditionEdge::ClearPaired( + FaultKey::new(FaultCategory::RateLimit).with_code("opencode/retry.rate_limit") + ), + ConditionEdge::Raise(remaining), + ] + ); + } + + /// CX-1: a `429` rateLimit fault and a `retry.rate_limit` fault share a CATEGORY and differ + /// only by code, so a category-scoped clear would silence the wrong one. + #[test] + fn a_paired_clear_names_the_full_code_not_the_category() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"APIError","data":{"status":429}}}}"#, + ), + 1_000, + ); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"retry","next":5000,"action":{"reason":"rate_limit"}}}}"#, + ), + 2_000, + ); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"idle"}}}"#, + ), + 3_000, + ); + assert_eq!( + standing(&machine).code.as_deref(), + Some("opencode/APIError.429"), + "the sibling code in the same category survives its neighbour's clear" + ); + } + + /// The one positive clearAll edge, and everything that is deliberately not it. This is the + /// test that keeps a failed turn from laundering itself clean. + #[test] + fn only_a_completed_assistant_message_clears_the_whole_axis() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ContentFilterError"}}}"#, + ), + 1_000, + ); + let fault = standing(&machine); + assert_eq!(fault.category, FaultCategory::Policy); + let _ = machine.take_edges(); + + for quiet in [ + r#"{"type":"session.idle","properties":{"sessionID":"ses_a"}}"#, + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"idle"}}}"#, + // The compaction summarizer's own message is an assistant message. + r#"{"type":"message.updated","properties":{"info":{"role":"assistant","summary":true,"time":{"completed":2}}}}"#, + // An unfinished turn. + r#"{"type":"message.updated","properties":{"info":{"role":"assistant","time":{"created":1}}}}"#, + // A finished turn that carried an error is not progress. + r#"{"type":"message.updated","properties":{"info":{"role":"assistant","time":{"completed":2},"error":{"name":"APIError"}}}}"#, + // `summary` is an OBJECT on user messages, and therefore truthy. + r#"{"type":"message.updated","properties":{"info":{"role":"user","summary":{"diffs":[]},"time":{"completed":2}}}}"#, + ] { + machine.apply_at(&event(quiet), 2_000); + assert_eq!(machine.winner(), Some(&fault), "{quiet} is not progress"); + assert_eq!(machine.take_edges(), vec![], "{quiet} writes no condition"); + } + + machine.apply_at( + &event( + r#"{"type":"message.updated","properties":{"info":{"role":"assistant","summary":false,"time":{"created":1,"completed":2}}}}"#, + ), + 3_000, + ); + assert_eq!(machine.winner(), None); + assert_eq!( + machine.take_edges(), + vec![ConditionEdge::ClearAll(ProgressProof::TurnCompleted)], + "a blanket clear names the progress it witnessed" + ); + } + + /// The two deliberate nulls of the mapping: an interruption is not a fault, and ordinary + /// compaction is the numeric axis's business. Neither raises, and neither clears. + #[test] + fn an_abort_and_a_compaction_are_not_faults_and_clear_nothing() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ProviderAuthError"}}}"#, + ), + 1_000, + ); + let wall = standing(&machine); + let _ = machine.take_edges(); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"MessageAbortedError"}}}"#, + ), + 2_000, + ); + machine.apply_at( + &event(r#"{"type":"session.compacted","properties":{"sessionID":"ses_a"}}"#), + 3_000, + ); + assert_eq!(machine.winner(), Some(&wall)); + assert_eq!(machine.take_edges(), vec![]); + assert_eq!( + framed(&machine).reason.as_deref(), + Some("error:MessageAbortedError"), + "the name still rides `reason`, diagnostically" + ); + } + + /// Both spellings of the same ask normalize to one ask, including the `data` payload root + /// OpenCode's own SDK shim downgrades the v2 frames to. The verbatim 1.18.19 v1 fixtures + /// above cover the other half, unchanged. No `review` ask exists on this surface and none is + /// synthesized. + #[test] + fn both_ask_spellings_enter_and_exit_the_same_ask() { + let mut machine = EventMachine::default(); + machine.seed_idle(); + machine.apply_at( + &event(r#"{"type":"permission.v2.asked","data":{"id":"per_1","sessionID":"ses_a"}}"#), + 1_000, + ); + assert_eq!( + framed(&machine).ask, + HumanAsk::Pending(AskKind::Permission) + ); + assert_eq!( + observed(&machine).ask, + Ask::Permission, + "the legacy pair normalizes the same frame identically" + ); + machine.apply_at( + &event( + r#"{"type":"permission.v2.replied","data":{"requestID":"per_1","reply":"once"}}"#, + ), + 2_000, + ); + assert_eq!(framed(&machine).ask, HumanAsk::None); + + machine.apply_at( + &event(r#"{"type":"question.v2.asked","properties":{"id":"que_1","sessionID":"ses_a"}}"#), + 3_000, + ); + assert_eq!(framed(&machine).ask, HumanAsk::Pending(AskKind::Question)); + machine.apply_at( + &event(r#"{"type":"question.v2.rejected","data":{"requestID":"que_1"}}"#), + 4_000, + ); + assert_eq!(framed(&machine).ask, HumanAsk::None); + assert_eq!(machine.take_edges(), vec![], "an ask is not a fault"); + } + + /// A name this build cannot classify is NOT a null: the harness reported a failure, so it + /// stays visible under the most conservative truthful category with its recovery unclaimed + /// and the name carried as diagnostic granularity. Guards the presence-tested-lookup + /// regression where an unknown name quietly becomes a neighbouring category. + #[test] + fn an_unclassified_error_stays_visible_under_the_conservative_category() { + for (name, code) in [ + ("UnknownError", "opencode/session_error.UnknownError"), + ("SomeFutureError", "opencode/session_error.SomeFutureError"), + ("weird name/slashed", "opencode/session_error.weird_name_slashed"), + ] { + let mut machine = EventMachine::default(); + machine.apply_at( + &event(&format!( + r#"{{"type":"session.error","properties":{{"sessionID":"ses_a","error":{{"name":"{name}"}}}}}}"# + )), + 1_000, + ); + let fault = standing(&machine); + assert_eq!(fault.category, FaultCategory::Harness, "{name}"); + assert_eq!(fault.code.as_deref(), Some(code)); + assert_eq!( + fault.recovery, + Recovery::Unknown, + "{name}: no recovery class is claimed for a verdict st2 cannot read" + ); + assert_eq!( + framed(&machine).reason.as_deref(), + Some(format!("error:{name}").as_str()) + ); + } + } + + /// `APIError` by the status the provider returned, split across the closed category set the + /// final vocabulary forces: 402 is a billing wall, 429 a throttle, 5xx the provider's own + /// failure. There is no `network` word and none is improvised. + #[test] + fn api_error_statuses_split_across_the_closed_category_set() { + for (status, category, code, recovery) in [ + ( + 401, + FaultCategory::Authentication, + "opencode/APIError.401", + Recovery::Human, + ), + ( + 403, + FaultCategory::Authentication, + "opencode/APIError.403", + Recovery::Human, + ), + ( + 402, + FaultCategory::Account, + "opencode/APIError.402", + Recovery::Human, + ), + ( + 429, + FaultCategory::RateLimit, + "opencode/APIError.429", + Recovery::Automatic, + ), + ( + 400, + FaultCategory::Configuration, + "opencode/APIError.400", + Recovery::Human, + ), + ( + 503, + FaultCategory::Provider, + "opencode/APIError.503", + Recovery::Automatic, + ), + ] { + let fault = api_error_fault( + &event(&format!(r#"{{"error":{{"data":{{"status":{status}}}}}}}"#)), + 1_000, + ); + assert_eq!( + (fault.category, fault.code.as_deref(), fault.recovery), + (category, Some(code), recovery), + "status {status}" + ); + } + // The harness's own verdict on retryability overrides the class in both directions… + assert_eq!( + api_error_fault( + &event(r#"{"error":{"data":{"status":429,"isRetryable":false}}}"#), + 1_000 + ) + .recovery, + Recovery::Human + ); + assert_eq!( + api_error_fault( + &event(r#"{"error":{"data":{"status":400,"isRetryable":true}}}"#), + 1_000 + ) + .recovery, + Recovery::Automatic + ); + // …and a status this version cannot read claims nothing narrower than the provider. + let unreadable = api_error_fault(&event(r#"{"error":{"data":{}}}"#), 1_000); + assert_eq!(unreadable.category, FaultCategory::Provider); + assert_eq!(unreadable.recovery, Recovery::Unknown); + assert_eq!(unreadable.code.as_deref(), Some("opencode/APIError")); + } + + /// The server threw its own instance away: a harness fault a person must act on, NOT this + /// record's last word — the child may still be reaped normally, and its exit is the + /// process-exit owner's to write. + #[test] + fn disposal_is_a_harness_fault_not_a_terminal() { + for kind in ["global.disposed", "server.instance.disposed"] { + let mut machine = EventMachine::default(); + machine.seed_idle(); + machine.apply_at( + &event(&format!(r#"{{"type":"{kind}","properties":{{}}}}"#)), + 1_000, + ); + let fault = standing(&machine); + assert_eq!(fault.category, FaultCategory::Harness, "{kind}"); + assert_eq!(fault.code.as_deref(), Some("opencode/disposed")); + assert_eq!(fault.recovery, Recovery::Human); + assert_eq!(framed(&machine).state, Activity::Idle, "{kind}"); + assert_eq!( + machine.observation().map(|observation| observation.state), + Some(Activity::Idle), + "{kind} writes no legacy terminal either" + ); + } + } + + /// A status word this version cannot read makes the busy map untrustworthy, not the fault + /// set: the ACTIVITY axis is withheld until a level seed rebuilds it, while the standing + /// fault stays readable and is restated for the record across the swap. + #[test] + fn a_poisoned_status_withholds_activity_but_not_a_standing_fault() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"StructuredOutputError"}}}"#, + ), + 1_000, + ); + let fault = standing(&machine); + let _ = machine.take_edges(); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"hibernating"}}}"#, + ), + 2_000, + ); + assert!(machine.poisoned); + assert_eq!(machine.frame(), None); + assert_eq!(machine.winner(), Some(&fault)); + + let mut reseeded = machine.reconnected(); + reseeded.seed_idle(); + assert_eq!( + reseeded.frame().map(|frame| frame.state), + Some(Activity::Idle), + "the level seed owns the activity axis" + ); + assert_eq!(reseeded.winner(), Some(&fault), "the fault rode across"); + assert_eq!( + reseeded.take_edges(), + vec![ConditionEdge::Raise(fault)], + "and is restated, because the record is the only other place it lives" + ); + } + + /// OHS-R22: the conversation reference is identity plus capability, from typed evidence only. + #[test] + fn the_conversation_is_linked_from_the_typed_session_id_or_unavailable() { + assert_eq!( + EventMachine::default().conversation_state(), + ConversationState::Unavailable(Some("no-session".to_string())), + "never `unsupported`: OpenCode HAS conversation identity, this seat has no session yet" + ); + + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"busy"}}}"#, + ), + 4_200, + ); + let ConversationState::Linked(link) = machine.conversation_state() else { + panic!("a linked conversation"); + }; + assert_eq!(link.driver, "opencode"); + assert_eq!(link.conversation, "ses_a"); + assert_eq!(link.history_mutability, HistoryMutability::Rewritable); + assert_eq!(link.capability_evidence, CapabilityEvidence::Probed); + assert_eq!(link.verified_through_ms, 4_200); + assert_eq!( + framed(&machine).conversation, + Some(ConversationState::Linked(link)) + ); + } + + /// I10 as a blanket property: whatever OpenCode reports on its own surface, the version 3 + /// projection never spells a terminal record. Only the wrapper's exit path does. + #[test] + fn no_native_event_ever_projects_a_terminal_frame() { + let mut machine = EventMachine::default(); + machine.seed_idle(); + for raw in [ + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ProviderAuthError"}}}"#, + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"APIError","data":{"status":402}}}}"#, + r#"{"type":"global.disposed","properties":{}}"#, + r#"{"type":"server.instance.disposed","properties":{}}"#, + r#"{"type":"session.idle","properties":{"sessionID":"ses_a"}}"#, + ] { + machine.apply_at(&event(raw), 1_000); + let frame = framed(&machine); + assert_ne!(frame.state, Activity::Ended, "{raw}"); + assert_eq!(frame.exit, None, "{raw}: a producer frame carries no exit"); + } + // …while the legacy projection keeps the version 2 word it has always written. + assert_eq!(observed(&machine).state, Activity::Ended); + } + + /// A version 3 record must have its condition axis STATED once: a claim fence has observed + /// nothing, so it carries no axis for `Unchanged` to inherit and the writer refuses that + /// frame as `Unstated`. This is what the retry states instead — the winning fault if one + /// stands, and a positive `clear` if none does. + #[test] + fn the_first_frame_states_the_condition_axis_a_claim_fence_left_absent() { + let mut machine = EventMachine::default(); + machine.seed_idle(); + assert_eq!( + machine.bootstrap_condition(), + ConditionReport::Clear, + "a seeded producer with no fault has earned the positive health claim" + ); + assert_eq!(machine.frame().unwrap().condition, ConditionReport::Unchanged); + assert_eq!( + machine + .frame_with(machine.bootstrap_condition()) + .unwrap() + .condition, + ConditionReport::Clear + ); + + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ProviderAuthError"}}}"#, + ), + 1_000, + ); + let fault = standing(&machine); + assert_eq!( + machine.bootstrap_condition(), + ConditionReport::Fault(fault.clone()), + "the bootstrap states the slot the reducer projects, not a blank clear" + ); + assert_eq!( + machine + .frame_with(machine.bootstrap_condition()) + .unwrap() + .condition, + ConditionReport::Fault(fault) + ); + // The ordinary frame still leaves the axis alone: only the bootstrap states it. + assert_eq!(machine.frame().unwrap().condition, ConditionReport::Unchanged); + } + + /// The write path end to end through a REAL [`Writer`] over a real claim, at the version this + /// build emits. It pins the two boundaries the fault axis must not disturb: the seat becomes + /// observable over the exitless fence, and the WRAPPER's exit is the only thing that ever + /// writes a terminal record. + #[test] + fn a_real_writer_bootstraps_over_the_claim_fence_and_only_ended_writes_the_terminal() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path().join("agents/h/worker"); + std::fs::create_dir_all(&agent_dir).unwrap(); + let token = harness_state::session_token(); + let seq = harness_state::claim(&agent_dir, "h.worker", "opencode", &token).unwrap(); + let mut writer = Writer::new(&agent_dir, "h.worker", "opencode", Some("worker".to_string())) + .with_ownership(token, seq); + let path = harness_state::harness_state_path(&agent_dir); + + // The claim fence has observed nothing, so it reads INDETERMINATE rather than as a + // definite terminal — which is exactly why the adapter's first frame has to state the + // condition axis instead of inheriting one. + let fence = harness_state::read(&path, None).expect("the claim fence"); + assert_eq!(fence.state, Activity::Unknown); + assert_eq!(fence.reason.as_deref(), Some("claimed")); + assert_eq!(fence.exit, None); + + let mut machine = EventMachine::default(); + machine.apply(&event( + r#"{"type":"session.status","properties":{"sessionID":"ses_a","status":{"type":"busy"}}}"#, + )); + machine.apply(&event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ContextOverflowError"}}}"#, + )); + + if writer.writes_condition_axis() { + let outcome = writer.publish(machine.frame().expect("a frame")).unwrap(); + if matches!(outcome, WriteOutcome::Refused(Refusal::Unstated)) { + let stated = machine + .frame_with(machine.bootstrap_condition()) + .expect("a stated frame"); + assert!( + writer.publish(stated).unwrap().accepted(), + "the bootstrap retry states the axis the fence left absent" + ); + } + for edge in machine.take_edges() { + let outcome = match edge { + ConditionEdge::Raise(fault) => writer.raise_fault(fault), + ConditionEdge::ClearPaired(key) => writer.clear_fault(key), + ConditionEdge::ClearAll(proof) => writer.clear_all(proof), + }; + assert!(outcome.unwrap().accepted()); + } + assert!( + harness_state::read(&path, None) + .expect("a live record") + .condition + .fault() + .is_some(), + "the standing fault reached the record" + ); + } else { + // The live path at this build: the axis is unrepresentable, so it is refused as a + // VALUE rather than cached anywhere, and the legacy observation is what lands. + assert_eq!( + writer + .raise_fault( + FaultReport::new(FaultCategory::Harness, Recovery::Human, 1_000) + .with_code("opencode/disposed") + ) + .unwrap(), + WriteOutcome::Refused(Refusal::ConditionUnrepresentable { + schema: harness_state::SCHEMA_V2 + }) + ); + writer.observe(machine.observation().expect("an observation")).unwrap(); + let live = harness_state::read(&path, None).expect("a live record"); + assert_eq!(live.state, Activity::Idle, "the error settled the turn"); + assert_eq!(live.reason.as_deref(), Some("error:ContextOverflowError")); + assert_eq!(live.exit, None, "a live record carries no exit"); + assert_eq!( + live.condition, + harness_state::ConditionView::Absent, + "version 2 has no condition axis and infers none from its legacy words" + ); + } + + // The terminal record is the wrapper's, through the one helper every exit owner uses, + // with the exit the reap observed. + write_terminal( + &mut writer, + "exit 0", + None, + machine.bootstrap_condition(), + ); + let terminal = harness_state::read(&path, None).expect("a terminal record"); + assert_eq!(terminal.state, Activity::Ended); + assert_eq!(terminal.exit.as_deref(), Some("exit 0")); + } + + /// A session can exit before it ever published a frame — a launch failure, an immediate reap — + /// and its terminal must still land over the virgin claim fence, which has no condition axis + /// to leave unchanged. Real claim, real [`Writer`], real read-back, for every exit owner's + /// exact `(exit, reason)` pair. + #[test] + fn a_terminal_lands_over_the_claim_fence_before_any_frame_was_published() { + for (exit, reason) in [ + // run(): spawn_provider failed. + ("exit unknown", Some("launch-error")), + // The STOP path's escalation cover, and its reap. + ("stopped", None), + ("exit 0", None), + ] { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path().join("agents/h/worker"); + std::fs::create_dir_all(&agent_dir).unwrap(); + let path = harness_state::harness_state_path(&agent_dir); + let token = harness_state::session_token(); + let seq = harness_state::claim(&agent_dir, "h.worker", "opencode", &token).unwrap(); + let mut writer = + Writer::new(&agent_dir, "h.worker", "opencode", Some("worker".to_string())) + .with_ownership(token, seq); + assert_eq!( + harness_state::read(&path, None).unwrap().reason.as_deref(), + Some("claimed"), + "nothing has been observed on this record yet" + ); + + write_terminal( + &mut writer, + exit, + reason, + EventMachine::default().bootstrap_condition(), + ); + + let terminal = harness_state::read(&path, None).expect("a terminal record"); + assert_eq!(terminal.state, Activity::Ended, "{exit}"); + assert_eq!(terminal.exit.as_deref(), Some(exit)); + assert_eq!(terminal.reason.as_deref(), reason, "{exit}"); + assert_eq!(terminal.blocked_on, BlockedOn::None); + assert_eq!(terminal.ask, Ask::None); + } + } + + /// Routing every exit owner through one helper moved no bytes: while the writer emits version + /// 2 the helper IS [`Writer::ended`]'s legacy statement. Once the writer flips, the same + /// helper's first terminal states the axis a fence left absent instead. + #[test] + fn the_terminal_helper_keeps_the_legacy_ended_bytes_until_the_writer_flips() { + let fixture = || { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path().join("agents/h/worker"); + std::fs::create_dir_all(&agent_dir).unwrap(); + let path = harness_state::harness_state_path(&agent_dir); + let seq = + harness_state::claim(&agent_dir, "h.worker", "opencode", "fixed-token").unwrap(); + let writer = + Writer::new(&agent_dir, "h.worker", "opencode", Some("worker".to_string())) + .with_ownership("fixed-token", seq); + (tmp, path, writer) + }; + // The write's own clock is the only thing two separate writes may differ in. + let normalized = |path: &Path| { + let mut value: Value = serde_json::from_str(&std::fs::read_to_string(path).unwrap()) + .unwrap(); + let object = value.as_object_mut().unwrap(); + object.remove("writtenAtMs"); + object.remove("sinceMs"); + value + }; + + let (_helper_tmp, helper_path, mut helper_writer) = fixture(); + if helper_writer.writes_condition_axis() { + // Version 3: the fence carries no axis, so the plain terminal is refused and the + // one-shot retry states it. + let mut probe = + Frame::new(Activity::Ended, InputBuffer::Unknown, ConditionReport::Unchanged, HumanAsk::None); + probe = probe.with_exit("exit 0"); + assert_eq!( + helper_writer.publish(probe).unwrap(), + WriteOutcome::Refused(Refusal::Unstated), + "an unstated axis over a fence is refused, which is what the helper retries" + ); + write_terminal(&mut helper_writer, "exit 0", None, ConditionReport::Clear); + let terminal = harness_state::read(&helper_path, None).expect("a terminal record"); + assert_eq!(terminal.state, Activity::Ended); + assert_eq!(terminal.exit.as_deref(), Some("exit 0")); + assert_eq!(terminal.condition, harness_state::ConditionView::Clear); + return; + } + + write_terminal(&mut helper_writer, "exit 0", None, ConditionReport::Clear); + let (_legacy_tmp, legacy_path, mut legacy_writer) = fixture(); + legacy_writer.ended("exit 0").unwrap(); + assert_eq!( + normalized(&helper_path), + normalized(&legacy_path), + "the version 2 terminal is exactly the legacy statement" + ); + } + + /// The level surface is the authoritative statement of who is retrying: a retry fault carried + /// across a reseed whose session the map no longer reports as retrying is over, because its + /// own exit arm passed while the stream was down. + #[test] + fn a_reseed_retires_a_retry_the_level_surface_no_longer_reports() { + let server = spawn_fake_server(); + let client = Client::new(server.port, "pw"); + + let mut machine = EventMachine::default(); + for session in ["ses_gone", "ses_busy", "ses_still"] { + machine.apply_at( + &event(&format!( + r#"{{"type":"session.status","properties":{{"sessionID":"{session}","status":{{"type":"retry","action":{{"reason":"rate_limit"}}}}}}}}"# + )), + 1_000, + ); + } + assert_eq!(machine.retrying.len(), 3); + let _ = machine.take_edges(); + + // `ses_gone` is absent (idle sessions are omitted), `ses_busy` reports another word, and + // only `ses_still` is still retrying. + *server.status_body.lock().unwrap() = Some( + r#"{"ses_busy":{"type":"busy"},"ses_still":{"type":"retry"}}"#.to_string(), + ); + assert!(seed_from_server(&client, &mut machine).is_ok()); + assert_eq!( + machine.retrying.keys().collect::>(), + vec!["ses_still"], + "only the session the level surface still reports keeps its retry fault" + ); + assert_eq!( + standing(&machine).code.as_deref(), + Some("opencode/retry.rate_limit"), + "the surviving retry still stands, under its own exact code" + ); + assert_eq!(framed(&machine).state, Activity::Active); + } + + /// `session.idle` is the retry signal's other end event on that exact session — and only on + /// that one. + #[test] + fn session_idle_retires_the_retry_of_that_exact_session_only() { + let mut machine = EventMachine::default(); + for session in ["ses_a", "ses_b"] { + machine.apply_at( + &event(&format!( + r#"{{"type":"session.status","properties":{{"sessionID":"{session}","status":{{"type":"retry","action":{{"reason":"provider_error"}}}}}}}}"# + )), + 1_000, + ); + } + let _ = machine.take_edges(); + machine.apply_at( + &event(r#"{"type":"session.idle","properties":{"sessionID":"ses_a"}}"#), + 2_000, + ); + assert_eq!( + machine.retrying.keys().collect::>(), + vec!["ses_b"], + "ses_b's retry is untouched" + ); + assert_eq!( + standing(&machine).code.as_deref(), + Some("opencode/retry.provider_error"), + "the same code still stands for the other session" + ); + machine.apply_at( + &event(r#"{"type":"session.idle","properties":{"sessionID":"ses_b"}}"#), + 3_000, + ); + assert_eq!(machine.winner(), None, "both retries have now ended"); + assert_eq!(machine.retrying.len(), 0); + } + + /// `session.status{retry}.next` is disambiguated by MAGNITUDE, never by guessing: a small + /// number is a delay from the observation, an epoch-scale number is the instant itself, and an + /// instant already past is due now rather than before the observation that carries it. + #[test] + fn the_retry_deadline_is_read_by_epoch_magnitude() { + let now = 1_800_000_000_000_u64; + for (next, due, why) in [ + (30_000_u64, now + 30_000, "a delay in milliseconds"), + (now + 45_000, now + 45_000, "an absolute instant"), + (now - 45_000, now, "an absolute instant already past is due now"), + ] { + let mut machine = EventMachine::default(); + machine.apply_at( + &event(&format!( + r#"{{"type":"session.status","properties":{{"sessionID":"ses_a","status":{{"type":"retry","next":{next},"action":{{"reason":"rate_limit"}}}}}}}}"# + )), + now, + ); + let fault = standing(&machine); + assert_eq!(fault.next_observation_due_ms, Some(due), "{why}"); + assert!( + fault.next_observation_due_ms >= Some(fault.observed_at_ms), + "{why}: a deadline never precedes its observation" + ); + } + // A zero or unreadable `next` is no deadline at all, never a deadline of now. + for next in ["0", "null", r#""soon""#] { + let mut machine = EventMachine::default(); + machine.apply_at( + &event(&format!( + r#"{{"type":"session.status","properties":{{"sessionID":"ses_a","status":{{"type":"retry","next":{next}}}}}}}"# + )), + now, + ); + assert_eq!(standing(&machine).next_observation_due_ms, None, "next={next}"); + } + } + + /// A queued edge is an observation that already happened. A reconnect is not evidence against + /// it, so the queue rides across the swap rather than being swallowed by the blip — and the + /// carried slot is restated exactly once, not once per reconnect. + #[test] + fn a_reconnect_holds_the_queued_condition_edges() { + let mut machine = EventMachine::default(); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"APIError","data":{"status":402}}}}"#, + ), + 1_000, + ); + machine.apply_at( + &event( + r#"{"type":"message.updated","properties":{"info":{"role":"assistant","time":{"completed":2}}}}"#, + ), + 2_000, + ); + machine.apply_at( + &event( + r#"{"type":"session.error","properties":{"sessionID":"ses_a","error":{"name":"ContentFilterError"}}}"#, + ), + 3_000, + ); + let queued = machine.edges.clone(); + assert_eq!(queued.len(), 3, "raise, blanket clear, raise"); + + let carried = machine.reconnected(); + assert_eq!( + carried.edges, queued, + "nothing queued is dropped, and the final edge already states the slot" + ); + assert_eq!(carried.winner(), machine.winner()); + + // A second reconnect over a queue that no longer ends in the slot's own raise restates it + // once. + let mut drained = machine.reconnected(); + let _ = drained.take_edges(); + let restated = drained.reconnected(); + assert_eq!( + restated.edges, + vec![ConditionEdge::Raise(standing(&machine))], + "the record is the only other place the slot lives" + ); + } + + #[test] + fn openapi_gate_names_every_missing_marker() { + let complete: Value = serde_json::from_str(&format!( + r#"{{"paths":{{}},"markers":{:?}}}"#, + REQUIRED_API_MARKERS + )) + .unwrap(); + check_openapi_subset(&complete).unwrap(); + + let error = check_openapi_subset(&serde_json::json!({"paths": {"/session": {}}})) + .unwrap_err() + .to_string(); + assert!(error.contains("prompt_async"), "{error}"); + assert!(error.contains("permission.asked"), "{error}"); + } + + #[test] + fn stable_message_ids_are_grammatical_and_distinct_per_binding() { + let id = stable_message_id("h.worker", "ses_a", "1786380000000-abc123.md"); + assert!(id.starts_with("msg")); + assert_eq!( + id, + stable_message_id("h.worker", "ses_a", "1786380000000-abc123.md") + ); + for other in [ + stable_message_id("h.other", "ses_a", "1786380000000-abc123.md"), + stable_message_id("h.worker", "ses_b", "1786380000000-abc123.md"), + stable_message_id("h.worker", "ses_a", "1786380000000-def456.md"), + ] { + assert_ne!(id, other); + } + } + + #[test] + fn base64_matches_known_vectors() { + for (input, expected) in [ + (&b""[..], ""), + (b"f", "Zg=="), + (b"fo", "Zm8="), + (b"foo", "Zm9v"), (b"opencode:pw", "b3BlbmNvZGU6cHc="), ] { assert_eq!(base64(input), expected, "{input:?}"); @@ -2138,14 +3738,16 @@ mod tests { let server = spawn_fake_server(); let client = Client::new(server.port, "pw"); let state_path = tmp.path().join("state/delivery-state.json"); - let (mut delivery, _filename) = delivery_fixture(tmp.path(), state_path.clone()); + let (mut delivery, filename) = delivery_fixture(tmp.path(), state_path.clone()); server.read_back_error.store(true, Ordering::SeqCst); delivery.pump(&client); assert_eq!(server.posts.lock().unwrap().len(), 1, "one POST, attempted"); - let state: DeliveryState = - serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); - assert_eq!(state.phase, DeliveryPhase::Attempted); + assert_eq!( + ledger_phase(&state_path, &filename), + Some(delivery_ledger::Phase::TransportAccepted), + "the POST landed; nothing yet proves the server holds it" + ); // While the read-back stays indeterminate, no pass may re-POST. delivery.pump(&client); @@ -2155,9 +3757,11 @@ mod tests { // The read-back recovering flips the same attempt to Accepted with no second POST. server.read_back_error.store(false, Ordering::SeqCst); delivery.pump(&client); - let state: DeliveryState = - serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); - assert_eq!(state.phase, DeliveryPhase::Accepted); + assert_eq!( + ledger_phase(&state_path, &filename), + Some(delivery_ledger::Phase::Persisted), + "a recovered read-back proves storage, not consumption" + ); assert_eq!(server.posts.lock().unwrap().len(), 1); } @@ -2299,8 +3903,26 @@ mod tests { (delivery, filename) } + /// Read the ledger back through its own loader and the real correlation derivation: a test + /// that read the bytes directly would not notice a record the pump itself would refuse. + fn reopen_ledger(legacy_path: &Path) -> delivery_ledger::Ledger { + delivery_ledger::Ledger::open( + legacy_path, + delivery_ledger::Harness::OpenCode.profile(), + "h.worker", + "h.worker", + |session, file| stable_message_id("h.worker", session, file), + ) + } + + fn ledger_phase(legacy_path: &Path, filename: &str) -> Option { + reopen_ledger(legacy_path) + .entry(filename) + .map(|entry| entry.phase) + } + #[test] - fn delivery_attempts_before_transport_and_accepts_only_the_read_back_receipt() { + fn a_durable_read_back_is_persistence_that_never_releases_the_inbox_entry() { let tmp = tempfile::tempdir().unwrap(); let server = spawn_fake_server(); let client = Client::new(server.port, "pw"); @@ -2313,43 +3935,62 @@ mod tests { server.posts.lock().unwrap().as_slice(), [expected_id.clone()] ); - let state: DeliveryState = - serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); - assert_eq!(state.phase, DeliveryPhase::Accepted); - assert_eq!(state.message_id, expected_id); + // Same server fixture, same single-POST conclusion, honest label: `GET 200` is storage. + let entry = reopen_ledger(&state_path).entry(&filename).cloned().unwrap(); + assert_eq!(entry.phase, delivery_ledger::Phase::Persisted); + assert_eq!(entry.correlation.value, expected_id); + assert_eq!( + reopen_ledger(&state_path).retention(&filename), + delivery_ledger::Retention::Hold(delivery_ledger::HoldReason::UnreadReceipt), + "storage is not admission: ownership is retained until the scheduler proves it, and \ + only the recipient's own archive settles the message" + ); + assert!( + message::inbox_dir(&tmp.path().join("agents/h/worker")) + .join(&filename) + .is_file(), + "the ledger never archives an inbox file" + ); - // Accepted is terminal for this file: further pumps send nothing. + // Persistence is terminal for the POST loop: further pumps send nothing. delivery.pump(&client); delivery.pump(&client); assert_eq!(server.posts.lock().unwrap().len(), 1); } #[test] - fn a_stale_attempt_reconciles_by_reading_back_instead_of_resending() { + fn a_v1_attempt_is_adopted_and_reconciled_by_reading_back_instead_of_resending() { let tmp = tempfile::tempdir().unwrap(); let server = spawn_fake_server(); let client = Client::new(server.port, "pw"); let state_path = tmp.path().join("state/delivery-state.json"); - let (_, filename) = delivery_fixture(tmp.path(), state_path.clone()); + let agent_dir = tmp.path().join("agents/h/worker"); + let inbox = message::inbox_dir(&agent_dir); + std::fs::create_dir_all(&inbox).unwrap(); + let filename = + message::send_to_inbox(&inbox, "h.sender", Some("subject"), None, &[], "body").unwrap(); - // A prior incarnation attempted this exact delivery and the server made it durable. + // The migration boundary: a prior binary attempted this exact delivery and the server made + // it durable, and no ledger exists yet. let message_id = stable_message_id("h.worker", "ses_target", &filename); server.durable.lock().unwrap().insert(message_id.clone()); - atomic_json( + // The v1 record, in the exact shape the old binary wrote it. + std::fs::create_dir_all(state_path.parent().unwrap()).unwrap(); + std::fs::write( &state_path, - &DeliveryState { - schema: DELIVERY_STATE_SCHEMA.to_string(), - agent: "h.worker".to_string(), - runtime_id: "h.worker".to_string(), - session_id: "ses_target".to_string(), - filename, - message_id: message_id.clone(), - phase: DeliveryPhase::Attempted, - }, + serde_json::to_vec(&json!({ + "schema": delivery_ledger::OPENCODE_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "sessionId": "ses_target", + "filename": &filename, + "messageId": &message_id, + "phase": "attempted", + })) + .unwrap(), ) .unwrap(); - let agent_dir = tmp.path().join("agents/h/worker"); let mut delivery = Delivery::with_state_path( tmp.path(), &agent_dir, @@ -2362,9 +4003,125 @@ mod tests { delivery.pump(&client); assert!(server.posts.lock().unwrap().is_empty(), "must not resend"); - let state: DeliveryState = - serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); - assert_eq!(state.phase, DeliveryPhase::Accepted); + assert_eq!( + ledger_phase(&state_path, &filename), + Some(delivery_ledger::Phase::Persisted), + "the adopted attempt reconciles to storage without a second POST" + ); + } + + /// The load-bearing rollback limit from the migration sweep: a delivery STARTED by the new + /// binary has no v1 record, so a rolled-back binary takes its `state == None` path and POSTs + /// before it ever requeries — appending the same messageID's parts a second time on 1.18.19, + /// at 9 of 9 crash positions. The v1-shaped floor is what removes that whole class, and it + /// only works if it passes v1's own load filter, which recomputes the messageID. + #[test] + fn a_post_migration_delivery_leaves_a_v1_readable_floor_so_a_rollback_cannot_re_post() { + let tmp = tempfile::tempdir().unwrap(); + let server = spawn_fake_server(); + let client = Client::new(server.port, "pw"); + let state_path = tmp.path().join("state/delivery-state.json"); + let (mut delivery, filename) = delivery_fixture(tmp.path(), state_path.clone()); + server.read_back_error.store(true, Ordering::SeqCst); + + delivery.pump(&client); + assert_eq!(server.posts.lock().unwrap().len(), 1); + assert!( + state_path + .with_file_name(delivery_ledger::LEDGER_FILE) + .is_file(), + "authority lives in the fresh namespace" + ); + + // v1's own load filter, verbatim: schema, agent, filename grammar, and the recomputed + // `stable_message_id`. A floor failing any of these is silently discarded and buys nothing. + let floor: Value = serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); + assert_eq!(floor["schema"], delivery_ledger::OPENCODE_LEGACY_SCHEMA); + assert_eq!(floor["agent"], "h.worker"); + assert!(message::is_message_filename( + floor["filename"].as_str().unwrap() + )); + assert_eq!( + floor["messageId"].as_str().unwrap(), + stable_message_id("h.worker", floor["sessionId"].as_str().unwrap(), &filename) + ); + // Never advanced, so v1 reads a true lower bound and reconciles instead of accepting. + assert_eq!(floor["phase"], "attempted"); + + // It stays exactly that on every later pass while the delivery is outstanding. + delivery.pump(&client); + assert_eq!( + serde_json::from_slice::(&std::fs::read(&state_path).unwrap()).unwrap(), + floor + ); + assert_eq!(server.posts.lock().unwrap().len(), 1); + } + + /// A true quarantine is operator-visible on the existing typed boundary — the transport is + /// unavailable — and nowhere else: no new diagnostic vocabulary, and the raw reason stays in + /// tracing so no unbounded prose reaches the record. + #[test] + fn a_quarantined_ledger_publishes_the_typed_delivery_boundary_and_sends_nothing() { + let tmp = tempfile::tempdir().unwrap(); + let server = spawn_fake_server(); + let client = Client::new(server.port, "pw"); + let state_path = tmp.path().join("state/delivery-state.json"); + let agent_dir = tmp.path().join("agents/h/worker"); + let inbox = message::inbox_dir(&agent_dir); + std::fs::create_dir_all(&inbox).unwrap(); + message::send_to_inbox(&inbox, "h.sender", Some("subject"), None, &[], "body").unwrap(); + + // A v1 record whose messageID contradicts its own session and filename is provably not + // this agent's, so it can be neither adopted nor ignored: it fails closed. + std::fs::create_dir_all(state_path.parent().unwrap()).unwrap(); + std::fs::write( + &state_path, + serde_json::to_vec(&json!({ + "schema": delivery_ledger::OPENCODE_LEGACY_SCHEMA, + "agent": "h.worker", + "runtimeId": "h.worker", + "sessionId": "ses_target", + "filename": "1786380000000-abc123.md", + "messageId": "msgtampered", + "phase": "attempted", + })) + .unwrap(), + ) + .unwrap(); + + let mut delivery = Delivery::with_state_path( + tmp.path(), + &agent_dir, + "h", + "h.worker", + "h.worker", + state_path.clone(), + ); + delivery.saw_session("ses_target"); + let mut diagnostics = DiagnosticPublisher::new( + &agent_dir, + DiagnosticDriver::OpenCode, + Some("1.18.19".to_string()), + DiagnosticSupport::Supported, + ); + delivery.pump_diagnosed(&client, &mut diagnostics); + + assert!( + server.posts.lock().unwrap().is_empty(), + "a quarantined ledger authorizes no transport" + ); + let crate::driver_diagnostic::Observed::Failure(failure) = + crate::driver_diagnostic::read(&crate::driver_diagnostic::path(&agent_dir)) + else { + panic!("a quarantined ledger must be diagnosed") + }; + assert_eq!(failure.stage, DiagnosticStage::Delivery); + assert_eq!(failure.reason, DiagnosticReason::DeliveryUnavailable); + assert_eq!(failure.source, DiagnosticSource::PromptTransport); + assert!( + state_path.exists(), + "a record we refuse to read is not a record we may destroy" + ); } #[test] @@ -2377,9 +4134,11 @@ mod tests { let (mut delivery, filename) = delivery_fixture(tmp.path(), state_path.clone()); delivery.pump(&client); - let state: DeliveryState = - serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); - assert_eq!(state.phase, DeliveryPhase::Attempted); + assert_eq!( + ledger_phase(&state_path, &filename), + Some(delivery_ledger::Phase::Attempted), + "a refused POST never reaches transportAccepted" + ); server.accept_posts.store(true, Ordering::SeqCst); delivery.next_attempt = Instant::now(); @@ -2390,9 +4149,10 @@ mod tests { server.posts.lock().unwrap().as_slice(), [expected_id.clone(), expected_id] ); - let state: DeliveryState = - serde_json::from_slice(&std::fs::read(&state_path).unwrap()).unwrap(); - assert_eq!(state.phase, DeliveryPhase::Accepted); + assert_eq!( + ledger_phase(&state_path, &filename), + Some(delivery_ledger::Phase::Persisted) + ); } #[test] diff --git a/src/pi_channel.rs b/src/pi_channel.rs index af2fdd11..6067a0bf 100644 --- a/src/pi_channel.rs +++ b/src/pi_channel.rs @@ -35,10 +35,22 @@ inbox by reading, acting on, replying when useful, and archiving each handled me resuming or starting work, set your status to busy; set available only when yielding or ready for \ new work."; -/// The wire version the shipped extension is written against. A mismatch is the extension's to -/// refuse: st2 never guesses what an older asset understands. +/// The wire version the hello ADVERTISES as its floor, and the only one an already-published +/// asset understands. It stays literally 1 forever: the pinned v1 asset compares this field for +/// strict equality and closes the channel on a mismatch, so advertising a newer number here would +/// make every hook set in the field refuse the channel and silently stop all delivery. pub const PROTOCOL: u32 = 1; +/// Every wire version this build speaks, offered beside the floor so a NEWER asset can negotiate +/// up without the floor ever moving. The asset selects `max(supported ∩ protocols)` and refuses +/// only on an empty intersection; an older asset ignores the list and keeps reading `protocol`. +/// Version 2 adds the condition frame — nothing else — so an asset that stays on 1 keeps its +/// exact existing behaviour. +/// +/// Additive by construction: an asset that does not know the field ignores it, and an asset that +/// does answers with a `client_hello` naming what it will speak. +pub const PROTOCOLS: [u32; 2] = [1, 2]; + /// Last-resort durable state when compaction begins before the agent authored a checkpoint. /// /// The stable text deliberately carries no extension-owned path or clock. Rust owns both the @@ -48,6 +60,12 @@ PreCompact fired before the model captured durable working state. Reconstruct fr recent commits, and the st2 inbox, then write a real checkpoint with `st2 context write`.\n"; const PRE_COMPACT_ERROR_REASON: &str = "pre-compact context recovery failed"; +/// The negotiated asset's diagnostic word for a refused approval. It is prose about an ASK that +/// is over — never a condition — and it exists only in the negotiated vocabulary, so the version +/// 2 projection withholds it: a record shape readers are pinned to must not grow a novel `reason` +/// on an unblocked frame because a newer asset started narrating one. +const APPROVAL_DENIED_REASON: &str = "approvalDenied"; + /// How pi is asked to hand one delivered message to the agent. /// /// `steer` is the only value st2 currently emits. It is the earliest point at which pi accepts @@ -71,6 +89,22 @@ fn channel_content(subject: Option<&str>, body: &str) -> String { } } +/// st2's hello: the version the asset must understand, and every version st2 would also accept. +/// +/// `protocol` stays 1 forever. The hello is st2 → asset and is written before any read, so a +/// control plane that raised it unilaterally would be REFUSED by every already-loaded asset — +/// and a refusal costs that seat its mail. The offer beside it is how a newer wire is reached +/// instead: additive, ignored by an old asset, answered by a new one. +fn hello(identity: &str, session_context: &str) -> Value { + json!({ + "type": "hello", + "protocol": PROTOCOL, + "protocols": PROTOCOLS, + "identity": identity, + "sessionContext": session_context, + }) +} + /// The harness-specific facts the shared channel loop needs: which env names carry the wrapper's /// exported ownership triple, what label goes on records and errors, and which native-driver /// diagnostic word — if any — this channel publishes under. @@ -88,6 +122,18 @@ pub struct ChannelKind { pub runtime_id_env: &'static str, pub session_env: &'static str, pub seq_env: &'static str, + /// What this kind's frames state on the version 3 ask axis when the wire claims no pending + /// ask. `HumanAsk::Unknown` is a POSITIVE statement — "this harness exposes no ask surface at + /// all", which is pi's case on every frame forever — and it is stronger than it looks: a kind + /// carrying it never derives a `pending` ask from any frame, because a channel that cannot see + /// the axis cannot vouch for a foreign asset's claim about it. `None` is the ordinary answer + /// for a kind that does see the axis (omp) and reports nothing waiting. + pub default_ask: harness_state::HumanAsk, + /// The conversation axis this kind can state with no evidence off the wire. pi has no + /// conversation identity to expose at all; omp demonstrably has sessions (`--no-session`, + /// `sessionManager`), so it states NOTHING until an event exposes one — never `Unsupported`, + /// which would be a false capability claim. + pub conversation: Option, } const PI_KIND: ChannelKind = ChannelKind { @@ -97,6 +143,8 @@ const PI_KIND: ChannelKind = ChannelKind { runtime_id_env: crate::pi_session::CHANNEL_RUNTIME_ID, session_env: crate::pi_session::CHANNEL_SESSION, seq_env: crate::pi_session::CHANNEL_SEQ, + default_ask: harness_state::HumanAsk::Unknown, + conversation: Some(harness_state::ConversationState::Unsupported), }; const OMP_KIND: ChannelKind = ChannelKind { @@ -106,6 +154,8 @@ const OMP_KIND: ChannelKind = ChannelKind { runtime_id_env: crate::omp_session::CHANNEL_RUNTIME_ID, session_env: crate::omp_session::CHANNEL_SESSION, seq_env: crate::omp_session::CHANNEL_SEQ, + default_ask: harness_state::HumanAsk::None, + conversation: None, }; /// Run the pi native message channel over stdio. @@ -134,15 +184,10 @@ fn run_for(catalog_root: &Path, identity: &str, kind: &ChannelKind) -> Result<() } }); let mut stdout = io::BufWriter::new(io::stdout().lock()); - write_json( - &mut stdout, - &json!({ - "type": "hello", - "protocol": PROTOCOL, - "identity": identity, - "sessionContext": session_context, - }), - )?; + // Named so the version negotiation is provable without a live channel: the floor and the + // offered set are the one thing in this file an already-published asset compares against, and + // getting them wrong stops all delivery silently. + write_json(&mut stdout, &hello(identity, &session_context))?; stdout.flush()?; // The channel owns the live half of observed harness state: it is the one process that sees // the harness's own turn events, and its stdio connection to the extension is the evidence @@ -236,6 +281,24 @@ fn channel_loop( let mut delivered = HashSet::new(); let label = kind.label; let mut next_heartbeat = Instant::now() + heartbeat_every; + // What this CONNECTION agreed to speak. A session replacement re-spawns this channel from a + // possibly-replaced binary while the loaded asset is the predecessor's, so the agreement is + // per connection and its absence is the default rather than a failure. + let mut negotiated: Option = None; + // The conversation identity, once the asset has forwarded one. It rides every later frame: + // the axis has no operation of its own, and restating an activity nobody observed just to + // carry an identity would refresh a stale state. + let mut conversation: Option = None; + // A raise the record had no observation of this session's to attach to. It is held rather + // than dropped and rather than published beside a fabricated activity: the seat is faulted, + // and the first genuine activity edge is what makes the fault recordable. Unreachable in + // practice — the extension seeds a state frame from `ctx.isIdle()` at open time — which is + // exactly why the fallback must not be a guess. + let mut deferred_fault: Option = None; + // The adapter-owned fault that is still TRUE, tracked separately from whatever occupies the + // record's single condition slot: a later provider fault may displace it there, and a turn + // that completes retires the provider's fault without touching st2's own. + let mut harness_fault: Option = None; loop { match input.recv_timeout(poll) { Ok(line) => { @@ -251,16 +314,107 @@ fn channel_loop( // The typed turn result, decoded once: it feeds two independent records and the // credential edge must not depend on the categorical write landing. let turn = frame.as_ref().and_then(turn_result); + if let Some(protocol) = frame.as_ref().and_then(negotiated_protocol) { + negotiated = Some(protocol); + } + // Which wire this channel writes is the WRITER's business alone: version 3 has + // the condition axis, version 2 does not, and a record has exactly one source of + // truth either way. Negotiation is a separate question about what the ASSET + // promised, and it narrows two axes rather than the whole write — see + // [`connection_frame`]. Withholding the version 3 tuple from an un-negotiated + // peer would be strictly worse: its faults are on the same typed turn frame, so + // a wedged seat would go unstated. + let states_tuple = writer.writes_condition_axis(); + let promoted = negotiated == Some(PROTOCOL_CONDITION_AXIS); + if promoted + && let Some(claim) = frame + .as_ref() + .and_then(|frame| conversation_claim(frame, crate::message::now_ms())) + { + conversation = Some(claim); + } if let Some(observation) = frame .as_ref() .and_then(state_observation) .or_else(|| turn.as_ref().and_then(turn_observation)) + { + if states_tuple { + let mut published = + connection_frame(kind, observation, promoted, conversation.clone()); + // The condition rides the SAME write as the activity it was observed + // with: they are one look at the harness, and correlating them across + // two writes is a race a reader can lose. A fault the record had nowhere + // to attach yet takes this frame instead of being dropped. + if let Some(fault) = turn + .as_ref() + .and_then(|turn| turn_fault(turn, crate::message::now_ms())) + .or_else(|| deferred_fault.take()) + { + published.condition = harness_state::ConditionReport::Fault(fault); + } + if let Err(error) = publish_frame(writer, published, label) { + tracing::warn!( + "st2 {label} channel: recording observed state failed: {error}" + ); + } // A queued live frame must never overwrite the wrapper's terminal record: // the channel and the wrapper are separate processes, so the flock alone // serializes but does not order their writes. - && let Err(error) = writer.observe_unless_ended(observation) + } else if let Err(error) = + writer.observe_unless_ended(legacy_observation(observation)) + { + tracing::warn!( + "st2 {label} channel: recording observed state failed: {error}" + ); + } + // An activity edge is the one thing that can retire a deferred raise's + // reason for waiting: the record now holds an observation of this session's + // for the fault to attach to. + if let Some(fault) = deferred_fault.take() { + deferred_fault = apply_condition(writer, ConditionEdge::Raise(fault), label); + } + } + // The condition axis, on its own frame and deliberately without an activity + // claim: pi's fault evidence carries none, and restating activity here would + // refresh a stale one from a frame that observed nothing about it. An activity + // edge never clears a condition, and a condition edge never restates activity. + if let Some(edge) = frame + .as_ref() + .and_then(|frame| condition_frame(frame, message::now_ms())) { - tracing::warn!("st2 {label} channel: recording observed state failed: {error}"); + if states_tuple { + // A clear the harness sent while a raise is still held retires that + // raise. Without this the held fault outlives the very edge that + // resolved it and the next activity edge republishes a condition the + // harness has already retracted. + if deferred_fault + .as_ref() + .is_some_and(|held| retires_deferred(&edge, held)) + { + deferred_fault = None; + } + if let Some(fault) = apply_condition(writer, edge, label) { + deferred_fault = Some(fault); + } + } else { + // Version 2 has nowhere to carry it and this record has exactly one + // source of truth, so the edge is dropped rather than cached where the + // session's sibling writers could not see it. + tracing::debug!( + "st2 {label} channel: condition frame dropped: this build's records \ + carry no condition axis" + ); + } + } + // The one positive success edge on the whole axis: a turn that reached its + // ordinary end. Nothing else clears everything — not an activity edge, not an + // approval resolution, not a compaction, and least of all a retry omp is about to + // make, which sends no frame at all. And it clears only what it is evidence + // about: an adapter-owned pre-compact failure that still holds is restated, since + // a working provider says nothing about st2's own failed write. + if states_tuple && matches!(turn.as_ref(), Some(TurnResult::Ordinary)) { + let _cleared = + apply_condition(writer, turn_completed_edge(harness_fault.as_ref()), label); } // The credential axis is a third record, independent of the numbers and of the // categorical state: a rejection stands until a turn reaches its ordinary end, @@ -284,22 +438,53 @@ fn channel_loop( } if frame.as_ref().is_some_and(|frame| { frame.get("type").and_then(Value::as_str) == Some("pre_compact") - }) && let Err(error) = ensure_pre_compact_context(agent_dir) - { - tracing::warn!( - "st2 {label} channel: writing pre-compact context stub failed: {error}" - ); - let actionable = harness_state::Observation::new( - harness_state::Activity::Active, - harness_state::BlockedOn::None, - harness_state::InputBuffer::Unknown, - ) - .with_reason(PRE_COMPACT_ERROR_REASON); - if let Err(state_error) = writer.observe_unless_ended(actionable) { - tracing::warn!( - "st2 {label} channel: recording pre-compact recovery failure failed: \ - {state_error}" - ); + }) { + match ensure_pre_compact_context(agent_dir) { + Err(error) => { + tracing::warn!( + "st2 {label} channel: writing pre-compact context stub failed: \ + {error}" + ); + if states_tuple { + // st2's own plumbing is what broke, so this is the one fault + // this adapter OWNS rather than observes. It is raised without + // restating the activity axis: nothing was learned about whether + // the model is working. It is also remembered, because only the + // next SUCCESSFUL pre-compact edge may retire it. + let edge = pre_compact_edge(false, crate::message::now_ms()); + if let ConditionEdge::Raise(fault) = &edge { + harness_fault = Some(fault.clone()); + } + deferred_fault = apply_condition(writer, edge, label); + } else { + let actionable = harness_state::Observation::new( + harness_state::Activity::Active, + harness_state::BlockedOn::None, + harness_state::InputBuffer::Unknown, + ) + .with_reason(PRE_COMPACT_ERROR_REASON); + if let Err(state_error) = writer.observe_unless_ended(actionable) { + tracing::warn!( + "st2 {label} channel: recording pre-compact recovery \ + failure failed: {state_error}" + ); + } + } + } + // The stub is there now, so the failure a previous edge recorded is over. + // The clear names the category AND the full code — never the category + // alone and never a blanket clear — so a standing credential rejection + // survives a compaction that went fine. + Ok(_) if states_tuple => { + deferred_fault = None; + harness_fault = None; + let _cleared = apply_condition( + writer, + pre_compact_edge(true, crate::message::now_ms()), + label, + ); + } + Ok(_) => {} } } } @@ -357,6 +542,297 @@ fn state_observation(frame: &Value) -> Option { Some(observation) } +/// The tagged ask axis as ONE kind's asset can state it. +/// +/// A kind whose default is `Unknown` cannot see an ask surface at all, so nothing it emits may be +/// promoted into a positive answer — not even the `blockedOn: none` its frames carry by default, +/// which is the pre-axis spelling of "nothing to report" rather than of "no human is waiting". +fn tagged_ask( + kind: &ChannelKind, + blocked_on: harness_state::BlockedOn, + ask: harness_state::Ask, +) -> harness_state::HumanAsk { + if kind.default_ask == harness_state::HumanAsk::Unknown { + return harness_state::HumanAsk::Unknown; + } + match blocked_on { + // A real pending ask. An unnamed or unrecognized kind stays indeterminate: the ask is + // real and its kind unstated, which is not the same as no ask. + harness_state::BlockedOn::Human => harness_state::HumanAsk::Pending(match ask { + harness_state::Ask::Permission => harness_state::AskKind::Permission, + harness_state::Ask::Question => harness_state::AskKind::Question, + harness_state::Ask::Review => harness_state::AskKind::Review, + harness_state::Ask::None | harness_state::Ask::Unknown => { + harness_state::AskKind::Unknown + } + }), + harness_state::BlockedOn::None => kind.default_ask, + harness_state::BlockedOn::Unknown => harness_state::HumanAsk::Unknown, + } +} + +/// The version 3 tuple one observation states for this kind. +/// +/// The condition axis is `Unchanged`, always: an activity or ask edge has learned NOTHING about +/// whether the provider is faulted, and a producer forced to pick `clear` there would fabricate +/// health several times a turn. The caller replaces it only where it genuinely observed a +/// condition with the same look at the harness. +fn kind_frame(kind: &ChannelKind, observation: harness_state::Observation) -> harness_state::Frame { + let mut frame = harness_state::Frame::new( + observation.state, + observation.input_buffer, + harness_state::ConditionReport::Unchanged, + tagged_ask(kind, observation.blocked_on, observation.ask), + ); + if let Some(conversation) = kind.conversation.clone() { + frame = frame.with_conversation(conversation); + } + if let Some(reason) = observation.reason { + frame = frame.with_reason(reason); + } + if let Some(exit) = observation.exit { + frame = frame.with_exit(exit); + } + frame +} + +/// What one `type: "condition"` frame asks of the condition axis. Three operations and no fourth: +/// there is no word here that ends a session, because a condition frame is never evidence that a +/// process exited — the outer session wrapper alone writes the terminal record. omp's adapters +/// mint the same three operations from its typed turn frame and its own pre-compact edge. +#[derive(Debug, Clone, PartialEq, Eq)] +enum ConditionEdge { + Raise(harness_state::FaultReport), + ClearPaired(harness_state::FaultKey), + ClearAll(harness_state::ProgressProof), +} + +/// Decode a condition frame, protocol 2. +/// +/// Same discipline as [`state_observation`], for the same reason: an unrecognized `op`, a category +/// outside the closed set, a recovery word this build does not know, a `code` that is not +/// `provider/code`, and a `clearAll` whose proof word is missing or unknown each drop the frame +/// silently rather than guessing. A newer asset must not be able to make this channel record +/// something it cannot vouch for — and of the two failure directions, dropping an edge leaves the +/// record as it was while guessing one publishes a claim nobody made. +/// +/// The observation instant is stamped HERE, by Rust: the asset and this process share no clock, +/// so a semantic timestamp off the wire would be another harness's idea of now. +fn condition_frame(frame: &Value, observed_at_ms: u64) -> Option { + if frame.get("type").and_then(Value::as_str) != Some("condition") { + return None; + } + let op = frame.get("op").and_then(Value::as_str)?; + // The unkeyed clear names no fault, so it must name the progress it witnessed. There is + // deliberately no default: an unproven blanket clear is exactly how a fault that still holds + // stops paging. + if op == "clearAll" { + return Some(ConditionEdge::ClearAll( + match frame.get("proof").and_then(Value::as_str)? { + "turnCompleted" => harness_state::ProgressProof::TurnCompleted, + "requestSucceeded" => harness_state::ProgressProof::RequestSucceeded, + _ => return None, + }, + )); + } + let category = fault_category(frame.get("category").and_then(Value::as_str)?)?; + // Absent is a real answer (a fault, or a clear, carrying no code at all); present-but-not- + // namespaced is a malformed frame, because a bare word is how two providers' codes collide. + let code = match frame.get("code") { + None | Some(Value::Null) => None, + Some(Value::String(code)) + if code + .split_once('/') + .is_some_and(|(provider, rest)| !provider.is_empty() && !rest.is_empty()) => + { + Some(code.clone()) + } + Some(_) => return None, + }; + match op { + "raise" => { + let recovery = fault_recovery(frame.get("recovery").and_then(Value::as_str)?)?; + let mut fault = harness_state::FaultReport::new(category, recovery, observed_at_ms); + if let Some(code) = code { + fault = fault.with_code(code); + } + // Diagnostic only, and the only place provider prose is allowed to ride: no consumer + // branches on it, and nothing above ever reclassifies the category from it. + if let Some(detail) = frame + .get("detail") + .and_then(Value::as_str) + .filter(|detail| !detail.is_empty()) + { + fault = fault.with_detail(detail); + } + Some(ConditionEdge::Raise(fault)) + } + // A paired clear names the EXACT `(category, code)` of the fault it clears — never the + // category alone, which is how one provider signal silences a different live fault in the + // same category. + "clear" => { + let mut key = harness_state::FaultKey::new(category); + if let Some(code) = code { + key = key.with_code(code); + } + Some(ConditionEdge::ClearPaired(key)) + } + _ => None, + } +} + +/// The closed category vocabulary, spelled out rather than inferred: a word outside it drops the +/// frame here instead of becoming an untyped fault, because a producer in this build has no +/// excuse for a category it cannot name. +fn fault_category(word: &str) -> Option { + Some(match word { + "authentication" => harness_state::FaultCategory::Authentication, + "account" => harness_state::FaultCategory::Account, + "quota" => harness_state::FaultCategory::Quota, + "rateLimit" => harness_state::FaultCategory::RateLimit, + "provider" => harness_state::FaultCategory::Provider, + "context" => harness_state::FaultCategory::Context, + "configuration" => harness_state::FaultCategory::Configuration, + "policy" => harness_state::FaultCategory::Policy, + "harness" => harness_state::FaultCategory::Harness, + _ => return None, + }) +} + +/// The closed recovery vocabulary. `unknown` is a word an asset may STATE — "the harness said +/// nothing about who clears this" — and it is never optimistic: it pages exactly like `human`. +fn fault_recovery(word: &str) -> Option { + Some(match word { + "automatic" => harness_state::Recovery::Automatic, + "human" => harness_state::Recovery::Human, + "terminal" => harness_state::Recovery::Terminal, + "unknown" => harness_state::Recovery::Unknown, + _ => return None, + }) +} + +/// Publish one activity edge, stating the condition axis once when the record has none. +/// +/// A version 3 record's condition axis is not writable as `absent` and silence is not a value +/// either, so the FIRST activity-only frame of an incarnation is refused with +/// [`harness_state::Refusal::Unstated`] — and pi's first frame is exactly that, the +/// `session_start` seed. Without the restatement a healthy pi seat would publish no record at +/// all, which reads as `unknown` for every consumer. +fn publish_frame( + writer: &mut harness_state::Writer, + frame: harness_state::Frame, + label: &str, +) -> Result<()> { + let outcome = writer.publish_unless_ended(frame.clone())?; + if let Some(stated) = restate_condition(&frame, &outcome) { + report_outcome( + label, + "observed state", + &writer.publish_unless_ended(stated)?, + ); + return Ok(()); + } + report_outcome(label, "observed state", &outcome); + Ok(()) +} + +/// Log what a typed write outcome means. Landing and coalescing are both success and say nothing. +/// A paired clear that matched no standing fault is the ORDINARY outcome of a successful +/// compaction on a healthy seat, so it is debug and never a warning — and it never falls back to +/// an unkeyed clear, which would silence whatever else stands. Every refusal is fail-open: +/// delivery never depends on a record landing. +fn report_outcome(label: &str, what: &str, outcome: &harness_state::WriteOutcome) { + match outcome.refusal() { + None => {} + Some(harness_state::Refusal::ConditionMismatch { current }) => tracing::debug!( + "st2 {label} channel: {what} matched no standing fault (standing: {current:?})" + ), + Some(refusal) => { + tracing::warn!("st2 {label} channel: {what} was refused: {refusal:?}"); + } + } +} + +/// The one frame worth restating, and only for the one refusal that proves it is safe. +/// +/// A version 3 record's condition axis is not writable as `absent`, so the FIRST activity-only +/// frame of a record whose axis nobody ever stated is refused as +/// [`harness_state::Refusal::Unstated`] — and that refusal is itself the proof that no condition +/// of this session's stands, because a standing one would have been inherited and stated. There +/// is therefore nothing to erase, and `clear` initializes the axis truthfully. Every other +/// refusal is a fact about ownership or a terminal record, which restating cannot help, and a +/// frame that already states a condition is never rewritten. +fn restate_condition( + frame: &harness_state::Frame, + outcome: &harness_state::WriteOutcome, +) -> Option { + if !matches!(outcome.refusal(), Some(harness_state::Refusal::Unstated)) + || !matches!(frame.condition, harness_state::ConditionReport::Unchanged) + { + return None; + } + let mut restated = frame.clone(); + restated.condition = harness_state::ConditionReport::Clear; + Some(restated) +} + +/// Whether one condition edge retires a raise that is still being held. +/// +/// A held raise is a fault the record has not accepted yet, so the ordinary clear paths cannot +/// reach it: it lives in this process. An unkeyed clear retires whatever stands, held included — +/// its proof is a positive observation that the harness made progress. A paired clear retires it +/// only on the EXACT key, the same rule the writer applies to a landed fault, so one signal +/// cannot silence a different held fault. A raise never retires another: replacement is +/// [`apply_condition`]'s business. +fn retires_deferred(edge: &ConditionEdge, held: &harness_state::FaultReport) -> bool { + match edge { + ConditionEdge::ClearAll(_) => true, + ConditionEdge::ClearPaired(key) => *key == held.key(), + ConditionEdge::Raise(_) => false, + } +} + +/// Apply one condition operation, returning a fault the record had nowhere to attach yet. +/// +/// A condition attaches to an OBSERVATION, so a raise that arrives before this session's first +/// frame is handed back to the caller to ride the next one rather than being dropped: st2's own +/// pre-compact failure is exactly that shape. A condition operation never states activity: the +/// writer carries the rest of the tuple — including the pty session that vouches for it — across +/// from this session's own record. +/// +/// A clear that matched nothing is the ORDINARY case and not a problem to report: most +/// compactions never failed, and most turns end with no fault standing. The writer answers with +/// what actually stands and writes nothing, which is the correct outcome. +fn apply_condition( + writer: &mut harness_state::Writer, + edge: ConditionEdge, + label: &str, +) -> Option { + let (what, deferred, outcome) = match edge { + ConditionEdge::Raise(fault) => ( + "a condition", + Some(fault.clone()), + writer.raise_fault(fault), + ), + ConditionEdge::ClearPaired(key) => ("a condition clear", None, writer.clear_fault(key)), + ConditionEdge::ClearAll(proof) => ("a condition clear", None, writer.clear_all(proof)), + }; + match outcome { + Err(error) => { + tracing::warn!("st2 {label} channel: recording {what} failed: {error}"); + None + } + Ok(harness_state::WriteOutcome::Refused(harness_state::Refusal::Unobserved)) => deferred, + Ok(harness_state::WriteOutcome::Refused( + harness_state::Refusal::ConditionMismatch { .. }, + )) => None, + Ok(outcome) => { + report_outcome(label, what, &outcome); + None + } + } +} + /// The observed-state reason a rejected provider credential publishes, shared verbatim with the /// OpenCode, Codex, and Claude producers: one word for one class, whatever named it. const PROVIDER_AUTH_REASON: &str = "providerAuth"; @@ -509,6 +985,248 @@ fn publish_provider_auth( } } +/// The protocol version whose whole content is a PROMISE BY THE ASSET: that it retires a +/// never-answered ask on a turn boundary (a denied ask emits no `tool_result` at all, DQ-OMP-1) +/// and that it forwards omp's own `sessionId`. No frame changes shape; what changes is what st2 is +/// entitled to STATE — a positive `none` on the ask axis, and a linked conversation — so the +/// agreement is per CONNECTION and never per binary: a session replacement re-spawns this channel +/// from a possibly-replaced binary while the loaded asset is the predecessor's, so either version +/// may be on the other end at any time. Offered on the hello beside `protocol`, which stays 1 +/// forever because an asset refuses a hello it does not understand and a refusal costs that seat +/// its mail. +const PROTOCOL_CONDITION_AXIS: u32 = 2; + +/// The asset's answer to the hello's offer, or `None` for every other frame — including an answer +/// naming a version st2 never offered, which is not an agreement but a frame this channel drops +/// like any other it cannot vouch for. +fn negotiated_protocol(frame: &Value) -> Option { + if frame.get("type").and_then(Value::as_str) != Some("client_hello") { + return None; + } + let answered = frame.get("protocol").and_then(Value::as_u64)?; + (answered == u64::from(PROTOCOL_CONDITION_AXIS)).then_some(PROTOCOL_CONDITION_AXIS) +} + +/// omp's fault codes: open, provider-namespaced, and diagnostic granularity UNDERNEATH the closed +/// category beside them — one code per measured class, so a reader can tell an exhausted +/// allowance from a refused key without reading prose, and no consumer has to. +mod omp_fault { + pub const AUTH_FAILED: &str = "omp/authFailed"; + pub const USAGE_LIMIT: &str = "omp/usageLimit"; + pub const ACCOUNT_POLICY: &str = "omp/accountPolicy"; + pub const TRANSIENT_EXHAUSTED: &str = "omp/transientExhausted"; + pub const PROVIDER_ERROR: &str = "omp/providerError"; + pub const UNCLASSIFIED: &str = "omp/unclassified"; + /// The one fault this ADAPTER owns rather than observes: st2's own last-resort pre-compact + /// checkpoint could not be written. Nothing about omp is wrong; st2's plumbing is. + pub const PRE_COMPACT_WRITE_FAILED: &str = "omp/preCompactContextWriteFailed"; +} + +/// The condition axis of one typed turn result, over the classifications measured on omp 18.1.7 +/// (`docs/vrs/06-omp-driver/.experiments/2026-09-05-omp-provider-credential-rejection.md`). +/// +/// Three decisions this encodes, none of which may be re-litigated silently: +/// +/// * `qe.Class` gates everything. Without it the same field carries a BARE HTTP STATUS, so a bit +/// test that skipped it would be reading digits — and reading digits is how a 403 about credits +/// becomes a refused credential. +/// * The negative flags outrank `AuthFailed`, in the order they were measured co-occurring with +/// it: an exhausted allowance is `quota`, an org or content refusal is `policy`, a throttle that +/// reached a turn END is `rateLimit`. Only `Class + AuthFailed` alone is `authentication`, which +/// is exactly [`provider_credential_rejected`] — the same rule, stated once for two axes. +/// * `Recovery::Unknown`, never `Automatic`, for the throttled and unclassified rows. The turn +/// frame carries no deadline, and an automatic fault without a `nextObservationDueMs` can never +/// escalate; `Unknown` is documented as never optimistic, so it pages. +/// +/// `UsageLimit` deliberately does not split `quota` from `account`: the 402 "insufficient balance" +/// and the 403 "out of credits" carry the SAME flag, and separating them would require reading +/// omp's prose (OMP-R06, OHS-R16 forbid it). +/// +/// No `detail` is attached. omp's own words already ride the record's `reason` verbatim, exactly +/// as they do today, and duplicating them into the fault would make its semantic clock restart +/// every time the provider reworded the same condition. +fn turn_fault(result: &TurnResult<'_>, observed_at_ms: u64) -> Option { + let TurnResult::ProviderError { classification, .. } = result else { + return None; + }; + use harness_state::{FaultCategory as Category, Recovery}; + let (category, code, recovery) = match classification { + Some(id) if id & omp_error::CLASSIFIED != 0 => { + let id = *id; + if id & omp_error::USAGE_LIMIT != 0 { + (Category::Quota, omp_fault::USAGE_LIMIT, Recovery::Human) + } else if id & omp_error::ACCOUNT_POLICY != 0 { + (Category::Policy, omp_fault::ACCOUNT_POLICY, Recovery::Human) + } else if id & omp_error::TRANSIENT != 0 { + ( + Category::RateLimit, + omp_fault::TRANSIENT_EXHAUSTED, + Recovery::Unknown, + ) + } else if id & omp_error::AUTH_FAILED != 0 { + ( + Category::Authentication, + omp_fault::AUTH_FAILED, + Recovery::Human, + ) + } else { + ( + Category::Provider, + omp_fault::PROVIDER_ERROR, + Recovery::Unknown, + ) + } + } + // A classification this reader cannot see is still a fault, and it stays VISIBLE: the + // turn died between omp and the provider, which is what `provider` says, and the code + // says st2 could not narrow it. `harness` would claim st2's own plumbing broke — a + // different and untrue statement — and `clear` would launder a wedged seat. + _ => ( + Category::Provider, + omp_fault::UNCLASSIFIED, + Recovery::Unknown, + ), + }; + Some(harness_state::FaultReport::new(category, recovery, observed_at_ms).with_code(code)) +} + +/// The version 2 projection of one observation: exactly the bytes this channel wrote before the +/// negotiated vocabulary existed. +/// +/// The only new word a negotiated asset puts on an UNBLOCKED state frame is the approval-denial +/// diagnostic, and version 2 has no ask axis that makes it meaningful, so it is withheld here +/// rather than appearing as a novel `reason`. Every reason the version 2 wire already carried is +/// untouched: a blocked frame's ask prose, a turn error's own words, and the pre-compact recovery +/// reason all ride through verbatim. +fn legacy_observation( + mut observation: harness_state::Observation, +) -> harness_state::Observation { + if observation.blocked_on != harness_state::BlockedOn::Human + && observation.reason.as_deref() == Some(APPROVAL_DENIED_REASON) + { + observation.reason = None; + } + observation +} + +/// What one pre-compact edge does to the adapter-owned fault: a failed stub write raises it, and +/// a SUCCESSFUL one is the only thing that retires it. No amount of provider-side progress can: +/// the two facts are unrelated, which is why the retirement is a paired clear naming this exact +/// code rather than anything blanket. +fn pre_compact_edge(succeeded: bool, observed_at_ms: u64) -> ConditionEdge { + if succeeded { + ConditionEdge::ClearPaired(pre_compact_fault_key()) + } else { + ConditionEdge::Raise(pre_compact_fault(observed_at_ms)) + } +} + +/// What an ordinary turn end states about the condition axis. +/// +/// A completed turn proves the PROVIDER accepted the credential and did the work, which is what +/// authorizes clearing a fault nobody watched resolve. It proves nothing whatsoever about st2's +/// own pre-compact write, so an adapter-owned fault that still holds is RESTATED instead of being +/// swept up by the blanket clear — and restating the same fault preserves the instant it was +/// first observed, so its semantic clock survives every turn that runs underneath it. +fn turn_completed_edge(standing: Option<&harness_state::FaultReport>) -> ConditionEdge { + match standing { + Some(fault) => ConditionEdge::Raise(fault.clone()), + None => ConditionEdge::ClearAll(harness_state::ProgressProof::TurnCompleted), + } +} + +/// The tuple as it may be stated on THIS connection. +/// +/// The condition axis rides the typed `turn` frame, which every protocol version sends +/// identically, so a fault is stated either way: a wedged seat must stay visible whatever the +/// asset negotiated. The conversation axis rests on a promise only a negotiated asset made — to +/// forward omp's own session id — so no wire-evidenced link is claimed without it, while a +/// kind-level capability claim (pi's `unsupported`) is a fact about the driver and stands on +/// every connection. +/// +/// The ask axis is downgraded in exactly ONE direction: a positive `none` becomes `unknown`, +/// because the promise that makes absence provable — retiring a never-answered ask on a turn +/// boundary — was never made, so an un-negotiated asset's `none` could be a denied ask nobody +/// retired. A PENDING ask is preserved verbatim, kind and all: the legacy `blockedOn`/`ask` pair +/// rides the same frame under both protocols, so a waiting human is equally proven either way, +/// and a frame blocked without a nameable kind is already `Pending(Unknown)` — a human is +/// waiting and the kind is unstated. Downgrading a pending ask to `unknown` would hide the one +/// thing this axis exists to surface, and dropping it would be worse still. +fn connection_frame( + kind: &ChannelKind, + observation: harness_state::Observation, + negotiated: bool, + conversation: Option, +) -> harness_state::Frame { + let mut frame = kind_frame(kind, observation); + if !negotiated { + frame.ask = match frame.ask { + harness_state::HumanAsk::None => harness_state::HumanAsk::Unknown, + pending => pending, + }; + return frame; + } + if let Some(claim) = conversation { + frame = frame.with_conversation(claim); + } + frame +} + +/// st2's own pre-compact recovery write failed: `harness`, because the harness plumbing is what +/// broke, and `human`, because nothing retries it — the next compaction edge is the only thing +/// that can prove it works again. +fn pre_compact_fault(observed_at_ms: u64) -> harness_state::FaultReport { + harness_state::FaultReport::new( + harness_state::FaultCategory::Harness, + harness_state::Recovery::Human, + observed_at_ms, + ) + .with_code(omp_fault::PRE_COMPACT_WRITE_FAILED) +} + +/// The EXACT pairing key the recovered edge clears: category and full code. Never the category +/// alone and never a blanket clear — a standing `authentication`/`omp/authFailed` from a failed +/// turn must survive a compaction that went fine, and a mismatch is answered with +/// [`harness_state::Refusal::ConditionMismatch`] and no write at all, which is the ordinary case +/// here because most compactions never failed in the first place. +fn pre_compact_fault_key() -> harness_state::FaultKey { + harness_state::FaultKey::new(harness_state::FaultCategory::Harness) + .with_code(omp_fault::PRE_COMPACT_WRITE_FAILED) +} + +/// omp's own conversation identity, off the `{"type":"conversation"}` frame a negotiated asset +/// sends the first time an event exposes one. +/// +/// The evidence is `sessionId`, measured on both halves of the approval pair and identical across +/// it (18.0.9 and 18.1.2). `Probed` because it was read off a live event rather than declared +/// from typings, and `Rewritable` because omp compacts its own session store — a prefix read once +/// may be gone. Before any frame arrives the axis is OMITTED, never `Unsupported`: omp +/// demonstrably has sessions (`--no-session`, `sessionManager`), so claiming it has none would be +/// a false capability claim, while saying nothing claims nothing. +fn conversation_claim( + frame: &Value, + verified_through_ms: u64, +) -> Option { + if frame.get("type").and_then(Value::as_str) != Some("conversation") { + return None; + } + let conversation = frame.get("sessionId").and_then(Value::as_str)?.trim(); + // A link with no positive verification bound is refused at the write boundary, so an + // unstampable observation is no observation. + if conversation.is_empty() || verified_through_ms == 0 { + return None; + } + Some(harness_state::ConversationState::Linked( + harness_state::ConversationClaim { + driver: OMP_KIND.label.to_string(), + conversation: conversation.to_owned(), + history_mutability: harness_state::HistoryMutability::Rewritable, + capability_evidence: harness_state::CapabilityEvidence::Probed, + verified_through_ms, + }, + )) +} + /// Write the recovery stub only when durable working state is absent or whitespace-only. /// /// The extension cannot perform this check: it owns neither the resolved agent directory nor the @@ -768,6 +1486,11 @@ mod tests { /// A pre-compaction edge creates a last-resort checkpoint only for whitespace-only state. The /// channel, not the TypeScript extension, resolves the durable path and performs the write. + /// + /// The record assertion at the end is the VERSION 2 projection of a failed stub write — an + /// `active` frame carrying the recovery reason — so the loop runs on a version 2 writer here. + /// Version 3 states the same failure as an adapter-owned fault instead, which is proved by + /// `the_pre_compact_fault_is_cleared_by_category_and_full_code` and the omp condition table. #[test] fn pre_compact_frame_writes_only_over_blank_context() { let tmp = tempfile::tempdir().unwrap(); @@ -779,7 +1502,8 @@ mod tests { let run_frame = || { let mut writer = - harness_state::Writer::new(agent_dir, "h.worker", "omp", Some("h.worker".into())); + harness_state::Writer::new(agent_dir, "h.worker", "omp", Some("h.worker".into())) + .with_emitted_schema(harness_state::SCHEMA_V2); let (tx, rx) = mpsc::channel(); tx.send(Ok(r#"{"type":"pre_compact"}"#.to_string())) .unwrap(); @@ -846,10 +1570,7 @@ mod tests { tx.send(Ok(r#"{"type":"state","state":"active"}"#.to_string())) .unwrap(); - let disconnect = thread::spawn(move || { - thread::sleep(Duration::from_millis(40)); - drop(tx); - }); + drop(tx); let mut out = Vec::new(); channel_loop( &rx, @@ -861,29 +1582,26 @@ mod tests { "h.worker", &PI_KIND, Duration::from_millis(2), - Duration::from_millis(5), + Duration::ZERO, ) .unwrap(); - disconnect.join().unwrap(); - let raw: Value = serde_json::from_slice(&std::fs::read(&record).unwrap()).unwrap(); assert_eq!(raw["state"], "active", "EOF must not rewrite the state"); assert!( raw["writtenAtMs"].as_u64().unwrap() > raw["sinceMs"].as_u64().unwrap(), - "a heartbeat re-stamped the record while the connection lived: {raw}" + "no heartbeat re-stamped the record while the connection lived: {raw}" ); - let after_eof = std::fs::read(&record).unwrap(); - thread::sleep(Duration::from_millis(15)); - assert_eq!( - std::fs::read(&record).unwrap(), - after_eof, - "nothing may write after the connection is gone" + assert!( + raw.get("exit").is_none(), + "EOF must not write a terminal record: {raw}" ); } /// The wrapper's terminal record is the incarnation's last word: a live frame the extension - /// queued before dying must not resurrect the session after the wrapper reaped it. + /// queued before dying must not resurrect the session after the wrapper reaped it. That holds + /// for every axis — a queued condition edge is refused by the same terminal fence, and the + /// record's bytes are unchanged by the whole queue. #[test] fn a_queued_live_frame_never_overwrites_the_wrappers_terminal_record() { let tmp = tempfile::tempdir().unwrap(); @@ -899,12 +1617,25 @@ mod tests { let mut wrapper_writer = harness_state::Writer::new(agent_dir, "h.worker", "pi", Some("h.worker".into())) .with_session(session); - wrapper_writer.ended("signal 9").unwrap(); + // Exactly what `src/pi_session.rs` does, including the one bootstrap retry version 3 + // needs: this wrapper observed nothing, so the axis is unstated until it states it. + crate::provider_session::write_terminal( + &mut wrapper_writer, + "signal 9", + None, + harness_state::ConditionReport::Clear, + ) + .unwrap(); let terminal = std::fs::read(&record).unwrap(); let (tx, rx) = mpsc::channel(); - tx.send(Ok(r#"{"type":"state","state":"idle"}"#.to_string())) - .unwrap(); + for frame in [ + r#"{"type":"state","state":"idle"}"#, + r#"{"type":"condition","op":"raise","category":"harness","code":"pi/assistantError","recovery":"unknown","detail":"401 Unauthorized"}"#, + r#"{"type":"condition","op":"clearAll","proof":"turnCompleted"}"#, + ] { + tx.send(Ok(frame.to_string())).unwrap(); + } drop(tx); let mut out = Vec::new(); channel_loop( @@ -942,7 +1673,9 @@ mod tests { .expect("a record must have been written") } - /// HC-R13, pinned to pi 0.84.2. The payload is verbatim from the credential-free pi lab: one + /// HC-R13, pinned to pi 0.84.4. The payload is verbatim from the credential-free pi lab + /// (captured on 0.84.2, and carried forward because `getContextUsage()` and + /// `calculateContextTokens` are byte-identical in the 0.84.4 tarball): one /// `message_end` for an assistant message, with `getContextUsage()` and the message's own /// `usage` side by side. /// @@ -954,11 +1687,11 @@ mod tests { /// pi reports a float that runs far above 100 when a turn overruns the window, and a producer /// or reader that clamped it would hide exactly the saturation this record exists to show. #[test] - fn the_pi_0_84_2_fixture_pins_total_tokens_as_the_numerator() { + fn the_pinned_pi_fixture_pins_total_tokens_as_the_numerator() { const MEASURED: &str = crate::pi_session::MEASURED_CONTEXT_VERSION; assert_eq!( - MEASURED, "0.84.2", - "the fixture below was captured on this build" + MEASURED, "0.84.4", + "the fixture below is pinned to the build the flake type-checks against" ); // Verbatim `event.message.usage` from the same event, carried so the assertion below can // name the number this producer must NOT publish. @@ -1546,5 +2279,1427 @@ mod tests { driver_diagnostic::read(&driver_diagnostic::path(agent_dir)), driver_diagnostic::Observed::Absent ); + // And the fault axis draws the same line. pi's error-ended turn arrives as prose that + // says `401`, and the category it decodes to is `harness` — the honest reading of "a pi + // turn failed" — never `authentication`, which would be inferred from that very string. + let raise = condition_frame( + &json!({"type":"condition","op":"raise","category":"harness", + "code":"pi/assistantError","recovery":"unknown", + "detail":"401 Unauthorized: invalid x-api-key"}), + 1_000, + ) + .unwrap(); + let ConditionEdge::Raise(fault) = &raise else { + panic!("an error tail raises a fault: {raise:?}"); + }; + assert_eq!(fault.category, harness_state::FaultCategory::Harness); + assert_ne!( + fault.category, + harness_state::FaultCategory::Authentication, + "pi has no error-classification field, so a credential verdict here would be prose" + ); + assert_eq!( + fault.detail.as_deref(), + Some("401 Unauthorized: invalid x-api-key"), + "the prose is carried, diagnostically, and nothing branches on it" + ); + } + + /// The measured false idle, and its fix. `agent_end` raises before `agent_settled` reports the + /// idle, and an activity edge carries the condition axis forward `Unchanged`, so the record + /// settles as idle BESIDE a standing fault instead of laundering a wedged seat into a clean + /// yield. + #[test] + fn a_failed_pi_turn_publishes_idle_beside_a_standing_fault() { + let raise = condition_frame( + &json!({"type":"condition","op":"raise","category":"harness", + "code":"pi/assistantError","recovery":"unknown","detail":"provider said no"}), + 4_200, + ) + .unwrap(); + let ConditionEdge::Raise(fault) = &raise else { + panic!("an error tail raises a fault: {raise:?}"); + }; + assert_eq!(fault.category, harness_state::FaultCategory::Harness); + assert_eq!(fault.code.as_deref(), Some("pi/assistantError")); + assert_eq!(fault.recovery, harness_state::Recovery::Unknown); + assert_ne!( + fault.recovery, + harness_state::Recovery::Automatic, + "an unsayable recovery must page, never wait for a deadline nobody set" + ); + assert_eq!( + fault.next_observation_due_ms, None, + "pi declares no deadline, and st2 never synthesizes one" + ); + // The semantic clock is Rust's, stamped when the frame was read: the asset and this + // process share no clock. + assert_eq!(fault.observed_at_ms, 4_200); + + // The idle that follows. It states activity and nothing else. + let settled = kind_frame( + &PI_KIND, + state_observation(&json!({"type":"state","state":"idle"})).unwrap(), + ); + assert_eq!(settled.state, harness_state::Activity::Idle); + assert_eq!( + settled.condition, + harness_state::ConditionReport::Unchanged, + "the idle must carry the standing fault forward, not clear it" + ); + assert_eq!(settled.ask, harness_state::HumanAsk::Unknown); + assert_eq!( + settled.conversation, + Some(harness_state::ConversationState::Unsupported) + ); + assert_eq!( + settled.reason, None, + "the prose belongs to the fault's detail, not to the activity axis" + ); + } + + /// A version 3 record's condition axis is not writable as `absent`, so the first activity-only + /// frame of an incarnation — pi's `session_start` seed — is refused as `Unstated`. That + /// refusal is its own evidence (a standing fault of this session's would have STATED the + /// axis), so the frame is restated once as `clear` and nothing else about it moves. Every + /// other outcome is left exactly alone: a retry there would argue with the record instead of + /// observing the harness. + #[test] + fn a_fresh_seats_first_frame_states_the_condition_axis_once() { + let seed = kind_frame( + &PI_KIND, + state_observation(&json!({"type":"state","state":"idle"})).unwrap(), + ); + assert_eq!(seed.condition, harness_state::ConditionReport::Unchanged); + + let restated = restate_condition( + &seed, + &harness_state::WriteOutcome::Refused(harness_state::Refusal::Unstated), + ) + .expect("an unstated axis must be stated once"); + assert_eq!(restated.condition, harness_state::ConditionReport::Clear); + assert_eq!(restated.state, seed.state, "the activity axis is unchanged"); + assert_eq!(restated.ask, seed.ask); + assert_eq!(restated.conversation, seed.conversation); + assert_eq!(restated.exit, None, "restating never terminates a record"); + + for outcome in [ + harness_state::WriteOutcome::Landed, + harness_state::WriteOutcome::Coalesced, + harness_state::WriteOutcome::Refused(harness_state::Refusal::Terminal), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Unobserved), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Unfenced), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Superseded { + on_disk_seq: 4, + ours: 3, + }), + harness_state::WriteOutcome::Refused(harness_state::Refusal::ConditionMismatch { + current: None, + }), + ] { + assert_eq!( + restate_condition(&seed, &outcome), + None, + "outcome: {outcome:?}" + ); + } + } + + /// An activity edge has learned nothing about the provider, so it may not clear a condition — + /// in either direction, and however many of them arrive. + #[test] + fn an_activity_edge_never_clears_a_standing_pi_fault() { + for word in ["active", "idle"] { + let frame = json!({"type":"state","state":word}); + let observation = state_observation(&frame).unwrap(); + assert_eq!( + kind_frame(&PI_KIND, observation).condition, + harness_state::ConditionReport::Unchanged, + "state:{word} must not state the condition axis" + ); + assert_eq!( + condition_frame(&frame, 1_000), + None, + "a state frame is not a condition edge" + ); + } + // Nor is the numeric axis: a compaction reading says nothing about a fault either. + assert_eq!( + condition_frame( + &json!({"type":"context","reading":{"usedTokens":10,"usedPercent":1.0}}), + 1_000 + ), + None + ); + } + + /// pi's one positive success edge. An unkeyed clear must name the progress it witnessed, so a + /// `clearAll` whose proof word is missing or unrecognized is dropped rather than believed: + /// that frame is the only operation that can silence a fault nobody saw resolve. + #[test] + fn a_clean_agent_end_is_pis_only_clear_all() { + assert_eq!( + condition_frame(&json!({"type":"condition","op":"clearAll","proof":"turnCompleted"}), 1), + Some(ConditionEdge::ClearAll( + harness_state::ProgressProof::TurnCompleted + )) + ); + for frame in [ + json!({"type":"condition","op":"clearAll"}), + json!({"type":"condition","op":"clearAll","proof":"probably"}), + json!({"type":"condition","op":"clearAll","proof":"assumed"}), + json!({"type":"condition","op":"clearAll","proof":null}), + json!({"type":"condition","op":"settled","proof":"turnCompleted"}), + ] { + assert_eq!(condition_frame(&frame, 1), None, "frame: {frame}"); + } + } + + /// CX-1, pinned for pi: a paired clear names the EXACT `(category, code)` pair. The compaction + /// success edge clears its own failure and nothing else — a category-only key, or one carrying + /// a sibling's code, is a different key and the writer refuses it as a mismatch. + #[test] + fn a_compaction_failure_is_cleared_only_by_its_own_key() { + let raised = condition_frame( + &json!({"type":"condition","op":"raise","category":"context", + "code":"pi/session_compact_failed","recovery":"human"}), + 9_000, + ) + .unwrap(); + let ConditionEdge::Raise(fault) = &raised else { + panic!("a compaction failure raises a fault: {raised:?}"); + }; + assert_eq!(fault.category, harness_state::FaultCategory::Context); + assert_eq!(fault.recovery, harness_state::Recovery::Human); + assert_eq!(fault.detail, None, "the typed event needs no prose"); + + let cleared = condition_frame( + &json!({"type":"condition","op":"clear","category":"context", + "code":"pi/session_compact_failed"}), + 9_100, + ) + .unwrap(); + assert_eq!(cleared, ConditionEdge::ClearPaired(fault.key())); + let ConditionEdge::ClearPaired(key) = cleared else { + panic!("a compaction success is a paired clear"); + }; + // The two keys a lazier producer would have sent instead. Neither matches the fault, so + // neither can silence it. + assert_ne!( + key, + harness_state::FaultKey::new(harness_state::FaultCategory::Context), + "a category-only clear would wipe any other context fault" + ); + assert_ne!( + key, + harness_state::FaultKey::new(harness_state::FaultCategory::Context) + .with_code("pi/session_compact") + ); + // A code that is not provider-namespaced is a malformed frame, not a bare-word key: two + // providers' codes collide exactly there. + assert_eq!( + condition_frame( + &json!({"type":"condition","op":"clear","category":"context", + "code":"session_compact_failed"}), + 9_100 + ), + None + ); + // And a category this build cannot name drops the frame rather than becoming untyped. + assert_eq!( + condition_frame( + &json!({"type":"condition","op":"raise","category":"vibes", + "recovery":"human"}), + 9_100 + ), + None + ); + // As does a recovery word it cannot name: the urgency axis is what routes the fault. + assert_eq!( + condition_frame( + &json!({"type":"condition","op":"raise","category":"context", + "recovery":"eventually"}), + 9_100 + ), + None + ); + } + + /// A raise the record could not accept yet lives in this process, so the ordinary clear paths + /// cannot reach it. An unkeyed clear and its own paired key must retire it anyway, or the held + /// fault outlives the very edge that resolved it and the next activity edge republishes a + /// condition the harness already retracted. + #[test] + fn a_clear_retires_a_raise_that_has_not_landed_yet() { + let held = harness_state::FaultReport::new( + harness_state::FaultCategory::Context, + harness_state::Recovery::Human, + 5_000, + ) + .with_code("pi/session_compact_failed"); + + assert!(retires_deferred( + &ConditionEdge::ClearAll(harness_state::ProgressProof::TurnCompleted), + &held + )); + assert!(retires_deferred( + &ConditionEdge::ClearPaired(held.key()), + &held + )); + // A different key is a different fault: it may not silence this one, held or standing. + assert!(!retires_deferred( + &ConditionEdge::ClearPaired(harness_state::FaultKey::new( + harness_state::FaultCategory::Context + )), + &held + )); + assert!(!retires_deferred( + &ConditionEdge::ClearPaired( + harness_state::FaultKey::new(harness_state::FaultCategory::Harness) + .with_code("pi/assistantError") + ), + &held + )); + // Replacing one raise with another is the writer's business, not a retirement. + assert!(!retires_deferred( + &ConditionEdge::Raise( + harness_state::FaultReport::new( + harness_state::FaultCategory::Harness, + harness_state::Recovery::Unknown, + 6_000, + ) + .with_code("pi/assistantError") + ), + &held + )); + } + + /// pi has no ask surface at all, so every pi frame states `unknown` — positively, forever — + /// and derives a pending ask from nothing, not even from a foreign asset's claim of one. A + /// fabricated `pending` injects a question into the ask queue that nobody can answer. + #[test] + fn pi_frames_never_claim_an_ask() { + assert_eq!(PI_KIND.default_ask, harness_state::HumanAsk::Unknown); + assert_eq!(OMP_KIND.default_ask, harness_state::HumanAsk::None); + + let foreign = json!({"type":"state","state":"active","blockedOn":"human", + "ask":"permission"}); + let observation = state_observation(&foreign).unwrap(); + assert_eq!( + kind_frame(&PI_KIND, observation.clone()).ask, + harness_state::HumanAsk::Unknown, + "a channel that cannot see the axis cannot vouch for somebody else's claim about it" + ); + assert_eq!( + kind_frame(&PI_KIND, observation.clone()).ask.pending(), + None + ); + // The same frame on a kind that DOES see the axis reads it, which is what makes the pi + // answer a statement about pi rather than a decoder that lost the field. + assert_eq!( + kind_frame(&OMP_KIND, observation).ask, + harness_state::HumanAsk::Pending(harness_state::AskKind::Permission) + ); + assert_eq!( + kind_frame( + &PI_KIND, + state_observation(&json!({"type":"state","state":"idle"})).unwrap() + ) + .ask, + harness_state::HumanAsk::Unknown + ); + } + + /// The channel writes no terminal record on any axis. `src/pi_session.rs` — the outer wrapper, + /// the only process that sees the provider die — owns that word, so neither an `ended` state + /// word nor a fault claiming terminal recovery may produce one. + #[test] + fn no_pi_frame_can_write_ended() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path(); + std::fs::create_dir_all(message::inbox_dir(agent_dir)).unwrap(); + let mut writer = + harness_state::Writer::new(agent_dir, "h.worker", "pi", Some("h.worker".into())); + let (tx, rx) = mpsc::channel(); + for frame in [ + r#"{"type":"state","state":"active"}"#, + r#"{"type":"state","state":"ended","exit":"signal 9"}"#, + r#"{"type":"condition","op":"raise","category":"harness","code":"pi/assistantError","recovery":"terminal"}"#, + ] { + tx.send(Ok(frame.to_string())).unwrap(); + } + drop(tx); + channel_loop( + &rx, + &mut Vec::new(), + &message::inbox_dir(agent_dir), + agent_dir, + &mut writer, + None, + "h.worker", + &PI_KIND, + Duration::from_millis(1), + Duration::from_secs(60), + ) + .unwrap(); + + let raw: Value = serde_json::from_slice( + &std::fs::read(harness_state::harness_state_path(agent_dir)).unwrap(), + ) + .unwrap(); + assert_eq!(raw["state"], "active", "no channel frame may terminate a session"); + assert!(raw["exit"].is_null()); + // The decoder itself has no terminal operation: `terminal` is a recovery word — nothing + // clears this fault for this incarnation — and never a session outcome. + let edge = condition_frame( + &json!({"type":"condition","op":"raise","category":"harness", + "code":"pi/assistantError","recovery":"terminal"}), + 1_000, + ) + .unwrap(); + assert!(matches!(edge, ConditionEdge::Raise(_))); + } + + /// Protocol 1 acceptance, which is what keeps every already-published hook set delivering + /// mail: the pinned asset compares `protocol` for strict equality and closes the channel on a + /// mismatch, so the floor stays 1 and the newer wire is offered beside it. + #[test] + fn protocol_one_assets_still_accept_the_hello() { + let hello = hello("h.worker", "restored"); + assert_eq!(hello["protocol"], json!(1)); + assert_eq!(PROTOCOL, 1); + assert_eq!(hello["protocols"], json!([1, 2])); + assert_eq!(PROTOCOLS, [1, 2]); + assert!( + PROTOCOLS.contains(&PROTOCOL), + "the floor must be one of the offered versions" + ); + assert_eq!(hello["identity"], json!("h.worker")); + assert_eq!(hello["sessionContext"], json!("restored")); + } + + /// On a writer that emits version 2 the condition axis has nowhere to live, and this record + /// has exactly one source of truth — so a condition edge is dropped, not cached, and the + /// legacy bytes are exactly the ones this channel already wrote. Production emits version 3 + /// now; this is the rollback and already-on-disk shape, reached through the test-only seam. + #[test] + fn condition_frames_change_nothing_while_the_record_carries_no_condition_axis() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path(); + std::fs::create_dir_all(message::inbox_dir(agent_dir)).unwrap(); + let record = harness_state::harness_state_path(agent_dir); + let run = |frames: &[&str], writer: &mut harness_state::Writer| { + let (tx, rx) = mpsc::channel(); + for frame in frames { + tx.send(Ok((*frame).to_string())).unwrap(); + } + drop(tx); + channel_loop( + &rx, + &mut Vec::new(), + &message::inbox_dir(agent_dir), + agent_dir, + writer, + None, + "h.worker", + &PI_KIND, + Duration::from_millis(1), + Duration::from_secs(60), + ) + .unwrap(); + }; + let mut writer = + harness_state::Writer::new(agent_dir, "h.worker", "pi", Some("h.worker".into())) + .with_emitted_schema(harness_state::SCHEMA_V2); + assert!( + !writer.writes_condition_axis(), + "the branch under test is the legacy one, reached through the version 2 seam" + ); + run(&[r#"{"type":"state","state":"idle"}"#], &mut writer); + let legacy = std::fs::read(&record).unwrap(); + + run( + &[ + r#"{"type":"condition","op":"raise","category":"harness","code":"pi/assistantError","recovery":"unknown","detail":"provider said no"}"#, + r#"{"type":"condition","op":"clear","category":"context","code":"pi/session_compact_failed"}"#, + r#"{"type":"condition","op":"clearAll","proof":"turnCompleted"}"#, + ], + &mut writer, + ); + assert_eq!( + std::fs::read(&record).unwrap(), + legacy, + "a condition edge may not touch the version 2 record" + ); + let raw: Value = serde_json::from_slice(&legacy).unwrap(); + assert_eq!(raw["state"], "idle"); + assert!(raw["condition"].is_null()); + assert!(raw["blockedOn"] == json!("none") || raw["blockedOn"].is_null()); + } + + /// One omp channel loop over a fixed incarnation, so two runs are comparable byte for byte. + fn omp_record(frames: &[&str]) -> Value { + omp_record_emitting(frames, harness_state::SCHEMA_V3) + } + + /// One omp channel loop over a fixed incarnation and an explicit emitted version, so two runs + /// are comparable byte for byte and the version 2 projection stays provable after activation. + fn omp_record_emitting(frames: &[&str], schema: &'static str) -> Value { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path(); + let inbox = message::inbox_dir(agent_dir); + std::fs::create_dir_all(&inbox).unwrap(); + let mut writer = + harness_state::Writer::new(agent_dir, "h.worker", "omp", Some("h.worker".into())) + .with_emitted_schema(schema) + .with_ownership("session-1", 1); + let (tx, rx) = mpsc::channel(); + for frame in frames { + tx.send(Ok((*frame).to_string())).unwrap(); + } + drop(tx); + channel_loop( + &rx, + &mut Vec::new(), + &inbox, + agent_dir, + &mut writer, + None, + "h.worker", + &OMP_KIND, + Duration::from_millis(1), + Duration::from_secs(60), + ) + .unwrap(); + let mut record: Value = serde_json::from_slice( + &std::fs::read(harness_state::harness_state_path(agent_dir)).unwrap(), + ) + .unwrap(); + // The two clocks are the only fields two identical runs may legitimately differ on. + for volatile in ["writtenAtMs", "sinceMs"] { + record.as_object_mut().unwrap().remove(volatile); + } + record + } + + /// The measured omp 18.1.7 table once more, now as the version 3 condition tuple: exactly one + /// (category, code, recovery) per class, beside the credential axis it must not disturb. + /// + /// The two rows that motivated this whole mapping are asserted by name. `0x1081000` (403 "out + /// of credits") is `quota` and `0x100d000` (403 `cyber_policy`) is `policy`, even though BOTH + /// set omp's `AuthFailed` flag: a classifier that stopped at that flag would send the operator + /// of a wedged, fully-paid seat to re-login. + #[test] + fn every_measured_omp_classification_maps_to_one_v3_condition() { + use harness_state::{FaultCategory as Category, Recovery}; + let observed_at_ms = 1_787_999_000_000; + // (case, errorId, category, code, recovery, is a rejected credential) + let cases = [ + ( + "401 invalid x-api-key", + 0x100_1000_u64, + Category::Authentication, + "omp/authFailed", + Recovery::Human, + true, + ), + ( + "401 OAuth invalid_grant", + 0x100_1000, + Category::Authentication, + "omp/authFailed", + Recovery::Human, + true, + ), + ( + "403 key lacks permission", + 0x100_1000, + Category::Authentication, + "omp/authFailed", + Recovery::Human, + true, + ), + ( + "403 run out of credits", + 0x108_1000, + Category::Quota, + "omp/usageLimit", + Recovery::Human, + false, + ), + ( + "402 insufficient balance", + 0x08_1000, + Category::Quota, + "omp/usageLimit", + Recovery::Human, + false, + ), + ( + "403 cyber_policy", + 0x100_d000, + Category::Policy, + "omp/accountPolicy", + Recovery::Human, + false, + ), + ( + "403 CONCURRENT_LIMIT", + 0x102_1000, + Category::RateLimit, + "omp/transientExhausted", + Recovery::Unknown, + false, + ), + ( + "429 rate limit", + 0x02_1000, + Category::RateLimit, + "omp/transientExhausted", + Recovery::Unknown, + false, + ), + // The residual classified row: omp classified it and none of the flags above is set. + // It is the shape of a provider-side failure rather than a captured case, and it must + // stay VISIBLE rather than becoming a credential rejection by elimination. + ( + "500 upstream failure", + 0x00_1000, + Category::Provider, + "omp/providerError", + Recovery::Unknown, + false, + ), + ]; + + for (case, error_id, category, code, recovery, rejected) in cases { + let frame = json!({"type":"turn","error":{"reason":case,"errorId":error_id}}); + let result = turn_result(&frame).expect("a turn frame decodes"); + let fault = turn_fault(&result, observed_at_ms).expect("a failed turn is a condition"); + assert_eq!(fault.category, category, "{case}"); + assert_eq!(fault.code.as_deref(), Some(code), "{case}"); + assert_eq!(fault.recovery, recovery, "{case}"); + assert_eq!( + fault.observed_at_ms, observed_at_ms, + "the SEMANTIC clock is the producer's own observation instant: {case}" + ); + assert_eq!( + fault.next_observation_due_ms, None, + "the turn frame carries no deadline, so no omp fault may claim one: {case}" + ); + assert_eq!( + fault.detail, None, + "omp's prose rides the record's `reason`, never the fault's clock-bearing \ + identity: {case}" + ); + assert_ne!( + fault.recovery, + Recovery::Automatic, + "an automatic recovery with no deadline can never escalate: {case}" + ); + // The credential axis is the SAME rule, stated once for two records, and unchanged. + assert_eq!( + provider_auth_edge(&result), + rejected.then_some(ProviderAuthEdge::Rejected), + "{case}" + ); + assert_eq!( + fault.category == Category::Authentication, + rejected, + "`authentication` and the credential edge are one rule: {case}" + ); + // And the legacy observation is byte-for-byte what it always was. + let observation = turn_observation(&result).expect("a failed turn is an observation"); + assert_eq!(observation.state, harness_state::Activity::Active, "{case}"); + assert_eq!(observation.blocked_on, harness_state::BlockedOn::None, "{case}"); + assert_eq!( + observation.reason.as_deref(), + Some(if rejected { PROVIDER_AUTH_REASON } else { case }), + "{case}" + ); + } + + // A turn that reached its ordinary end states no fault at all. It is the ONLY positive + // success edge, and what it authorizes is the blanket clear — never a fault of its own. + let ordinary_frame = json!({"type": "turn"}); + let ordinary = turn_result(&ordinary_frame).expect("an ordinary end decodes"); + assert!( + turn_fault(&ordinary, observed_at_ms).is_none(), + "an ordinary turn end is progress, not a condition" + ); + assert_eq!( + provider_auth_edge(&ordinary), + Some(ProviderAuthEdge::Accepted) + ); + } + + /// An error omp itself did not classify stays VISIBLE, under the most conservative category + /// that is still true: the turn died between omp and the provider. `harness` would claim st2's + /// own plumbing broke and a `clear` would launder a wedged seat; both are false statements. + #[test] + fn an_unclassified_error_id_is_a_visible_provider_fault_not_a_credential_rejection() { + // A bare HTTP status (no `qe.Class` bit), an absent field, and a field this decoder + // cannot read as a number are all the same thing: no classification. + for unclassified in [json!(403), json!(0), json!(429), Value::Null, json!("403")] { + let frame = json!({"type":"turn","error":{"reason":"403 …","errorId":unclassified}}); + let result = turn_result(&frame).expect("a turn frame decodes"); + let fault = turn_fault(&result, 7).expect("an unreadable class is still a fault"); + assert_eq!(fault.category, harness_state::FaultCategory::Provider, "{unclassified}"); + assert_eq!(fault.code.as_deref(), Some("omp/unclassified"), "{unclassified}"); + assert_eq!(fault.recovery, harness_state::Recovery::Unknown, "{unclassified}"); + assert_eq!( + provider_auth_edge(&result), + None, + "silence about the class is not a verdict on the credential: {unclassified}" + ); + } + } + + /// The axes are independent: an activity edge, an ask, a compaction, an approval denial, the + /// negotiation answer, and the conversation statement all state NOTHING about the condition. + /// A producer forced to pick `clear` on any of them would fabricate health several times a + /// turn, and a retry in flight — which omp reports by sending no frame at all — would be the + /// worst of them. + #[test] + fn the_edges_that_are_not_faults_state_no_condition() { + for frame in [ + json!({"type":"state","state":"active"}), + json!({"type":"state","state":"idle"}), + json!({"type":"state","state":"active","blockedOn":"human","ask":"question","reason":"Which target?"}), + json!({"type":"state","state":"active","blockedOn":"human","ask":"permission","reason":"bash"}), + json!({"type":"state","state":"idle","reason":"approvalDenied"}), + json!({"type":"context","reading":{"usedPercent":42.0}}), + json!({"type":"pre_compact"}), + json!({"type":"client_hello","protocol":2}), + json!({"type":"conversation","sessionId":"2f8c"}), + json!({"type":"delivered","meta":{}}), + ] { + assert!( + turn_result(&frame).is_none(), + "only a turn result carries a condition: {frame}" + ); + } + // A denied approval is an interruption, not a fault: the ask is simply over, the word is + // prose, and the condition axis is untouched. + let denied = + state_observation(&json!({"type":"state","state":"idle","reason":"approvalDenied"})) + .expect("a denial resolves the ask"); + assert_eq!(denied.blocked_on, harness_state::BlockedOn::None); + assert_eq!(denied.ask, harness_state::Ask::None); + assert_eq!(denied.reason.as_deref(), Some("approvalDenied")); + } + + /// The axes are independent in the WRITE, not just in the decode: every activity and ask edge + /// carries the condition forward `Unchanged`, so a standing fault survives a whole turn of + /// traffic. And omp's ask axis is positive: it owns both ask surfaces, so `none` is an + /// observation rather than an absence of one. + #[test] + fn an_activity_edge_carries_the_condition_forward_untouched() { + use harness_state::{AskKind, HumanAsk}; + let rows = [ + (json!({"type":"state","state":"active"}), HumanAsk::None), + (json!({"type":"state","state":"idle"}), HumanAsk::None), + ( + json!({"type":"state","state":"idle","reason":"approvalDenied"}), + HumanAsk::None, + ), + ( + json!({"type":"state","state":"active","blockedOn":"human","ask":"question","reason":"Which target?"}), + HumanAsk::Pending(AskKind::Question), + ), + ( + json!({"type":"state","state":"active","blockedOn":"human","ask":"permission","reason":"bash"}), + HumanAsk::Pending(AskKind::Permission), + ), + ( + json!({"type":"state","state":"active","blockedOn":"human","ask":"sacrifice"}), + HumanAsk::Pending(AskKind::Unknown), + ), + ]; + for (raw, ask) in rows { + let observation = state_observation(&raw).expect("a state frame decodes"); + let published = kind_frame(&OMP_KIND, observation); + assert!( + matches!( + published.condition, + harness_state::ConditionReport::Unchanged + ), + "an activity edge has learned nothing about the provider: {raw}" + ); + assert_eq!(published.ask, ask, "{raw}"); + assert_eq!(published.input_buffer, harness_state::InputBuffer::Unknown); + assert_eq!( + published.conversation, None, + "the axis is stated from wire evidence only: {raw}" + ); + } + + // A failed turn is the one row that states a condition, and it still says `active`: the + // seat needs an operator, and `idle` would read as a healthy yield. + let credits = json!({ + "type":"turn","error":{"reason":"403 run out of credits","errorId":17305600} + }); + let result = turn_result(&credits).unwrap(); + let faulted = kind_frame(&OMP_KIND, turn_observation(&result).unwrap()); + assert_eq!(faulted.state, harness_state::Activity::Active); + assert_eq!(faulted.ask, HumanAsk::None); + assert_eq!( + OMP_KIND.conversation, None, + "omp never states `unsupported`: it demonstrably has sessions" + ); + assert_eq!(OMP_KIND.default_ask, HumanAsk::None); + } + + /// The condition axis is not writable as `absent`, so a virgin version 3 record refuses the + /// first activity-only frame — and ONLY that refusal authorizes restating it as `clear`. Every + /// other refusal is a fact about ownership or a terminal record that a restatement cannot fix, + /// and a frame that already states a condition is never rewritten into one that does not. + #[test] + fn only_an_unstated_axis_is_restated_as_clear() { + let observation = state_observation(&json!({"type":"state","state":"idle"})).unwrap(); + let frame = kind_frame(&OMP_KIND, observation); + let unstated = harness_state::WriteOutcome::Refused(harness_state::Refusal::Unstated); + let restated = restate_condition(&frame, &unstated).expect("an unstated axis is stated"); + assert!(matches!( + restated.condition, + harness_state::ConditionReport::Clear + )); + assert_eq!(restated.state, frame.state, "only the condition axis moves"); + assert_eq!(restated.ask, frame.ask); + assert_eq!(restated.input_buffer, frame.input_buffer); + assert_eq!(restated.conversation, frame.conversation); + assert_eq!(restated.reason, frame.reason); + + for outcome in [ + harness_state::WriteOutcome::Landed, + harness_state::WriteOutcome::Coalesced, + harness_state::WriteOutcome::Refused(harness_state::Refusal::Terminal), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Unobserved), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Unfenced), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Unclaimed), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Unreadable), + harness_state::WriteOutcome::Refused(harness_state::Refusal::ConditionMismatch { + current: None, + }), + harness_state::WriteOutcome::Refused(harness_state::Refusal::Superseded { + on_disk_seq: 4, + ours: 1, + }), + ] { + assert!( + restate_condition(&frame, &outcome).is_none(), + "nothing else authorizes stating the axis: {outcome:?}" + ); + } + + // A frame that already carries a fault is never rewritten into one that clears it. + let mut faulted = frame.clone(); + faulted.condition = + harness_state::ConditionReport::Fault(pre_compact_fault(message::now_ms())); + assert!(restate_condition(&faulted, &unstated).is_none()); + } + + /// Which wire is written is the writer's business; negotiation narrows two axes, not the + /// write. Under version 3 an un-negotiated peer is still published — its faults ride the same + /// typed turn frame, so withholding the tuple would hide a wedged seat — but the ask axis + /// reads `unknown`, because the promise that makes a positive `none` provable (retiring a + /// never-answered ask on a turn boundary) was never made, and no conversation is claimed. + #[test] + fn an_un_negotiated_peer_is_published_without_the_promoted_axes() { + use harness_state::{AskKind, ConversationState, HumanAsk}; + let link = ConversationState::Linked(harness_state::ConversationClaim { + driver: "omp".to_owned(), + conversation: "2f8c-4d11".to_owned(), + history_mutability: harness_state::HistoryMutability::Rewritable, + capability_evidence: harness_state::CapabilityEvidence::Probed, + verified_through_ms: 1_787_999_000_000, + }); + for raw in [ + json!({"type":"state","state":"idle"}), + json!({"type":"state","state":"active"}), + json!({"type":"state","state":"active","blockedOn":"human","ask":"question","reason":"Which target?"}), + ] { + let observation = state_observation(&raw).unwrap(); + let bare = connection_frame(&OMP_KIND, observation.clone(), false, Some(link.clone())); + let legacy_pending = observation.blocked_on == harness_state::BlockedOn::Human; + if legacy_pending { + assert_eq!( + bare.ask, + kind_frame(&OMP_KIND, observation.clone()).ask, + "a waiting human is equally proven on either protocol: {raw}" + ); + assert!(bare.ask.pending().is_some(), "{raw}"); + } else { + assert_eq!( + bare.ask, + HumanAsk::Unknown, + "only a positive `none` is unprovable without the retirement promise: {raw}" + ); + } + assert_eq!( + bare.conversation, None, + "no promise, no claimed conversation: {raw}" + ); + assert!( + matches!(bare.condition, harness_state::ConditionReport::Unchanged), + "{raw}" + ); + assert_eq!( + bare.state, + state_observation(&raw).unwrap().state, + "the activity axis is stated on every connection: {raw}" + ); + + // The same frame from a negotiated asset states both promoted axes. + let promoted = connection_frame(&OMP_KIND, observation, true, Some(link.clone())); + assert_eq!(promoted.conversation, Some(link.clone()), "{raw}"); + assert_ne!(promoted.ask, HumanAsk::Unknown, "{raw}"); + } + assert_eq!( + connection_frame( + &OMP_KIND, + state_observation(&json!({"type":"state","state":"idle"})).unwrap(), + true, + None, + ) + .ask, + HumanAsk::None, + "omp owns both ask surfaces, so a negotiated `none` is a positive observation" + ); + assert_eq!( + connection_frame( + &OMP_KIND, + state_observation( + &json!({"type":"state","state":"active","blockedOn":"human","ask":"permission"}) + ) + .unwrap(), + true, + None, + ) + .ask, + HumanAsk::Pending(AskKind::Permission) + ); + // A blocked frame with no nameable kind is a waiting human whose question is unstated — + // `Pending(Unknown)` — and never a dropped ask. + assert_eq!( + connection_frame( + &OMP_KIND, + state_observation(&json!({"type":"state","state":"active","blockedOn":"human"})) + .unwrap(), + false, + None, + ) + .ask, + HumanAsk::Pending(AskKind::Unknown) + ); + + // A fault is stated on either connection: it rides the typed turn frame, which every + // protocol version sends identically. + let credits = json!({"type":"turn","error":{"reason":"403 run out of credits","errorId":17305600}}); + let result = turn_result(&credits).unwrap(); + assert!(turn_fault(&result, 5).is_some()); + // A driver-level capability claim is not a negotiated axis: pi's `unsupported` is a fact + // about pi and stands on every connection. + assert_eq!( + connection_frame( + &PI_KIND, + state_observation(&json!({"type":"state","state":"idle"})).unwrap(), + false, + None, + ) + .conversation, + Some(ConversationState::Unsupported) + ); + } + + /// The mixed-version seat that motivates the whole downgrade rule: a version 3 record written + /// for an UN-NEGOTIATED asset that is blocked on a human must still summon one. The ask + /// survives the downgrade, so the shared disposition reads `waitingHuman` / `now` / `answer` — + /// the same verdict the legacy projection of that frame produces, which is the property that + /// makes the version 3 rollout invisible to an operator. + #[test] + fn an_un_negotiated_blocked_frame_still_summons_a_human() { + let raw = json!({ + "type":"state","state":"active","blockedOn":"human","ask":"question", + "reason":"Which deployment target?" + }); + let observation = state_observation(&raw).unwrap(); + let published = connection_frame(&OMP_KIND, observation.clone(), false, None); + assert_eq!( + published.ask, + harness_state::HumanAsk::Pending(harness_state::AskKind::Question) + ); + + // The record such a frame projects, read back through the shared disposition. Nothing is + // faulted and no diagnostic stands: the ask alone must carry the verdict. + let observed = harness_state::Observed { + state: published.state, + blocked_on: harness_state::BlockedOn::Human, + input_buffer: published.input_buffer, + ask: harness_state::Ask::Question, + harness: Some("omp".to_owned()), + since_ms: Some(message::now_ms()), + exit: None, + reason: published.reason.clone(), + subject: None, + schema: Some(harness_state::SCHEMA_V3.to_owned()), + indeterminacy: None, + condition: harness_state::ConditionView::Clear, + human_ask: published.ask, + conversation: None, + }; + let disposition = + harness_state::disposition(Some(&observed), &driver_diagnostic::Observed::Absent); + assert_eq!( + disposition.state, + harness_state::DispositionState::WaitingHuman + ); + assert_eq!(disposition.attention, harness_state::Attention::Now); + assert_eq!( + disposition.primary_action, + harness_state::PrimaryAction::Answer + ); + + // Had the downgrade swallowed the pending ask, the same seat would read as merely worth + // observing — nobody would be summoned. + let muted = harness_state::Observed { + human_ask: harness_state::HumanAsk::Unknown, + ..observed + }; + let muted = + harness_state::disposition(Some(&muted), &driver_diagnostic::Observed::Absent); + assert_ne!( + muted.state, + harness_state::DispositionState::WaitingHuman, + "this is the regression the downgrade rule exists to prevent" + ); + } + + /// The adapter-owned fault outlives provider progress. A completed turn is evidence about the + /// PROVIDER; st2's own failed pre-compact write is a different fact, and only the next + /// successful pre-compact edge retires it — by category and full code, never by the blanket + /// clear that a turn authorizes. + #[test] + fn a_completed_turn_never_retires_the_adapter_owned_fault() { + let mut standing: Option = None; + + // With nothing of ours standing, a completed turn clears the whole axis. + assert_eq!( + turn_completed_edge(standing.as_ref()), + ConditionEdge::ClearAll(harness_state::ProgressProof::TurnCompleted) + ); + + // The stub write fails: the fault is raised AND remembered. + let raised = pre_compact_edge(false, 1_787_999_000_000); + let ConditionEdge::Raise(fault) = &raised else { + panic!("a failed stub write raises: {raised:?}") + }; + assert_eq!(fault.key(), pre_compact_fault_key()); + standing = Some(fault.clone()); + + // Two ordinary turn ends later it still stands, restated rather than swept up — and + // restating the same fault is what preserves the instant it was first observed. + for _turn in 0..2 { + assert_eq!( + turn_completed_edge(standing.as_ref()), + ConditionEdge::Raise(fault.clone()), + "a working provider says nothing about st2's own failed write" + ); + } + // Even a provider fault that displaced it in the record's single condition slot does not + // retire it: the next completed turn restates ours rather than clearing everything. + assert_ne!( + turn_completed_edge(standing.as_ref()), + ConditionEdge::ClearAll(harness_state::ProgressProof::TurnCompleted) + ); + + // Only the successful edge retires it, and only by its exact key. + let recovered = pre_compact_edge(true, 1_787_999_100_000); + assert_eq!( + recovered, + ConditionEdge::ClearPaired(pre_compact_fault_key()) + ); + standing = None; + assert_eq!( + turn_completed_edge(standing.as_ref()), + ConditionEdge::ClearAll(harness_state::ProgressProof::TurnCompleted) + ); + } + + /// The negotiated vocabulary is inert on the version 2 wire in BOTH directions: the denial + /// prose a negotiated asset narrates is withheld rather than becoming a novel `reason` on an + /// unblocked frame, while every reason version 2 already carried rides through verbatim. + #[test] + fn the_denial_diagnostic_never_reaches_the_version_two_wire() { + let denial = state_observation(&json!({ + "type":"state","state":"idle","reason":"approvalDenied" + })) + .unwrap(); + assert_eq!(denial.reason.as_deref(), Some(APPROVAL_DENIED_REASON)); + assert_eq!( + legacy_observation(denial).reason, None, + "version 2 has no ask axis that makes this word meaningful" + ); + + // Everything the version 2 wire already said keeps saying it. + for raw in [ + json!({"type":"state","state":"active","blockedOn":"human","ask":"permission","reason":"bash"}), + json!({"type":"state","state":"active","blockedOn":"human","ask":"question","reason":"Which target?"}), + json!({"type":"state","state":"active","blockedOn":"human","ask":"permission","reason":"approvalDenied"}), + ] { + let observation = state_observation(&raw).unwrap(); + let reason = observation.reason.clone(); + assert_eq!( + legacy_observation(observation).reason, + reason, + "a blocked frame's prose is untouched: {raw}" + ); + } + let credits = json!({"type":"turn","error":{"reason":"403 run out of credits","errorId":17305600}}); + let faulted = turn_observation(&turn_result(&credits).unwrap()).unwrap(); + assert_eq!( + legacy_observation(faulted).reason.as_deref(), + Some("403 run out of credits"), + "a turn error's own words are how a reader learns which 4xx it was" + ); + let recovery = harness_state::Observation::new( + harness_state::Activity::Active, + harness_state::BlockedOn::None, + harness_state::InputBuffer::Unknown, + ) + .with_reason(PRE_COMPACT_ERROR_REASON); + assert_eq!( + legacy_observation(recovery).reason.as_deref(), + Some(PRE_COMPACT_ERROR_REASON) + ); + + // And end to end on the version 2 wire: a denial leaves a record indistinguishable from + // the plain idle frame it resolved to, so no reader pinned to version 2 sees a new field. + let v2 = |frames: &[&str]| omp_record_emitting(frames, harness_state::SCHEMA_V2); + assert_eq!( + v2(&[ + r#"{"type":"client_hello","protocol":2}"#, + r#"{"type":"state","state":"idle","reason":"approvalDenied"}"#, + ]), + v2(&[r#"{"type":"state","state":"idle"}"#]), + ); + } + + /// The adapter-owned harness fault, and the exactness of its clear. A compaction whose stub + /// write now succeeds retires THAT fault and only that fault: a standing credential rejection + /// must survive it, so the key names the category AND the full code — a category-wide key + /// would be how one healthy compaction silences a wedged seat. + #[test] + fn the_pre_compact_fault_is_cleared_by_category_and_full_code() { + let fault = pre_compact_fault(9); + assert_eq!(fault.category, harness_state::FaultCategory::Harness); + assert_eq!(fault.recovery, harness_state::Recovery::Human); + assert_eq!( + fault.code.as_deref(), + Some("omp/preCompactContextWriteFailed") + ); + assert_eq!(fault.observed_at_ms, 9); + assert_eq!(fault.next_observation_due_ms, None); + assert_eq!(pre_compact_fault_key(), fault.key()); + assert_ne!( + pre_compact_fault_key(), + harness_state::FaultKey::new(harness_state::FaultCategory::Harness), + "a codeless key matches a codeless fault, which is not this one" + ); + let credential = turn_fault( + &turn_result(&json!({"type":"turn","error":{"errorId":0x100_1000}})).unwrap(), + 9, + ) + .unwrap(); + assert_ne!( + pre_compact_fault_key(), + credential.key(), + "a healthy compaction must not clear a refused credential" + ); + } + + /// The conversation axis is populated from omp's OWN typed evidence — the `sessionId` measured + /// on both halves of the approval pair — and from nothing else. Before one is observed the + /// axis is omitted; it is never `Unsupported`, because omp demonstrably has sessions and + /// claiming otherwise would be a false capability claim. + #[test] + fn a_conversation_is_linked_only_from_typed_session_evidence() { + let state = conversation_claim( + &json!({"type":"conversation","sessionId":" 2f8c-4d11 "}), + 1_787_999_000_000, + ) + .expect("a session id is a link"); + let link = match state { + harness_state::ConversationState::Linked(link) => link, + other => panic!("omp states a LINK or nothing at all: {other:?}"), + }; + assert_eq!(link.driver, "omp"); + assert_eq!(link.conversation, "2f8c-4d11"); + assert_eq!( + link.history_mutability, + harness_state::HistoryMutability::Rewritable, + "omp compacts its own session store, so a prefix read once may be gone" + ); + assert_eq!( + link.capability_evidence, + harness_state::CapabilityEvidence::Probed, + "read off a live event, not declared from typings" + ); + assert_eq!(link.verified_through_ms, 1_787_999_000_000); + + for frame in [ + json!({"type":"conversation"}), + json!({"type":"conversation","sessionId":" "}), + json!({"type":"conversation","sessionId":42}), + json!({"type":"state","state":"idle","sessionId":"2f8c"}), + ] { + assert!( + conversation_claim(&frame, 1_787_999_000_000).is_none(), + "nothing to prove, nothing to state: {frame}" + ); + } + assert!( + conversation_claim(&json!({"type":"conversation","sessionId":"2f8c"}), 0).is_none(), + "a link with no positive verification bound is refused at the write boundary" + ); + } + + /// Negotiation. st2's hello version never rises — the asset refuses a hello it cannot read, + /// and a refusal costs that seat its mail — so the offer is additive and the AGREEMENT is the + /// asset's answer. Anything else, including an answer naming a version st2 never offered, is + /// dropped like every other frame this channel cannot vouch for. + #[test] + fn only_an_answer_to_the_offer_negotiates_the_condition_axis() { + assert_eq!(PROTOCOL, 1); + assert_eq!( + negotiated_protocol(&json!({"type":"client_hello","protocol":2})), + Some(PROTOCOL_CONDITION_AXIS) + ); + for frame in [ + json!({"type":"client_hello"}), + json!({"type":"client_hello","protocol":1}), + json!({"type":"client_hello","protocol":3}), + json!({"type":"client_hello","protocol":"2"}), + json!({"type":"client_hello","protocol":-2}), + json!({"type":"state","state":"idle","protocol":2}), + json!({"protocol":2}), + ] { + assert_eq!(negotiated_protocol(&frame), None, "frame: {frame}"); + } + } + + /// On a writer emitting version 2 the whole negotiated vocabulary is INERT: the answer and + /// the conversation statement change no byte of the record, because the version 2 wire has + /// nowhere to carry them and this record has exactly one source of truth. That was what made + /// the adapter safe to land before the selector flipped, and it is now what makes a rollback + /// and every version 2 record already on disk safe. + #[test] + fn a_negotiated_peers_new_frames_never_reach_the_version_two_wire() { + let tmp = tempfile::tempdir().unwrap(); + let writer = harness_state::Writer::new(tmp.path(), "h.worker", "omp", Some("w".into())) + .with_emitted_schema(harness_state::SCHEMA_V2); + assert!( + !writer.writes_condition_axis(), + "this test is the proof of the version 2 projection, reached through the seam" + ); + + let v2 = |frames: &[&str]| omp_record_emitting(frames, harness_state::SCHEMA_V2); + let legacy = v2(&[r#"{"type":"state","state":"active"}"#]); + let negotiated = v2(&[ + r#"{"type":"client_hello","protocol":2}"#, + r#"{"type":"conversation","sessionId":"2f8c-4d11"}"#, + r#"{"type":"state","state":"active"}"#, + ]); + assert_eq!(legacy, negotiated); + assert_eq!(legacy["schema"], harness_state::SCHEMA_V2); + assert_eq!(legacy["state"], "active"); + assert_eq!(legacy["blockedOn"], "none"); + assert!( + legacy.get("condition").is_none() && legacy.get("conversationRef").is_none(), + "neither axis exists on this wire: {legacy}" + ); + + // A failed turn still publishes exactly the legacy row it always did — the condition it + // now also implies is representable nowhere, so it changes nothing here. + let faulted = v2(&[ + r#"{"type":"client_hello","protocol":2}"#, + r#"{"type":"turn","error":{"reason":"401 invalid x-api-key","errorId":16781312}}"#, + ]); + assert_eq!(faulted["state"], "active"); + assert_eq!(faulted["reason"], PROVIDER_AUTH_REASON); + assert!(faulted.get("condition").is_none()); + } + + /// The whole negotiated vocabulary respects the incarnation's last word. `src/omp_session.rs` + /// alone writes `ended`, and every frame the extension queued before the wrapper reaped the + /// session — the answer, an activity edge, a failed turn, the conversation statement, and the + /// success edge that would otherwise clear everything — is refused rather than resurrecting + /// a session nobody is watching. + #[test] + fn no_negotiated_frame_resurrects_the_wrappers_terminal_omp_record() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path(); + std::fs::create_dir_all(message::inbox_dir(agent_dir)).unwrap(); + let record = harness_state::harness_state_path(agent_dir); + // The wrapper mints the token and the channel adopts it: that sharing is what makes the + // wrapper's terminal record this session's last word rather than a foreign one. + let session = harness_state::session_token(); + let mut channel_writer = + harness_state::Writer::new(agent_dir, "h.worker", "omp", Some("h.worker".into())) + .with_session(session.clone()); + let mut wrapper_writer = + harness_state::Writer::new(agent_dir, "h.worker", "omp", Some("h.worker".into())) + .with_session(session); + // Exactly what `src/omp_session.rs` does, bootstrap retry included. + crate::provider_session::write_terminal( + &mut wrapper_writer, + "exit 0", + None, + harness_state::ConditionReport::Clear, + ) + .unwrap(); + let terminal = std::fs::read(&record).unwrap(); + + let (tx, rx) = mpsc::channel(); + for frame in [ + r#"{"type":"client_hello","protocol":2}"#, + r#"{"type":"state","state":"active"}"#, + r#"{"type":"turn","error":{"reason":"401 invalid x-api-key","errorId":16781312}}"#, + r#"{"type":"conversation","sessionId":"2f8c-4d11"}"#, + r#"{"type":"turn"}"#, + ] { + tx.send(Ok(frame.to_string())).unwrap(); + } + drop(tx); + channel_loop( + &rx, + &mut Vec::new(), + &message::inbox_dir(agent_dir), + agent_dir, + &mut channel_writer, + None, + "h.worker", + &OMP_KIND, + Duration::from_millis(2), + Duration::from_secs(60), + ) + .unwrap(); + + assert_eq!( + std::fs::read(&record).unwrap(), + terminal, + "a refused write changes no byte" + ); + } + + /// Activation, end to end for pi: the REAL channel loop drives the REAL production writer, + /// and the record is read back through [`harness_state::read`] rather than inspected as an + /// in-memory frame. A pi assistant failure lands as a version 3 fault, and the idle edge that + /// follows leaves the seat honestly idle WITH the fault still standing — activity never + /// clears a condition, which is the whole ordering rule of this adapter. + #[test] + fn a_pi_failure_lands_a_version_three_fault_beside_an_honest_idle() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path(); + std::fs::create_dir_all(message::inbox_dir(agent_dir)).unwrap(); + let path = harness_state::harness_state_path(agent_dir); + let mut writer = + harness_state::Writer::new(agent_dir, "h.worker", "pi", Some("h.worker".into())); + assert!( + writer.writes_condition_axis(), + "this is the production writer after activation" + ); + + let (tx, rx) = mpsc::channel(); + for frame in [ + r#"{"type":"state","state":"active"}"#, + r#"{"type":"condition","op":"raise","category":"harness","code":"pi/assistantError","recovery":"unknown","detail":"401 Unauthorized"}"#, + r#"{"type":"state","state":"idle"}"#, + ] { + tx.send(Ok(frame.to_string())).unwrap(); + } + drop(tx); + channel_loop( + &rx, + &mut Vec::new(), + &message::inbox_dir(agent_dir), + agent_dir, + &mut writer, + None, + "h.worker", + &PI_KIND, + Duration::from_millis(1), + Duration::from_secs(60), + ) + .unwrap(); + + let observed = harness_state::read(&path, None).expect("a version 3 record"); + assert_eq!(observed.schema.as_deref(), Some(harness_state::SCHEMA_V3)); + assert_eq!(observed.state, harness_state::Activity::Idle); + let harness_state::ConditionView::Fault(fault) = &observed.condition else { + panic!("the raise must reach the record: {observed:?}"); + }; + assert_eq!( + fault.category, + Some(harness_state::FaultCategory::Harness) + ); + assert_eq!(fault.code.as_deref(), Some("pi/assistantError")); + assert_eq!( + observed.human_ask, + harness_state::HumanAsk::Unknown, + "pi exposes no ask surface and says so positively" + ); + } + + /// Activation, end to end for omp: a negotiated approval pair lands a version 3 ask and the + /// conversation identity omp forwarded, read back through the real reader. Both axes exist + /// only on this wire, so before the flip this record could not have carried either. + #[test] + fn an_omp_approval_lands_a_version_three_ask_and_links_its_conversation() { + let tmp = tempfile::tempdir().unwrap(); + let agent_dir = tmp.path(); + std::fs::create_dir_all(message::inbox_dir(agent_dir)).unwrap(); + let path = harness_state::harness_state_path(agent_dir); + let mut writer = + harness_state::Writer::new(agent_dir, "h.worker", "omp", Some("h.worker".into())) + .with_ownership("session-1", 1); + + let (tx, rx) = mpsc::channel(); + for frame in [ + r#"{"type":"client_hello","protocol":2}"#, + r#"{"type":"conversation","sessionId":"2f8c-4d11"}"#, + r#"{"type":"state","state":"active","blockedOn":"human","ask":"permission"}"#, + ] { + tx.send(Ok(frame.to_string())).unwrap(); + } + drop(tx); + channel_loop( + &rx, + &mut Vec::new(), + &message::inbox_dir(agent_dir), + agent_dir, + &mut writer, + None, + "h.worker", + &OMP_KIND, + Duration::from_millis(1), + Duration::from_secs(60), + ) + .unwrap(); + + let observed = harness_state::read(&path, None).expect("a version 3 record"); + assert_eq!(observed.schema.as_deref(), Some(harness_state::SCHEMA_V3)); + assert_eq!( + observed.human_ask, + harness_state::HumanAsk::Pending(harness_state::AskKind::Permission) + ); + let link = observed + .conversation + .as_ref() + .expect("the forwarded session id is a link"); + assert!( + format!("{link:?}").contains("2f8c-4d11"), + "the link carries omp's own conversation identity: {link:?}" + ); } } diff --git a/src/pi_session.rs b/src/pi_session.rs index bb59c25c..20391e7a 100644 --- a/src/pi_session.rs +++ b/src/pi_session.rs @@ -19,6 +19,7 @@ use anyhow::{Context as _, Result}; use crate::provider_session::{ PROVIDER_POLL, ProviderOutcome, STOP, install_signal_handler, run_provider_observed, + write_terminal, }; use crate::{harness_state, hooks, message, status}; @@ -32,7 +33,13 @@ const EXTENSION: &str = "pi-channel.ts"; /// type-checks and runtime-smokes the shipped asset against exactly this tarball. That makes the /// flake pin the gate for this constant, and `pi_channel`'s fixture asserts the two agree: a pi /// bump that changes what `getContextUsage().tokens` means must move both together or fail. -pub const MEASURED_CONTEXT_VERSION: &str = "0.84.2"; +/// +/// Moved 0.84.2 -> 0.84.4 with the flake pin, and the arithmetic was re-verified rather than +/// assumed: `getContextUsage()` (`dist/core/agent-session.js`) and `calculateContextTokens` +/// (`dist/core/compaction/compaction.js:86-88`, `usage.totalTokens || input + output + cacheRead +/// + cacheWrite`) are byte-identical between the two published tarballs, so the recorded fixture +/// still means what it says under the newer pin. +pub const MEASURED_CONTEXT_VERSION: &str = "0.84.4"; /// The exact st2 executable the pi extension must spawn for its channel. pub const CHANNEL_BIN: &str = "ST2_PI_CHANNEL_BIN"; @@ -106,14 +113,15 @@ pub fn run( Some(runtime_id.clone()), ) .with_ownership(session.clone(), seq); - let _ = writer.observe( - harness_state::Observation::new( - harness_state::Activity::Ended, - harness_state::BlockedOn::None, - harness_state::InputBuffer::Unknown, - ) - .with_reason("launch-error") - .with_exit("exit unknown"), + // The launch error is this incarnation's FIRST write: the claim fence above states + // no condition and fences are never carried forward, so the version 3 write needs + // the one bootstrap retry [`write_terminal`] owns or the seat would keep reading a + // takeover placeholder instead of a launch that never ran. + let _ = write_terminal( + &mut writer, + "exit unknown", + Some("launch-error"), + harness_state::ConditionReport::Clear, ); return Err(error); } @@ -173,7 +181,15 @@ fn record_session_end( let mut writer = harness_state::Writer::new(agent_dir, identity, "pi", Some(runtime_id.to_string())) .with_ownership(session, seq); - if let Err(error) = writer.ended(label) { + // The channel may have published nothing at all — a pi that died before loading its + // extension is exactly that — so this terminal write states the condition axis itself when + // nothing else did, and otherwise carries the channel's standing condition forward. + if let Err(error) = write_terminal( + &mut writer, + &label, + None, + harness_state::ConditionReport::Clear, + ) { tracing::warn!("st2 pi driver: recording session end failed: {error}"); } } @@ -284,11 +300,16 @@ mod tests { let agent_dir = tmp.path(); let mut channel_writer = crate::harness_state::Writer::new(agent_dir, "h.worker", "pi", Some("h.worker".into())); + // The channel STATES the tuple, exactly as `pi_channel` does: version 3 refuses an + // activity edge whose condition axis no producer of this session ever stated, so a legacy + // observation here would write nothing and the terminal record would have no counter to + // continue. channel_writer - .observe(crate::harness_state::Observation::new( + .publish(crate::harness_state::Frame::new( crate::harness_state::Activity::Active, - crate::harness_state::BlockedOn::None, crate::harness_state::InputBuffer::Unknown, + crate::harness_state::ConditionReport::Clear, + crate::harness_state::HumanAsk::Unknown, )) .unwrap(); drop(channel_writer); @@ -434,11 +455,15 @@ mod tests { let mut channel = harness_state::Writer::new(tmp.path(), "h.worker", "pi", Some("h.worker".to_string())) .with_ownership(session.clone(), seq); + // The channel STATES its tuple, as `pi_channel` does. A legacy observation here would be + // refused as unstated over the claim fence above, leaving `live` as the fence bytes and + // making the no-heartbeat assertion vacuous. channel - .observe(harness_state::Observation::new( + .publish(harness_state::Frame::new( Activity::Active, - harness_state::BlockedOn::None, harness_state::InputBuffer::Unknown, + harness_state::ConditionReport::Clear, + harness_state::HumanAsk::Unknown, )) .unwrap(); let live = std::fs::read(&record).unwrap(); diff --git a/src/provider_session.rs b/src/provider_session.rs index 09648ff3..c0736032 100644 --- a/src/provider_session.rs +++ b/src/provider_session.rs @@ -145,25 +145,95 @@ impl SessionObserver { } /// Best-effort terminal record; observation must never turn a clean teardown into an error. + /// + /// A wrapper observed nothing about its harness, so `clear` is the only axis it may state: + /// see [`write_terminal`] for why that is truthful rather than fabricated health. pub(crate) fn ended(&self, exit: &str) { - let _ = self.writer().ended(exit); + let _ = write_terminal( + &mut self.writer(), + exit, + None, + harness_state::ConditionReport::Clear, + ); } /// The terminal record for a session whose provider never ran (or could no longer be /// checked): a real ended record, so the claim placeholder is not the last word. pub(crate) fn launch_error(&self) { - let _ = self.writer().observe( - harness_state::Observation::new( - harness_state::Activity::Ended, - harness_state::BlockedOn::None, - harness_state::InputBuffer::Unknown, - ) - .with_reason("launch-error") - .with_exit("exit unknown"), + let _ = write_terminal( + &mut self.writer(), + "exit unknown", + Some("launch-error"), + harness_state::ConditionReport::Clear, ); } } +/// THE terminal record, in whichever vocabulary this writer emits, with the one bootstrap retry +/// version 3 requires. Every process-exit owner in this crate goes through here — the wrappers, +/// the pi and omp session drivers, Codex's TUI end, and OpenCode's four exit paths — because they +/// are the only writers of `ended` and the version they write it in must be decided once. +/// +/// A terminal write is the FIRST write of many incarnations: `claim` planted a fence, fences are +/// deliberately excluded from carry-forward, and a provider that died before any producer +/// published leaves the condition axis never stated. Version 3 refuses that write as +/// [`harness_state::Refusal::Unstated`] rather than inventing an `absent` it cannot serialize, so +/// without this retry a wrapper-only `ended` would silently stop landing and the seat would read +/// `unknown` for every consumer. +/// +/// The first attempt always rides `Unchanged`: an exit says nothing new about a fault, and +/// whatever stood is the incarnation's last word about that too. Only the refusal — which is +/// itself the proof that no condition of this session's stands, because a standing one would have +/// been inherited and stated — admits the retry, and the retry states `bootstrap` exactly ONCE. +/// The caller names that value because only it knows what it observed: a wrapper that never saw +/// its harness passes `clear`, while a driver holding a reduced verdict passes what its own +/// reducer projects. +/// +/// Under version 2 nothing can be unstated and the refusal never occurs, so this makes the EXACT +/// legacy statement [`harness_state::Writer::ended`] makes and the shipped bytes do not move. +/// `None` is returned there: the legacy surface has no typed outcome to report. +pub(crate) fn write_terminal( + writer: &mut harness_state::Writer, + exit: &str, + reason: Option<&str>, + bootstrap: harness_state::ConditionReport, +) -> anyhow::Result> { + if !writer.writes_condition_axis() { + let mut observation = harness_state::Observation::new( + harness_state::Activity::Ended, + harness_state::BlockedOn::None, + harness_state::InputBuffer::Unknown, + ) + .with_exit(exit); + if let Some(reason) = reason { + observation = observation.with_reason(reason); + } + writer.observe(observation)?; + return Ok(None); + } + let terminal = |condition: harness_state::ConditionReport| { + let mut frame = harness_state::Frame::new( + harness_state::Activity::Ended, + harness_state::InputBuffer::Unknown, + condition, + harness_state::HumanAsk::None, + ) + .with_exit(exit); + if let Some(reason) = reason { + frame = frame.with_reason(reason); + } + frame + }; + let outcome = writer.publish(terminal(harness_state::ConditionReport::Unchanged))?; + if matches!( + outcome.refusal(), + Some(harness_state::Refusal::Unstated) + ) { + return writer.publish(terminal(bootstrap)).map(Some); + } + Ok(Some(outcome)) +} + fn describe_exit(exit: ExitStatus) -> String { match (exit.code(), exit.signal()) { (Some(code), _) => format!("exit {code}"), diff --git a/tests/delivery_ledger_migration.rs b/tests/delivery_ledger_migration.rs new file mode 100644 index 00000000..5588eb4f --- /dev/null +++ b/tests/delivery_ledger_migration.rs @@ -0,0 +1,407 @@ +//! The delivery ledger's migration and rollback contract, asserted from outside the drivers. +//! +//! Two failure classes are load-bearing here, and both were measured before this code existed: +//! +//! * **An in-place schema bump has no rollback at all.** v1's Codex loader `ensure!`s its schema +//! and denies unknown fields, and that error propagates through `CodexInboxDelivery::new` into +//! the control connection's startup — a v2 body at `delivery-state.json` makes the old binary +//! refuse to start, deterministically, at every crash position. v1's OpenCode loader instead +//! discards a body it cannot read and then POSTs before requerying, appending the same +//! `messageID`'s parts a second time. So the ledger must live in a fresh namespace and the v1 +//! path must only ever hold a v1 body. +//! * **A delivery started by the new binary has no v1 record.** Rolled back, the old OpenCode pump +//! takes its `state == None` path and re-POSTs (9 of 9 crash positions duplicated). The +//! v1-*shaped* `Attempted` floor removes that class — but only if it passes v1's own load +//! filter, which recomputes the derived correlation and silently discards a record that differs. +//! +//! These tests therefore mirror both v1 record types verbatim, including `deny_unknown_fields`, +//! and recompute both derived correlations independently of the driver modules. + +use std::fs; +use std::path::{Path, PathBuf}; + +use serde::Deserialize; +use serde_json::{Value, json}; +use sha2::{Digest as _, Sha256}; +use st2::delivery_ledger::{ + self, Begin, CODEX_LEGACY_SCHEMA, Correlation, Evidence, Harness, HoldReason, LEDGER_FILE, + LEDGER_SCHEMA, LEGACY_FILE, Ledger, NegativeReceipt, OPENCODE_LEGACY_SCHEMA, Phase, Retention, + RetryDecision, +}; + +const AGENT: &str = "h.worker"; +const FILE_A: &str = "1786380000000-aaa111.md"; +const THREAD: &str = "thread-main"; +const SESSION: &str = "ses_target"; +const INCARNATION: &str = "incarnation-1"; + +/// v1's `CodexDeliveryState`, mirrored verbatim. `deny_unknown_fields` is the point: it is why an +/// additively-extended record at this path is an outage rather than a degraded read. +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct CodexV1 { + schema: String, + agent: String, + runtime_id: String, + runtime_incarnation: String, + thread_id: String, + filename: String, + client_id: String, + phase: String, +} + +/// v1's OpenCode `DeliveryState`, mirrored verbatim. +#[derive(Debug, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct OpenCodeV1 { + schema: String, + agent: String, + runtime_id: String, + session_id: String, + filename: String, + message_id: String, + phase: String, +} + +fn digest(domain: &[u8], parts: [&str; 3]) -> String { + let mut hash = Sha256::new(); + hash.update(domain); + for value in parts { + hash.update((value.len() as u64).to_be_bytes()); + hash.update(value.as_bytes()); + } + format!("{:x}", hash.finalize()) +} + +/// Recomputed here, independently of `codex_app_server`: v1 validates the client ID against this +/// exact derivation, so a floor that does not match is a floor v1 refuses. +fn codex_client_id(thread: &str, filename: &str) -> String { + format!( + "st2:{}", + digest(b"st2.codex-client-user-message.v1", [AGENT, thread, filename]) + ) +} + +/// Recomputed here, independently of `opencode_session`: v1 recomputes this and silently discards +/// a record whose `messageId` differs, which would put the duplicate-POST class straight back. +fn opencode_message_id(session: &str, filename: &str) -> String { + format!( + "msg{:.26}", + digest(b"st2.opencode-client-message.v1", [AGENT, session, filename]) + ) +} + +fn legacy_path(dir: &Path) -> PathBuf { + dir.join("state").join(LEGACY_FILE) +} + +fn ledger_path(dir: &Path) -> PathBuf { + dir.join("state").join(LEDGER_FILE) +} + +fn open(dir: &Path, harness: Harness) -> Ledger { + match harness { + Harness::Codex => Ledger::open( + &legacy_path(dir), + harness.profile(), + AGENT, + AGENT, + |thread, filename| codex_client_id(thread, filename), + ), + Harness::OpenCode => Ledger::open( + &legacy_path(dir), + harness.profile(), + AGENT, + AGENT, + |session, filename| opencode_message_id(session, filename), + ), + } +} + +fn begin(ledger: &mut Ledger, harness: Harness) { + let begin = match harness { + Harness::Codex => { + let client_id = codex_client_id(THREAD, FILE_A); + Begin { + filename: FILE_A.to_string(), + binding: THREAD.to_string(), + correlation: Correlation::native(client_id.clone()), + incarnation: Some(INCARNATION.to_string()), + legacy_floor: delivery_ledger::codex_floor( + AGENT, + AGENT, + INCARNATION, + THREAD, + FILE_A, + &client_id, + ), + } + } + Harness::OpenCode => { + let message_id = opencode_message_id(SESSION, FILE_A); + Begin { + filename: FILE_A.to_string(), + binding: SESSION.to_string(), + correlation: Correlation::native(message_id.clone()), + incarnation: None, + legacy_floor: delivery_ledger::opencode_floor( + AGENT, AGENT, SESSION, FILE_A, &message_id, + ), + } + } + }; + ledger.begin(begin).unwrap(); +} + +fn read_json(path: &Path) -> Value { + serde_json::from_slice(&fs::read(path).unwrap()).unwrap() +} + +/// v1's own load filter for each harness, applied to whatever is at the v1 path. `Ok(phase)` means +/// the old binary would load this record and act on it; `Err` means it would refuse to start +/// (Codex) or silently discard it and re-POST (OpenCode). +fn v1_would_load(path: &Path, harness: Harness) -> Result { + let bytes = fs::read(path).map_err(|error| error.to_string())?; + match harness { + Harness::Codex => { + let state: CodexV1 = + serde_json::from_slice(&bytes).map_err(|error| error.to_string())?; + if state.schema != CODEX_LEGACY_SCHEMA { + return Err(format!("unsupported schema '{}'", state.schema)); + } + if state.agent != AGENT || state.runtime_id != AGENT { + return Err("belongs to a different runtime".to_string()); + } + if state.runtime_incarnation.is_empty() || state.thread_id.is_empty() { + return Err("invalid runtime binding".to_string()); + } + if state.client_id != codex_client_id(&state.thread_id, &state.filename) { + return Err("client ID does not match its binding".to_string()); + } + Ok(state.phase) + } + Harness::OpenCode => { + let state: OpenCodeV1 = + serde_json::from_slice(&bytes).map_err(|error| error.to_string())?; + if state.schema != OPENCODE_LEGACY_SCHEMA + || state.agent != AGENT + || state.message_id != opencode_message_id(&state.session_id, &state.filename) + { + return Err("discarded by the v1 filter".to_string()); + } + Ok(state.phase) + } + } +} + +#[test] +fn in_place_schema_bump_is_rejected_by_construction() { + for harness in [Harness::Codex, Harness::OpenCode] { + let tmp = tempfile::tempdir().unwrap(); + let mut ledger = open(tmp.path(), harness); + begin(&mut ledger, harness); + ledger.record(FILE_A, Evidence::TransportAccepted).unwrap(); + match harness { + Harness::Codex => ledger.record(FILE_A, Evidence::Consumed).unwrap(), + Harness::OpenCode => ledger.record(FILE_A, Evidence::Persisted).unwrap(), + }; + + // Authority is in the fresh namespace, under the ledger's own schema. + assert_eq!(read_json(&ledger_path(tmp.path()))["schema"], LEDGER_SCHEMA); + + // And the v1 path never holds a ledger body — the shape that makes Codex refuse to start + // and OpenCode duplicate. It holds a v1 body or nothing at all. + if legacy_path(tmp.path()).exists() { + let v1 = read_json(&legacy_path(tmp.path())); + assert_eq!(v1["schema"], harness.legacy_schema()); + assert!( + v1.get("entries").is_none() && v1.get("harness").is_none(), + "no ledger field ever reaches the v1 path: {v1}" + ); + } + } +} + +#[test] +fn old_binary_rollback_neither_duplicates_nor_refuses_to_start() { + // A delivery STARTED by the new binary: the case with no pre-existing v1 record, and the one + // that duplicated at every crash position without a floor. + for harness in [Harness::Codex, Harness::OpenCode] { + let tmp = tempfile::tempdir().unwrap(); + let mut ledger = open(tmp.path(), harness); + begin(&mut ledger, harness); + + // Boundary 1: crash right after the first transport was owned. + assert_eq!( + v1_would_load(&legacy_path(tmp.path()), harness), + Ok("attempted".to_string()), + "the rolled-back binary loads a true lower bound and reconciles" + ); + let exact = read_json(&legacy_path(tmp.path())); + + // Boundary 2: the transport landed. The floor still says only `attempted`, so the old + // binary reconciles rather than treating it as acceptance. + ledger.record(FILE_A, Evidence::TransportAccepted).unwrap(); + assert_eq!(read_json(&legacy_path(tmp.path())), exact); + + // Boundary 3: the harness's own receipt. Codex proves consumption and releases; OpenCode + // proves only storage, holds, and keeps the floor for a rollback. + match harness { + Harness::Codex => { + ledger.record(FILE_A, Evidence::Consumed).unwrap(); + assert_eq!(ledger.retention(FILE_A), Retention::Release); + assert!( + !legacy_path(tmp.path()).exists(), + "release clears the floor: nothing is outstanding to protect" + ); + } + Harness::OpenCode => { + ledger.record(FILE_A, Evidence::Persisted).unwrap(); + assert_eq!( + ledger.retention(FILE_A), + Retention::Hold(HoldReason::UnreadReceipt) + ); + assert_eq!(read_json(&legacy_path(tmp.path())), exact); + } + } + + // At no boundary does a rolled-back binary see something it must refuse to start on. + if legacy_path(tmp.path()).exists() { + assert!(v1_would_load(&legacy_path(tmp.path()), harness).is_ok()); + } + } +} + +#[test] +fn a_lost_ledger_write_after_the_floor_recovers_as_an_ambiguous_attempt() { + // The exact interleaving the sweep found: the floor landed, the ledger's own record did not. + // Adoption reads the floor back as the ambiguous attempt it is — held, surfaced, never + // replayed — instead of starting a second delivery. + for harness in [Harness::Codex, Harness::OpenCode] { + let tmp = tempfile::tempdir().unwrap(); + let floor = match harness { + Harness::Codex => delivery_ledger::codex_floor( + AGENT, + AGENT, + INCARNATION, + THREAD, + FILE_A, + &codex_client_id(THREAD, FILE_A), + ), + Harness::OpenCode => delivery_ledger::opencode_floor( + AGENT, + AGENT, + SESSION, + FILE_A, + &opencode_message_id(SESSION, FILE_A), + ), + }; + fs::create_dir_all(tmp.path().join("state")).unwrap(); + fs::write( + legacy_path(tmp.path()), + serde_json::to_vec(&floor).unwrap(), + ) + .unwrap(); + + let mut recovered = open(tmp.path(), harness); + assert_eq!(recovered.entry(FILE_A).unwrap().phase, Phase::Attempted); + assert_eq!( + recovered.retry(FILE_A), + RetryDecision::Hold(HoldReason::AdoptedWithoutFreshEvidence) + ); + + // Only fresh evidence about the world moves it, and then it is a first-class attempt. + recovered.negative(FILE_A, NegativeReceipt::Absent).unwrap(); + assert_eq!(recovered.retry(FILE_A), RetryDecision::Retry); + } +} + +#[test] +fn adoption_downgrades_each_v1_label_to_the_evidence_it_proved() { + // Codex `Accepted` came from the typed completed user message: consumption, and its true + // ceiling, so it releases. + let codex = tempfile::tempdir().unwrap(); + fs::create_dir_all(codex.path().join("state")).unwrap(); + fs::write( + legacy_path(codex.path()), + serde_json::to_vec(&json!({ + "schema": CODEX_LEGACY_SCHEMA, + "agent": AGENT, + "runtimeId": AGENT, + "runtimeIncarnation": INCARNATION, + "threadId": THREAD, + "filename": FILE_A, + "clientId": codex_client_id(THREAD, FILE_A), + "phase": "accepted", + })) + .unwrap(), + ) + .unwrap(); + let ledger = open(codex.path(), Harness::Codex); + assert_eq!(ledger.quarantined(), None); + assert_eq!(ledger.entry(FILE_A).unwrap().phase, Phase::Consumed); + assert_eq!(ledger.retention(FILE_A), Retention::Release); + + // OpenCode `Accepted` came from a `GET 200`: storage. Mapping it to consumption would make + // the stored-but-never-admitted class permanently unretryable, so it adopts as persisted and + // keeps holding. + let opencode = tempfile::tempdir().unwrap(); + fs::create_dir_all(opencode.path().join("state")).unwrap(); + fs::write( + legacy_path(opencode.path()), + serde_json::to_vec(&json!({ + "schema": OPENCODE_LEGACY_SCHEMA, + "agent": AGENT, + "runtimeId": AGENT, + "sessionId": SESSION, + "filename": FILE_A, + "messageId": opencode_message_id(SESSION, FILE_A), + "phase": "accepted", + })) + .unwrap(), + ) + .unwrap(); + let ledger = open(opencode.path(), Harness::OpenCode); + assert_eq!(ledger.entry(FILE_A).unwrap().phase, Phase::Persisted); + assert_eq!( + ledger.retention(FILE_A), + Retention::Hold(HoldReason::UnreadReceipt), + "ownership is held because admission is unread, not because adoption is unproven" + ); + // Two reasons apply to this entry — storage without admission, and a carried-forward claim + // with no fresh evidence — and the retry gate names the stricter one. Retention describes what + // the harness has; retry describes what st2 may do about it, and a v1 record is never a reason + // to transport. + assert_eq!( + ledger.retry(FILE_A), + RetryDecision::Hold(HoldReason::AdoptedWithoutFreshEvidence), + "adoption alone authorizes no transport" + ); +} + +#[test] +fn the_ledger_never_moves_an_inbox_file() { + // The ledger's whole vocabulary is retention: may-resend, hold and surface, release FIFO + // ownership. Archive stays the recipient agent's act, so the ledger touches nothing but its + // own two files. + let tmp = tempfile::tempdir().unwrap(); + let inbox = tmp.path().join("agents/h/worker/resources/inbox"); + fs::create_dir_all(&inbox).unwrap(); + fs::write(inbox.join(FILE_A), "body").unwrap(); + + let mut ledger = open(tmp.path(), Harness::Codex); + begin(&mut ledger, Harness::Codex); + ledger.record(FILE_A, Evidence::Consumed).unwrap(); + assert_eq!(ledger.retention(FILE_A), Retention::Release); + assert!( + inbox.join(FILE_A).is_file(), + "released ownership is not an archive" + ); + + // Only the recipient archiving it retires the entry, and even then only the ledger's own + // record changes. + fs::remove_file(inbox.join(FILE_A)).unwrap(); + ledger.prune(|_| false).unwrap(); + assert!(ledger.entries().is_empty()); + assert_eq!(read_json(&ledger_path(tmp.path()))["entries"], json!([])); +}