Merge pull request #106 from compoundingtech/schickling-assistant/2026-07-30-catalog-publication

Publish agent specs transactionally

Author: schickling-assistant

INVARIANTS.md

@@ -14,7 +14,7 @@ materialization, messaging, DING, or presence must preserve them.
 | **Fail-closed observed native DING** | Each unread message becomes one normalized `[DING]` frame. Fresh delivery records ownership, then preserves the one combined bracketed-paste, 0.5 second delay, and Return transaction. PTY and Return success are transport only: `Delivered` additionally requires adapter classification of the expected notice text in a submitted-prompt or queued-message pattern while the lowest live composer is empty or an accepted idle placeholder. Every other receipt retains staged ownership. Retry never re-pastes and may send one bare Return only after two adjacent `RetainedSafe` observations; human, modal, active, changed, timed-out, and unknown retry states receive no input. Ownership prevents duplicate paste across command failures, receipt ambiguity, archive races, and restart adoption. Startup backlog otherwise becomes one generic recovery DING; new arrivals remain FIFO; `busy` delivers immediately; only fresh `dnd` defers. | `src/ding/mod.rs::poke_text_normalizes_and_bounds_untrusted_fields`; `src/ding/mod.rs::malicious_controls_cannot_escape_the_single_paste_frame`; `src/ding/mod.rs::pty_delivery_uses_face607_delay_order_and_seconds`; `src/ding/mod.rs::maintained_composer_classifiers_require_exact_idle_state`; `src/ding/mod.rs::successful_transport_with_retained_or_unproven_pixels_is_not_delivered`; `src/ding/mod.rs::ambiguous_transport_receipt_and_retry_errors_retain_staged_ownership`; `src/ding/mod.rs::adapter_recognized_notice_with_an_empty_live_composer_is_a_positive_receipt`; `src/ding/mod.rs::staged_retry_submits_only_retained_safe_and_requires_a_receipt`; `src/ding/mod.rs::staged_ownership_survives_archive_and_never_repastes`; `src/ding/mod.rs::pty_commands_have_a_real_outer_timeout`; `src/ding/mod.rs::session_watch_has_startup_grace_debounce_and_live_reset`; `src/ding/mod.rs::new_arrivals_is_fifo_and_archive_receipts_prevent_reding`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry`; `src/ding/mod.rs::startup_recovery_notice_retries_in_memory`; `src/ding/mod.rs::startup_backlog_gets_one_generic_recovery_then_new_arrivals_poke` |
 | **Mutation-only filesystem wakeups** | Supervisor and DING filesystem watchers ignore read/open access events and wake early only for create, modify, rename, or remove events. Their own catalog and inbox reads therefore cannot bypass the bounded timer cadence or form a Linux inotify CPU loop. | `src/watch.rs::only_mutations_wake_watch_loops`; `src/watch.rs::linux_reads_are_silent_but_real_mutations_wake`; `src/ding/mod.rs::idle_ding_does_not_spin_on_its_own_inbox_reads`; `src/run.rs::idle_supervisor_does_not_spin_on_its_own_catalog_reads` |
 | **Bounded DING PTY probe churn** | An unsafe or active composer retains its FIFO notice but deferred delivery retries use a bounded backoff, so each inbox poll cannot spawn another short-lived PTY probe. | `src/ding/mod.rs::deferred_delivery_backoff_bounds_short_lived_pty_attempts` |
-| **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/compile_agent.rs::compile_agent_generates_claude_then_materializes_verbatim_persona`; `tests/compile_agent.rs::compile_agent_generates_codex_then_materializes_composed_agents_md`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` |
+| **Agent-declared presence discipline** | The shipped bus contract requires agents to declare `busy` before executing work, use `available` only while yielding or ready, and reserve `dnd` for an explicit hold. Both native harnesses materialize that contract. Busy remains observable but does not suppress DING; fresh `dnd` is the only delivery gate. | `tests/native_only.rs::clean_path_executes_the_maintained_native_authoring_guide`; `src/ding/mod.rs::pending_delivery_ignores_busy_but_respects_fresh_dnd_archive_and_retry` |
 | **Stable roster JSON** | `st2 agents --json [--enrich]` preserves field names, order, null handling, presence, explicit retirement state, opaque declared Resource descriptors, activity, and inbox counts. Human output marks retired declarations without changing active rows. | `src/agents.rs::agents_json_has_stable_wire_shape`; `src/agents.rs::agents_json_preserves_opaque_declared_resource_descriptors`; `tests/status_agents.rs::roster_json_and_human_output_distinguish_retirement_from_presence` |
 | **Agent-declared presence** | Refresh preserves non-DND declared status and only advances liveness; a missing status starts as `available`, while `dnd` is never refreshed and an unrefreshed declaration ages to `unknown`. | `src/status.rs::refresh_preserves_value_and_bumps_mtime`; `src/status.rs::refresh_leaves_dnd_to_age_out`; `src/status.rs::refresh_missing_writes_available_default`; `src/status.rs::stale_mtime_reads_as_unknown_regardless_of_contents` |
 | **Retirement health** | A retired declaration is healthy only after every declared task ID is absent. Any live or dead declared task record reports incomplete retirement; retired declarations do not require presence. Live declarations retain their existing task and presence checks. | `tests/doctor.rs::retired_declaration_is_healthy_when_tasks_and_presence_are_absent`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_declared_task_is_alive`; `tests/doctor.rs::retired_declaration_is_unhealthy_while_a_dead_task_record_remains` |

README.md

@@ -4,8 +4,8 @@ st2 runs a declarative network of Codex and Claude agents from one catalog. It o
 reconciliation, native messages, normalized terminal DING delivery, presence, durable context,
 workspace materialization, and explicit teardown.
 
-Hand-authored KDL is the canonical interface. `st2 compile-agent` is experimental and must be
-reviewed before its output is materialized.
+Canonical KDL is the authoring interface. Publish one explicit Agent Spec transactionally with
+`st2 agent publish`; st2 does not compile human intent into declarations.
 
 ## Product intent and implementation contract
 
@@ -99,17 +99,17 @@ Start from the maintained [Codex](examples/native/agent-codex.kdl) or
 
 ```sh
 export CATALOG="${XDG_STATE_HOME:-$HOME/.local/state}/st2/default/catalog"
-mkdir -p "$CATALOG/agents/<host>/<identity>" "$CATALOG/_templates"
-cp examples/native/agent-codex.kdl "$CATALOG/agents/<host>/<identity>/agent.kdl"
-${EDITOR:-vi} "$CATALOG/agents/<host>/<identity>/agent.kdl"
+bundle="$(mktemp -d)"
+mkdir -p "$bundle/assets"
+cp examples/native/agent-codex.kdl "$bundle/agent.kdl"
+cp ./composed-AGENTS.md "$bundle/assets/AGENTS.md"
+${EDITOR:-vi} "$bundle/agent.kdl"
+st2 agent publish --catalog "$CATALOG" --bundle "$bundle" --expect-absent --json
 ```
 
-Replace `<host>`, `<identity>`, `<workspace>`, and `<boot prompt>`. Add every file referenced by
-`copy` under `$CATALOG/_templates`. The maintained declaration does not add workspace trust.
-`compile-agent` also omits trust by default. Pass `--harness codex --trust-workspace` to opt in to an
-argv-local Codex `projects` trust override. The generator serializes the declared workspace as the
-exact decoded key; other harnesses reject the flag. This is a launch convention inside opaque argv,
-not agent-spec grammar enforced by st2.
+Replace `<host>`, `<identity>`, `<workspace>`, and `<boot prompt>`. Include every file referenced by
+`copy` in the bundle. For a later declaration-only update, publish `agent.kdl` with the current
+declaration's SHA-256 via `--spec ... --expect-sha256 HEX`; sibling assets and state are preserved.
 
 The compact declaration shape is:
 
@@ -128,7 +128,7 @@ agent "<identity>" {
   ding
 
   render {
-    copy "_templates/<host>.<identity>.AGENTS.md" "AGENTS.md"
+    copy "assets/AGENTS.md" "AGENTS.md"
     json-upsert ".codex/hooks.json" #"""
 {"hooks":{"SessionStart":[{"hooks":[{"type":"command","command":"$ST_HOOKS/codex-session-start.sh","timeout":5}]}]}}
 """#
@@ -183,7 +183,7 @@ st2 rename <stable-id> --clear
 ```
 
 These commands preserve unrelated KDL bytes and serialize local writers through the persistent
-private `.st2/presentation-authoring.lock`. They refuse TOML, JSON, and
+shared `.st2/catalog-authoring.lock`. They refuse TOML, JSON, and
 explicitly `meta { managed-by "nix" }` targets. Nix generators must emit that marker before the
 compatible st2 binary is activated. In the trusted single-operator fleet, caller-supplied
 `ST_AGENT` limits an invocation to itself or declared descendants; it is a guardrail rather than
@@ -415,7 +415,7 @@ ls, up, down, validate, doctor
 message, ding, agents, status, context, resource, rename, describe
 env, pty, shell, pretrust
 hooks, service, eval
-compile-agent (experimental)
+agent publish
 completions
 ```
 
@@ -475,5 +475,6 @@ interviewer reply at-or-after the exact kickoff receipt completes it. Canonical
 verdict. Without the directive, Agent Spec-shaped files inside a fixture remain inert and compact
 evals retain their flat bus and completion semantics.
 
-`st2 compile-agent` remains experimental. Hand-authored KDL is the canonical st2 authoring
-interface, and generated output must be reviewed before materialization.
+`st2 agent publish --catalog ROOT (--spec FILE | --bundle DIR) (--expect-absent |
+--expect-sha256 HEX)` is the sole catalog-declaration writer. The publisher admits the complete
+prospective catalog under a compare-and-swap lock before making one atomic change.

docs/vrs/.decisions/0002-stable-agent-identity-and-mutable-presentation.md

@@ -37,7 +37,7 @@ descendants; this is an operational guardrail, not an authenticated capability,
 and absence selects the operator path. Declarations explicitly marked Nix-owned
 remain writable only at their Nix source, and Nix emitters must publish that
 marker before authoring is activated. st2 serializes these
-edits with the private persistent `.st2/presentation-authoring.lock`, preserves
+edits with the shared persistent `.st2/catalog-authoring.lock`, preserves
 unrelated source bytes, detects stale source, and atomically replaces the
 declaration. The lock covers cooperating local st2 writers in one POSIX
 filesystem/kernel lock domain; it does not claim exclusion across independently

docs/vrs/requirements.md

@@ -134,6 +134,18 @@ accepted.
   inspection exposes every Resource binding without interpreting its type or URI.
   Resource-only declaration changes do not alter a task's effective launch
   definition and do not stop, replace, or relaunch healthy work.
+- **R27 Transactional catalog authoring:** One st2 publication operation admits
+  exactly one canonical KDL Agent Spec, with explicit host and identity, against
+  the complete prospective catalog. Publication is compare-and-swap, durable,
+  and atomic: readers observe either the previous declaration set or the next
+  complete set. Reconciliation holds one coherent declaration snapshot through
+  materialization, runtime observation, planning, and execution, so a retirement
+  cannot commit and then be followed by a launch from stale catalog input.
+  A durable incomplete-apply marker fences every declaration-plane snapshot and
+  action after a crashed whole-catalog apply; a resident supervisor stays alive
+  but performs zero lifecycle actions until the transaction is completed.
+  Presence, messages, context, and Resource state remain independently writable
+  and are never serialized behind catalog authoring.
 - **R23 Fail-closed task inventory:** One read-only machine command exposes
   every desired local PTY and exec task by agent identity, task name, runtime
   id, kind, lifecycle, retirement, desired state, runtime state, PID, creation
@@ -161,8 +173,8 @@ accepted.
 - **R25 Constrained presentation authoring:** `st2 rename` and `st2 describe`
   set or clear only their corresponding direct field in one canonical KDL
   declaration selected by stable identity. They preserve unrelated source
-  bytes, serialize cooperating local writers through the persistent private
-  `.st2/presentation-authoring.lock`, reject a stale source before atomic
+  bytes, serialize cooperating local writers through the persistent shared
+  `.st2/catalog-authoring.lock`, reject a stale source before atomic
   replacement, fsync the result, and return classified receipts. The lock inode
   is never removed or stale-recovered and defines one local POSIX
   filesystem/kernel exclusion domain; it is not cross-host coordination or OS

docs/vrs/spec.md

@@ -89,7 +89,7 @@ st2 neither reads, writes, migrates, nor interprets it.
 `--clear`. They edit canonical KDL only. The operation:
 
 1. acquires the persistent exclusive
-   `<catalog>/.st2/presentation-authoring.lock` before discovery;
+   `<catalog>/.st2/catalog-authoring.lock` before discovery;
 2. resolves exactly one declaration and applies the caller-supplied `ST_AGENT`
    self/descendant guardrail when present;
 3. refuses declarations explicitly marked `meta { managed-by "nix" }`,
@@ -105,7 +105,7 @@ marker before activating a binary with authoring commands; st2 cannot infer an
 unmarked generator from KDL bytes.
 
 The lock file is a persistent real inode and is never removed or stale-recovered.
-It serializes cooperating st2 presentation writers in one local POSIX
+It serializes cooperating st2 declaration readers and writers in one local POSIX
 filesystem/kernel lock domain. Direct same-UID writes and independently
 synchronized hosts do not participate; the source recheck detects observed
 interference but is not a distributed CAS or lock service. The classified
@@ -153,6 +153,58 @@ renderer integration in [st2#61](https://github.com/compoundingtech/st2/issues/6
 and the portable Agent Spec envelope in
 [evals#41](https://github.com/compoundingtech/evals/issues/41).
 
+## Transactional catalog authoring
+
+`st2 agent publish --catalog ROOT (--spec FILE | --bundle DIR)
+(--expect-absent | --expect-sha256 HEX) --json` is the sole supported
+declaration writer. It accepts exactly one canonical KDL `agent` node with an
+explicit, path-safe host and identity. st2 no longer exposes an intent compiler:
+external renderers own the transformation from human intent to exact Agent Spec
+bytes or a create-only publication bundle.
+
+The persistent `<catalog>/.st2/catalog-authoring.lock` defines one cooperative
+read/write transaction domain:
+
+```text
+publisher (EX)  : snapshot input -> CAS -> full-catalog admission -> atomic publish + fsync
+reader (SH)     : discover -> materialize/observe -> plan -> execute
+state plane     : message | context | Resource | status                 (unlocked)
+```
+
+The publisher derives the destination from the captured declaration, replaces
+only `agent.kdl` for a hash-authorized update, and preserves all sibling runtime
+state. A bundle is create-only and is renamed from a hidden same-filesystem
+stage; retry reports `unchanged` only when every projected bundle file already
+matches. `--expect-absent` is idempotent for identical input.
+`--expect-sha256` rejects a stale writer. Full-catalog admission rejects any
+structural validation error before publication. The typed result is
+`published` or `unchanged`.
+
+The lock file is never removed: replacing its inode would split the transaction
+domain for processes that already hold it open. Reconciliation holds SH from
+discovery through execution. Validation, doctor, roster, listing,
+materialization-only, targeted reconciliation, and catalog teardown take a
+coherent SH snapshot. State-plane commands deliberately do not: their atomic
+files remain live while a declaration is admitted.
+
+`<catalog>/.st2/catalog-apply-incomplete` is the durable whole-catalog
+transaction fence. Any presence is authoritative, including malformed content.
+The reserved canonical record is:
+
+```json
+{"schema":"st2.catalog-apply-incomplete.v1","desiredStorePath":"/nix/store/..."}
+```
+
+After taking its authoring lock, st2 refuses publication, validation,
+materialization, teardown, roster, doctor, and catalog listing while the marker
+exists. One-shot and selected reconcile fail explicitly. A resident supervisor
+instead remains alive, reports a skipped/incomplete pass, and performs no
+runtime observation or lifecycle action, avoiding a service restart storm.
+Message, context, Resource, and status operations remain available. The future
+whole-catalog apply command owns marker creation, admission, durable apply,
+verification, and clearing inside one transaction; external lock execution and
+bypass flags are not part of the contract.
+
 ## Host-local scheduling and supervision
 
 ```text

examples/README.md

@@ -1,22 +1,23 @@
 # Native st2 examples
 
 [`native/`](native/) contains the maintained, hand-authored Codex and Claude agent declaration
-shapes. Copy one to:
+shapes. Render one outside the catalog as:
 
 ```text
 <catalog>/agents/<host>/<identity>/agent.kdl
 ```
 
-Replace every placeholder, add its referenced files under `<catalog>/_templates/`, then gate the
-result before starting a process:
+Replace every placeholder, put its referenced files in the bundle's `assets/` directory, then publish
+and gate the result before starting a process:
 
 ```sh
 st2 hooks install
 st2 hooks verify
+st2 agent publish --catalog <catalog> --bundle <bundle> --expect-absent --json
 st2 validate --catalog <catalog>
 st2 up --catalog <catalog> --host <host> --materialize-only
 st2 up --catalog <catalog> --host <host> --once
 ```
 
-Hand-authored KDL is canonical. `st2 compile-agent` is an experimental generation aid; inspect all
-of its KDL and workspace targets before use.
+Canonical KDL is the boundary; st2 does not compile intent. Inspect all KDL and workspace targets
+before publication.

examples/native/README.md

@@ -10,26 +10,22 @@ These maintained, hand-authored declarations are the canonical starting points:
 
 The examples use `<host>`, `<identity>`, and `<workspace>` placeholders. st2 provides `CATALOG`,
 `ST_ROOT`, `PTY_ROOT`, and `ST_HOOKS` when it starts a task, so hook declarations contain no
-machine-specific install paths. Copy the appropriate file into
-`<catalog>/agents/<host>/<identity>/agent.kdl`, replace every placeholder, and add the referenced
-catalog-owned templates. `role` is optional metadata; `supervisor` is optional runtime routing.
-Uncomment them when the agent has an assigned role or reports to another bus identity. In the Codex
-declaration, workspace trust is absent by default. The experimental generator adds an argv-local
-Codex project trust override only with `compile-agent --harness codex --trust-workspace`. It preserves
-the exact decoded workspace bytes as the key. st2 treats that argv as opaque; the trust flag is not
-part of generic catalog validation.
+machine-specific install paths. Copy the appropriate file outside the catalog, replace every
+placeholder, assemble it and the referenced files under the publication bundle's `assets/`
+directory, then publish the bundle. `role` is optional metadata; `supervisor` is optional runtime routing.
+Uncomment them when the seat has an assigned role or reports to another bus identity.
 
 ## Lifecycle
 
-`compile-agent` is an experimental generation aid. It writes one declaration, catalog-owned
-templates, and the agent's `resources/{inbox,archive,context,links}` directories without changing
-the workspace. Inspect all generated KDL and workspace targets before use.
+st2 accepts exact canonical KDL or a create-only bundle; it does not compile human intent. Inspect
+all KDL and workspace targets before publication.
 
-Use this sequence for hand-authored or generated declarations:
+Use this sequence:
 
 ```sh
 st2 hooks install
 st2 hooks verify
+st2 agent publish --catalog <catalog> --bundle <bundle> --expect-absent --json
 st2 validate <catalog>
 st2 up <catalog> --host <host> --materialize-only
 st2 up <catalog> --host <host> --once

examples/native/agent-claude.kdl

@@ -17,8 +17,8 @@ agent "<identity>" {
   ding
 
   render {
-    copy "_templates/<host>.<identity>.persona.md" ".st2/PERSONA.md"
-    copy "_templates/bus.st2.md" ".st2/bus.md"
+    copy "assets/PERSONA.md" ".st2/PERSONA.md"
+    copy "assets/bus.st2.md" ".st2/bus.md"
     ensure-line ".claude/rules/st2.md" "@../../.st2/PERSONA.md"
     ensure-line ".claude/rules/st2.md" "@../../.st2/bus.md"
     json-upsert ".claude/settings.local.json" #"""