Add crash-recoverable catalog snapshot and apply

README.md

@@ -104,12 +104,35 @@ mkdir -p "$bundle/assets"
 cp examples/native/agent-codex.kdl "$bundle/agent.kdl"
 cp ./composed-AGENTS.md "$bundle/assets/AGENTS.md"
 ${EDITOR:-vi} "$bundle/agent.kdl"
-st2 agent publish --catalog "$CATALOG" --bundle "$bundle" --expect-absent --json
+input_sha256="$(st2 agent digest --bundle "$bundle")"
+st2 agent publish --catalog "$CATALOG" --bundle "$bundle" \
+  --input-sha256 "$input_sha256" --expect-absent --json
 ```
 
 Replace `<host>`, `<identity>`, `<workspace>`, and `<boot prompt>`. Include every file referenced by
 `copy` in the bundle. For a later declaration-only update, publish `agent.kdl` with the current
 declaration's SHA-256 via `--spec ... --expect-sha256 HEX`; sibling assets and state are preserved.
+Bind either operation to the exact captured source with the SHA-256 returned by
+`st2 agent digest`.
+
+To prepare and apply a complete declaration-plane replacement without copying
+runtime state or workspaces:
+
+```sh
+st2 catalog snapshot --catalog "$CATALOG" --output ./prepared --json
+# Edit/render ./prepared, then retain the rootSha256 from the snapshot receipt.
+st2 catalog apply --catalog "$CATALOG" --prepared ./prepared \
+  --expect-sha256 <rootSha256> --json
+```
+
+`catalog apply` is policy-free. It rejects state/control content, symlinks,
+unprojected workspace facts, catalog-local/default PTY roots, and effective
+PTY-root changes. Fresh bootstrap is a separate st2+pty transaction, not an
+apply mode. A crash leaves a durable marker and content-addressed stage;
+`st2 catalog apply --catalog "$CATALOG" --resume --json` resumes without the
+original prepared source. Snapshots own the complete bounded `_templates`
+library and empty canonical per-agent `.workspace` directory facts, but never
+traverse, hash, copy, or delete workspace content.
 
 The compact declaration shape is:
 
@@ -415,7 +438,8 @@ ls, up, down, validate, doctor
 message, ding, agents, status, context, resource, rename, describe
 env, pty, shell, pretrust
 hooks, service, eval
-agent publish
+agent digest, agent publish
+catalog snapshot, catalog apply
 completions
 ```
 
@@ -475,6 +499,9 @@ interviewer reply at-or-after the exact kickoff receipt completes it. Canonical
 verdict. Without the directive, Agent Spec-shaped files inside a fixture remain inert and compact
 evals retain their flat bus and completion semantics.
 
-`st2 agent publish --catalog ROOT (--spec FILE | --bundle DIR) (--expect-absent |
---expect-sha256 HEX)` is the sole catalog-declaration writer. The publisher admits the complete
-prospective catalog under a compare-and-swap lock before making one atomic change.
+`st2 agent publish --catalog ROOT (--spec FILE | --bundle DIR) --input-sha256 HEX
+(--expect-absent | --expect-sha256 HEX)` is the single-agent declaration writer.
+`st2 catalog apply --catalog ROOT
+(--prepared DIR --expect-sha256 ROOT_HEX | --resume)` is the complete
+declaration-plane writer. Each admits the complete prospective catalog under a
+compare-and-swap lock before making one atomic change.

crates/agent-spec/src/discovery.rs

@@ -80,6 +80,19 @@ pub fn is_catalog_path(root: &Path, path: &Path) -> bool {
         return false;
     }
 
+    // Canonical state has a stable address independent of whether a declaration is currently
+    // present. This keeps orphan state after retirement/removal out of discovery and out of a
+    // whole-catalog transaction's declaration identity.
+    if components.first().and_then(|name| name.to_str()) == Some("agents")
+        && components.len() >= 4
+        && matches!(
+            components[3].to_str(),
+            Some("resources" | "archive" | "inbox" | "status")
+        )
+    {
+        return false;
+    }
+
     let mut parent = root.to_path_buf();
     for name in components {
         if matches!(name.to_str(), Some("resources" | "archive" | "inbox"))

docs/vrs/requirements.md

@@ -144,8 +144,30 @@ accepted.
   A durable incomplete-apply marker fences every declaration-plane snapshot and
   action after a crashed whole-catalog apply; a resident supervisor stays alive
   but performs zero lifecycle actions until the transaction is completed.
+  Every successful declaration commit advances a durable monotonic catalog
+  generation; whole-catalog apply advances it before its marker clears. Unlocked
+  diagnostic readers therefore detect even a completed declaration ABA across
+  their observation. A durable incomplete-generation intent fences readers
+  across each single-writer commit and is conservatively recovered by the next
+  exclusive writer. Writer staging exists only in the reserved control plane,
+  never among authoritative declaration leaves.
   Presence, messages, context, and Resource state remain independently writable
   and are never serialized behind catalog authoring.
+  A caller binds single-agent publication to the exact no-follow source capture
+  with an authoritative input digest. A canonical whole-catalog snapshot
+  externalizes the declaration-root digest while excluding runtime state and
+  workspace content. Its closed projection includes every regular file in a
+  bounded `_templates` library and exact declared canonical workspace directory
+  facts. Whole-catalog apply accepts only that projection, rechecks the root
+  digest under the exclusive lock, durably stages the desired bytes, and resumes
+  after interruption solely from a closed marker and its content-addressed
+  stage. Version 1 requires one explicit external PTY root and rejects effective
+  PTY-root changes. Fresh-catalog bootstrap is a separate cross-producer
+  transaction, not a catalog-apply mode. Apply never traverses, hashes, deletes,
+  or relocates workspace or runtime state. An absent canonical identity becomes
+  visible only as a complete bundle; a preexisting declared workspace skeleton
+  remains safe because the durable marker fences declaration readers and
+  marker-time state routing throughout leaf publication and verification.
 - **R23 Fail-closed task inventory:** One read-only machine command exposes
   every desired local PTY and exec task by agent identity, task name, runtime
   id, kind, lifecycle, retirement, desired state, runtime state, PID, creation
@@ -160,6 +182,9 @@ accepted.
   incomplete; the external backend may already have recreated a concurrently
   removed registry. This diagnostic boundary is not transactionally serialized
   with catalog or runtime writers and is not control-plane cutover authority.
+  It samples the durable catalog generation and incomplete marker around
+  discovery and runtime observation; any marker, malformed fence, or generation
+  change makes the envelope incomplete.
 - **R24 Stable identity and bounded presentation:** The positional Agent Spec
   identity and its host-qualified bus identity remain the sole stable keys for
   routing, ownership, adoption, lifecycle, and automation. Agent Specs may

docs/vrs/spec.md

@@ -95,8 +95,9 @@ st2 neither reads, writes, migrates, nor interprets it.
 3. refuses declarations explicitly marked `meta { managed-by "nix" }`,
    unsupported formats, malformed catalogs, and ambiguous targets;
 4. applies one span-bounded edit, reparses and validates the candidate;
-5. fsyncs a same-directory temporary, rechecks the original inode/version and
-   bytes, atomically renames it, then fsyncs the declaration directory.
+5. fsyncs a temporary under the reserved `.st2` control plane, rechecks the
+   original inode/version and bytes, atomically renames it through retained
+   no-follow directory capabilities, then fsyncs the declaration directory.
 
 `ST_AGENT` is a runner-provided convention in this trusted single-operator fleet,
 not an authenticated capability: a same-UID caller can alter or remove it, and
@@ -155,9 +156,11 @@ and the portable Agent Spec envelope in
 
 ## Transactional catalog authoring
 
+`st2 agent digest (--spec FILE | --bundle DIR)` captures a source through
+retained no-follow file descriptors and returns its authoritative digest.
 `st2 agent publish --catalog ROOT (--spec FILE | --bundle DIR)
-(--expect-absent | --expect-sha256 HEX) --json` is the sole supported
-declaration writer. It accepts exactly one canonical KDL `agent` node with an
+--input-sha256 HEX (--expect-absent | --expect-sha256 HEX) --json` binds
+publication to that exact capture. It accepts exactly one canonical KDL `agent` node with an
 explicit, path-safe host and identity. st2 no longer exposes an intent compiler:
 external renderers own the transformation from human intent to exact Agent Spec
 bytes or a create-only publication bundle.
@@ -168,18 +171,86 @@ read/write transaction domain:
 ```text
 publisher (EX)  : snapshot input -> CAS -> full-catalog admission -> atomic publish + fsync
 reader (SH)     : discover -> materialize/observe -> plan -> execute
-state plane     : message | context | Resource | status                 (unlocked)
+bulk apply (EX) : root CAS -> durable stage+marker -> converge -> verify+clear
+state plane     : message | context | Resource | status                    (unlocked)
 ```
 
+Every publication temporary, including a not-yet-visible identity bundle, is
+staged under `.st2`. Cross-directory rename therefore stays on the catalog
+filesystem while a crash can leave debris only in the non-projected control
+plane; declaration directories never contain writer-private leaves.
+
+Before a declaration writer mutates the live projection it durably creates
+`.st2/catalog-generation-incomplete`. Shared declaration readers and read fences
+fail closed while this intent exists. After the declaration and its parent are
+durable, the writer advances and fsyncs `catalog-generation`, then clears and
+fsyncs the intent. The next exclusive writer recovers an orphan intent by
+conservatively advancing the generation before clearing it. A crash after the
+advance but before intent removal may therefore skip a generation on recovery;
+the contract is monotonic change detection, not an exactly-once counter.
+
+The lock file is a persistent real inode: replacing or removing it would split
+the lock domain for a process that already has it open. Consequently, the first
+coherent declaration reader may initialize exactly `.st2` and this lock even
+when its requested operation later refuses. Refusal still performs no
+declaration, workspace, or state mutation.
+
 The publisher derives the destination from the captured declaration, replaces
 only `agent.kdl` for a hash-authorized update, and preserves all sibling runtime
 state. A bundle is create-only and is renamed from a hidden same-filesystem
 stage; retry reports `unchanged` only when every projected bundle file already
 matches. `--expect-absent` is idempotent for identical input.
-`--expect-sha256` rejects a stale writer. Full-catalog admission rejects any
+`--input-sha256` rejects a caller/source swap and `--expect-sha256` rejects a
+stale declaration writer. Full-catalog admission rejects any
 structural validation error before publication. The typed result is
 `published` or `unchanged`.
 
+`st2 catalog snapshot --catalog ROOT --output DIR --json` holds SH while it
+captures the canonical declaration projection: `catalog.kdl`, exact
+`agents/<host>/<identity>/agent.kdl` files, static files inside those bounded
+agent bundles, and every regular file in `_templates` whether or not a current
+render references it. `_templates` is bounded to depth 8 below its root, 256
+files, 1 MiB per file, and 32 MiB total; symlinks, hard links, special nodes,
+and reserved control/state names are rejected. Runtime state, `.git`, `.st2`,
+the native `pty` registry, and workspace content are excluded. A
+catalog-contained Agent `workspace` or Task `cwd` is valid only when it names
+that agent bundle's canonical real `.workspace`; the empty directory itself is
+an exact declaration fact, while its descendants are never traversed. The
+classification uses launch-equivalent variable expansion, resolves relative
+values from the Agent Spec bundle, and lexically normalizes before comparing
+against the logical catalog. A relative spelling is accepted only when it
+normalizes to that bundle's canonical `.workspace`; unresolved variables and
+every other effective relative path fail closed. The
+scanner always excludes a canonical `.workspace` subtree, including an orphan
+left after an agent move or removal. External workspaces remain valid and are
+not part of the projection. The output is a create-only durable directory; an
+identical retry is `unchanged`. Its domain-separated, path-sorted root SHA-256
+covers normalized relative paths, file bytes, executable bits, and empty
+workspace directory facts.
+
+`st2 catalog apply --catalog ROOT --prepared DIR --expect-sha256 HEX --json`
+rejects any prepared state/control path, symlink, special node, unprojected
+file/directory, malformed declaration, nonempty prepared workspace fact,
+catalog-local/default PTY root, or effective PTY-root change. Hash-CAS captures
+and validates exact prepared bytes, takes EX, rechecks the canonical live root,
+and either reports `unchanged` for exact equality or creates a durable
+content-addressed stage before publishing the marker. Version 1 requires an
+explicit PTY root outside the canonical catalog. Fresh bootstrap is a separate
+cross-producer transaction because catalog EX cannot reserve a PTY registry
+against external producers. Hash-CAS permits declared live workspace facts and
+their real ancestry to contain content. It changes
+declaration leaves only; desired workspace facts must already exist, and
+workspace content and canonical state are never traversed, deleted, or hashed.
+When an identity path is absent, its complete bundle uses an exclusive
+directory rename. When its declared workspace skeleton already exists, the
+durable marker fences declaration readers and marker-time state routing until
+every declaration leaf has been published and verified. Applied leaves and
+their parents are fsynced, the live root is re-hashed and fully admitted, then
+the marker is unlinked and `.st2` is fsynced.
+The catalog parent is fsynced when `.st2` is first created, including the
+concurrent create/observe race. Retained source capture rejects a staging
+destination contained by its source before enumerating that source.
+
 The lock file is never removed: replacing its inode would split the transaction
 domain for processes that already hold it open. Reconciliation holds SH from
 discovery through execution. Validation, doctor, roster, listing,
@@ -192,18 +263,34 @@ transaction fence. Any presence is authoritative, including malformed content.
 The reserved canonical record is:
 
 ```json
-{"schema":"st2.catalog-apply-incomplete.v1","desiredStorePath":"/nix/store/..."}
+{"schema":"st2.catalog-apply-incomplete.v1","stageName":"catalog-apply-stage-<prepared-root-sha256>","expectedRootSha256":"<previous-root-sha256>","preparedRootSha256":"<prepared-root-sha256>","originalPaths":["<sorted-owned-declaration-leaf>", "..."]}
 ```
 
 After taking its authoring lock, st2 refuses publication, validation,
 materialization, teardown, roster, doctor, and catalog listing while the marker
 exists. One-shot and selected reconcile fail explicitly. A resident supervisor
 instead remains alive, reports a skipped/incomplete pass, and performs no
 runtime observation or lifecycle action, avoiding a service restart storm.
-Message, context, Resource, and status operations remain available. The future
-whole-catalog apply command owns marker creation, admission, durable apply,
-verification, and clearing inside one transaction; external lock execution and
-bypass flags are not part of the contract.
+Message, context, Resource, and status operations remain available. While the
+marker exists they resolve canonical state from a validated address book: the
+marker's original canonical agent keys union currently published real specs.
+State-only directories are addressable only for original keys with recognized
+real state; an incomplete or arbitrary new identity does not fall back to a
+flat bus. Every host, identity, and message-box path is opened component by
+component without following symlinks, and state mutations remain relative to
+those retained capabilities.
+A dotted bare identity is tried as
+the complete local identity alongside every possible qualified bus-address
+split; exactly one distinct canonical address must exist. Only real state
+directories and a real regular status file can establish marker-time
+addressability. Only `catalog apply --resume --catalog ROOT --json` may open an
+existing marker. The closed marker and internal content-addressed stage are
+sufficient recovery authority; the original prepared path and CAS precondition are
+neither required nor consulted. Marker authority proves the original
+precondition already passed, so recovery converges the partial live tree from
+the durable desired stage and original owned-leaf list without re-enforcing
+that stale precondition. Malformed or mismatched records remain fenced.
+External lock execution and bypass flags are not part of the contract.
 
 ## Host-local scheduling and supervision
 
@@ -300,9 +387,13 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
   time, and opaque generation id derived from stable backend evidence.
 
   Discovery runs before and after runtime observation. A semantic declaration
-  change across those passes makes the result incomplete. This detects
-  observed drift but does not serialize catalog writers or claim a
-  transactional snapshot. A runtime root positively absent at admission is
+  change across those passes makes the result incomplete. The reader also
+  samples `<catalog>/.st2/catalog-generation` and the incomplete marker around
+  discovery and runtime observation. Every successful declaration writer
+  advances and fsyncs that monotonic generation after its durable commit; apply
+  does so after live verification and before clearing its marker. Even a
+  completed declaration ABA is therefore incomplete. This remains an observational
+  seqlock and does not serialize catalog writers. A runtime root positively absent at admission is
   empty and is not passed to its backend. An admitted PTY root that is removed
   or replaced during `pty list` is indeterminate; because the external backend
   creates an absent registry, concurrent root deletion is not a zero-write

examples/README.md

@@ -13,7 +13,9 @@ and gate the result before starting a process:
 ```sh
 st2 hooks install
 st2 hooks verify
-st2 agent publish --catalog <catalog> --bundle <bundle> --expect-absent --json
+input_sha256="$(st2 agent digest --bundle <bundle>)"
+st2 agent publish --catalog <catalog> --bundle <bundle> \
+  --input-sha256 "$input_sha256" --expect-absent --json
 st2 validate --catalog <catalog>
 st2 up --catalog <catalog> --host <host> --materialize-only
 st2 up --catalog <catalog> --host <host> --once

examples/native/README.md

@@ -25,7 +25,9 @@ Use this sequence:
 ```sh
 st2 hooks install
 st2 hooks verify
-st2 agent publish --catalog <catalog> --bundle <bundle> --expect-absent --json
+input_sha256="$(st2 agent digest --bundle <bundle>)"
+st2 agent publish --catalog <catalog> --bundle <bundle> \
+  --input-sha256 "$input_sha256" --expect-absent --json
 st2 validate <catalog>
 st2 up <catalog> --host <host> --materialize-only
 st2 up <catalog> --host <host> --once