Skip to content

Commit 63563f4

Browse files
feat: add transactional catalog bootstrap
agent-session-id: ed878dac-3735-4276-b3e0-ea1b1cd65291 agent-tool: Codex CLI agent-tool-version: 0.145.0 agent-model: unknown agent-runtime-profile: /nix/store/mnx8agbdq3wiyb6vz63lhgscgazkrn98-coding-agent-runtime-profile/share/coding-agents/profile.json agent-skills-manifest: /nix/store/5r69m9k2llmri3na81518zx0a7y0d3cn-agent-skills-corpus/share/agent-skills/manifest.json tooling-profile: dotfiles@0fb7e03
1 parent 19d658b commit 63563f4

8 files changed

Lines changed: 1120 additions & 28 deletions

File tree

README.md

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -125,10 +125,21 @@ st2 catalog apply --catalog "$CATALOG" --prepared ./prepared \
125125
--expect-sha256 <rootSha256> --json
126126
```
127127

128+
To publish that exact snapshot as a new, absent catalog:
129+
130+
```sh
131+
st2 catalog bootstrap --catalog "$NEW_CATALOG" --prepared ./prepared \
132+
--input-sha256 <rootSha256> --json
133+
```
134+
128135
`catalog apply` is policy-free. It rejects state/control content, symlinks,
129136
unprojected workspace facts, catalog-local/default PTY roots, and effective
130-
PTY-root changes. Fresh bootstrap is a separate st2+pty transaction, not an
131-
apply mode. A crash leaves a durable marker and content-addressed stage;
137+
PTY-root changes. Bootstrap is a separate create-only declaration transaction,
138+
not an apply mode. It atomically publishes absence or the complete catalog,
139+
initializes its persistent lock and generation before visibility, and never
140+
reads or writes the external PTY registry. Process adoption and PTY-root
141+
migration remain separate because that registry has independent producers. A
142+
crash during apply leaves a durable marker and content-addressed stage;
132143
`st2 catalog apply --catalog "$CATALOG" --resume --json` resumes without the
133144
original prepared source. Snapshots own the complete bounded `_templates`
134145
library and empty canonical per-agent `.workspace` directory facts, but never
@@ -439,7 +450,7 @@ message, ding, agents, status, context, resource, rename, describe
439450
env, pty, shell, pretrust
440451
hooks, service, eval
441452
agent digest, agent publish
442-
catalog snapshot, catalog apply
453+
catalog bootstrap, catalog snapshot, catalog apply
443454
completions
444455
```
445456

@@ -505,3 +516,6 @@ evals retain their flat bus and completion semantics.
505516
(--prepared DIR --expect-sha256 ROOT_HEX | --resume)` is the complete
506517
declaration-plane writer. Each admits the complete prospective catalog under a
507518
compare-and-swap lock before making one atomic change.
519+
`st2 catalog bootstrap --catalog ROOT --prepared DIR --input-sha256 ROOT_HEX`
520+
is the create-only writer for an absent catalog. An exact completed replay is
521+
`unchanged`; any different or incomplete existing target fails closed.

docs/vrs/requirements.md

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -149,8 +149,10 @@ accepted.
149149
diagnostic readers therefore detect even a completed declaration ABA across
150150
their observation. A durable incomplete-generation intent fences readers
151151
across each single-writer commit and is conservatively recovered by the next
152-
exclusive writer. Writer staging exists only in the reserved control plane,
153-
never among authoritative declaration leaves.
152+
exclusive writer. Existing-catalog writer staging exists only in the reserved
153+
control plane, never among authoritative declaration leaves; fresh bootstrap
154+
stages one non-authoritative sibling because its control plane does not exist
155+
yet.
154156
Presence, messages, context, and Resource state remain independently writable
155157
and are never serialized behind catalog authoring.
156158
A caller binds single-agent publication to the exact no-follow source capture
@@ -162,9 +164,17 @@ accepted.
162164
digest under the exclusive lock, durably stages the desired bytes, and resumes
163165
after interruption solely from a closed marker and its content-addressed
164166
stage. Version 1 requires one explicit external PTY root and rejects effective
165-
PTY-root changes. Fresh-catalog bootstrap is a separate cross-producer
166-
transaction, not a catalog-apply mode. Apply never traverses, hashes, deletes,
167-
or relocates workspace or runtime state. An absent canonical identity becomes
167+
PTY-root changes. Fresh-catalog bootstrap is a distinct create transaction,
168+
not a catalog-apply mode: it binds an exact captured prepared projection to a
169+
caller-supplied digest, initializes the persistent authoring lock and first
170+
catalog generation before visibility, and publishes the complete catalog by
171+
one durable no-replace directory rename. A retry is unchanged only when the
172+
completed existing catalog has the exact prepared declaration root. Bootstrap
173+
validates and preserves one explicit external PTY root but performs no PTY
174+
registry I/O; process adoption and PTY-root migration remain outside its
175+
atomic boundary because the registry has independent producers. Apply never
176+
traverses, hashes, deletes, or relocates workspace or runtime state. An absent
177+
canonical identity becomes
168178
visible only as a complete bundle; a preexisting declared workspace skeleton
169179
remains safe because the durable marker fences declaration readers and
170180
marker-time state routing throughout leaf publication and verification.

docs/vrs/spec.md

Lines changed: 29 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -235,10 +235,8 @@ catalog-local/default PTY root, or effective PTY-root change. Hash-CAS captures
235235
and validates exact prepared bytes, takes EX, rechecks the canonical live root,
236236
and either reports `unchanged` for exact equality or creates a durable
237237
content-addressed stage before publishing the marker. Version 1 requires an
238-
explicit PTY root outside the canonical catalog. Fresh bootstrap is a separate
239-
cross-producer transaction because catalog EX cannot reserve a PTY registry
240-
against external producers. Hash-CAS permits declared live workspace facts and
241-
their real ancestry to contain content. It changes
238+
explicit PTY root outside the canonical catalog. Hash-CAS permits declared live
239+
workspace facts and their real ancestry to contain content. It changes
242240
declaration leaves only; desired workspace facts must already exist, and
243241
workspace content and canonical state are never traversed, deleted, or hashed.
244242
When an identity path is absent, its complete bundle uses an exclusive
@@ -292,6 +290,33 @@ the durable desired stage and original owned-leaf list without re-enforcing
292290
that stale precondition. Malformed or mismatched records remain fenced.
293291
External lock execution and bypass flags are not part of the contract.
294292

293+
`st2 catalog bootstrap --catalog ROOT --prepared DIR --input-sha256 HEX --json`
294+
is the create-only declaration transaction for an absent catalog. `ROOT` must
295+
be one absent final component below an existing canonical real parent. st2
296+
captures `DIR` through retained no-follow capabilities, verifies its declaration
297+
root against `HEX`, admits the complete projection against logical `ROOT`, and
298+
requires one explicit external PTY root. It materializes a 0700 sibling stage,
299+
creates the persistent authoring lock and generation `1` inside it, takes EX on
300+
that lock, fsyncs the complete tree, and publishes it with a capability-relative
301+
no-replace directory rename followed by a parent fsync. Readers therefore see
302+
absence or a complete catalog and cannot cross the already-published lock before
303+
the parent entry is durable.
304+
305+
There is no bootstrap marker or resume mode: interruption before the rename
306+
leaves `ROOT` absent, while interruption after it leaves the complete target. A
307+
retry re-captures its source and returns `unchanged` only after taking the
308+
existing lock, rejecting incomplete markers, proving a durable generation,
309+
fully validating the catalog, matching the exact declaration root, proving the
310+
locked directory remains bound to `ROOT`, and fsyncing the retained parent. A
311+
different, malformed, symlinked, rebound, or uninitialized existing target fails
312+
without mutation. Random sibling stages are non-authoritative and are cleaned
313+
only by the invocation that created them; no broad orphan cleanup is permitted.
314+
315+
Bootstrap performs zero reads or writes below the declared PTY root. The PTY
316+
registry has independent producers which catalog EX cannot reserve, so atomic
317+
process adoption, continuity, or PTY-root migration requires a separate PTY
318+
registry protocol. Bootstrap claims only atomic declaration publication.
319+
295320
## Host-local scheduling and supervision
296321

297322
```text

flake.nix

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,19 @@
135135
];
136136
});
137137

138+
# Bootstrap's crash/race tests use causal barrier hooks compiled only with debug
139+
# assertions. Keep the package's release-mode test boundary unchanged and gate this
140+
# transaction family in one explicit, non-vacuous derivation.
141+
st2CatalogBootstrap = st2.overrideAttrs (_: {
142+
pname = "st2-catalog-bootstrap-check";
143+
CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS = "true";
144+
cargoTestFlags = [
145+
"--test"
146+
"catalog_apply"
147+
"bootstrap_"
148+
];
149+
});
150+
138151
hookSuccessorSource = pkgs.runCommand "st2-hook-successor-source" { } ''
139152
cp -R ${self} $out
140153
chmod -R u+w $out
@@ -168,6 +181,7 @@
168181
# wants them.
169182
checks.st2 = st2;
170183
checks.atomic-pty-snapshot = st2AtomicPtySnapshot;
184+
checks.catalog-bootstrap = st2CatalogBootstrap;
171185

172186
# Real producer-consumer contract: st2 consumes `pty list --json` from the exact pty
173187
# revision that owns fleet observation. Fake CLI fixtures below still cover malformed

src/catalog_lock.rs

Lines changed: 52 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -146,25 +146,36 @@ enum Mode {
146146
pub struct CatalogLock {
147147
file: File,
148148
control: File,
149+
root: File,
149150
}
150151

151152
impl CatalogLock {
152153
pub fn shared(catalog: &Path) -> Result<Self> {
153-
Self::acquire(catalog, Mode::Shared, false)
154+
Self::acquire(catalog, Mode::Shared, false, true)
155+
}
156+
157+
/// Acquire the existing catalog lock without initializing missing control state.
158+
pub(crate) fn shared_existing(catalog: &Path) -> Result<Self> {
159+
Self::acquire(catalog, Mode::Shared, false, false)
154160
}
155161

156162
pub fn exclusive(catalog: &Path) -> Result<Self> {
157-
Self::acquire(catalog, Mode::Exclusive, false)
163+
Self::acquire(catalog, Mode::Exclusive, false, true)
158164
}
159165

160166
/// The whole-catalog transaction is the only operation allowed to inspect and recover an
161167
/// incomplete apply. Every other declaration reader/writer must keep using `shared` or
162168
/// `exclusive`, which fail closed while the marker exists.
163169
pub(crate) fn exclusive_for_catalog_apply(catalog: &Path) -> Result<Self> {
164-
Self::acquire(catalog, Mode::Exclusive, true)
170+
Self::acquire(catalog, Mode::Exclusive, true, true)
165171
}
166172

167-
fn acquire(catalog: &Path, mode: Mode, allow_incomplete_apply: bool) -> Result<Self> {
173+
fn acquire(
174+
catalog: &Path,
175+
mode: Mode,
176+
allow_incomplete_apply: bool,
177+
initialize: bool,
178+
) -> Result<Self> {
168179
let catalog = catalog
169180
.canonicalize()
170181
.with_context(|| format!("canonicalize catalog root {}", catalog.display()))?;
@@ -175,6 +186,8 @@ impl CatalogLock {
175186
"catalog root is not a real directory: {}",
176187
catalog.display()
177188
);
189+
let root = crate::catalog_transaction::open_dir_beneath(&catalog, &catalog)
190+
.with_context(|| format!("open catalog root capability {}", catalog.display()))?;
178191

179192
let control = catalog.join(CONTROL_DIR);
180193
let control_branch = match fs::symlink_metadata(&control) {
@@ -187,6 +200,11 @@ impl CatalogLock {
187200
"observed"
188201
}
189202
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
203+
anyhow::ensure!(
204+
initialize,
205+
"catalog control directory is absent: {}",
206+
control.display()
207+
);
190208
test_control_creation_checkpoint();
191209
let branch = match fs::create_dir(&control) {
192210
Ok(()) => {
@@ -221,33 +239,45 @@ impl CatalogLock {
221239
// incomplete marker. Persist its catalog-parent entry before any caller can publish
222240
// declaration leaves. This is unconditional after observing a real control dir: its creator
223241
// may have crashed after mkdir but before its own parent fsync.
224-
File::open(&catalog)
225-
.with_context(|| format!("open catalog root {}", catalog.display()))?
226-
.sync_all()
242+
root.sync_all()
227243
.with_context(|| format!("sync catalog root {}", catalog.display()))?;
228244
#[cfg(debug_assertions)]
229245
if let Ok(path) = std::env::var("ST2_TEST_CATALOG_CONTROL_BRANCH") {
230246
let _ = fs::write(path, control_branch);
231247
}
232248

233-
let control_file = retained_control(&catalog)?
234-
.context("catalog control directory disappeared while acquiring its lock")?
235-
.0;
249+
let control_file = crate::catalog_transaction::openat_dir_nofollow(
250+
&root,
251+
std::ffi::OsStr::new(CONTROL_DIR),
252+
)
253+
.context("catalog control directory disappeared while acquiring its lock")?;
236254
let control = crate::catalog_transaction::retained_dir_path(&control_file)?;
237255
let path = control.join(LOCK_FILE);
238-
let file = OpenOptions::new()
256+
let mut options = OpenOptions::new();
257+
options
239258
.read(true)
240259
.write(true)
241-
.create(true)
242260
.mode(0o600)
243-
.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW)
261+
.custom_flags(libc::O_CLOEXEC | libc::O_NOFOLLOW);
262+
if initialize {
263+
options.create(true);
264+
}
265+
let file = options
244266
.open(&path)
245267
.with_context(|| format!("open catalog authoring lock {}", path.display()))?;
246268
let operation = match mode {
247269
Mode::Shared => libc::LOCK_SH,
248270
Mode::Exclusive => libc::LOCK_EX,
249271
};
250272
#[cfg(debug_assertions)]
273+
if let Ok(path) = std::env::var("ST2_TEST_CATALOG_LOCK_ANY_ATTEMPT") {
274+
let value = match mode {
275+
Mode::Shared => b"shared".as_slice(),
276+
Mode::Exclusive => b"exclusive".as_slice(),
277+
};
278+
let _ = fs::write(path, value);
279+
}
280+
#[cfg(debug_assertions)]
251281
if matches!(mode, Mode::Exclusive)
252282
&& let Ok(path) = std::env::var("ST2_TEST_CATALOG_LOCK_ATTEMPT")
253283
{
@@ -285,6 +315,7 @@ impl CatalogLock {
285315
let lock = Self {
286316
file,
287317
control: control_file,
318+
root,
288319
};
289320
if matches!(mode, Mode::Exclusive) {
290321
lock.recover_generation_intent()?;
@@ -297,6 +328,14 @@ impl CatalogLock {
297328
advance_generation(&self.control)
298329
}
299330

331+
pub(crate) fn generation(&self) -> Result<Option<u64>> {
332+
read_generation_from_control(&self.control)
333+
}
334+
335+
pub(crate) fn root(&self) -> &File {
336+
&self.root
337+
}
338+
300339
pub(crate) fn begin_generation_commit(&self) -> Result<GenerationCommit<'_>> {
301340
let control = crate::catalog_transaction::retained_dir_path(&self.control)?;
302341
let intent = control.join(GENERATION_INTENT_FILE);

0 commit comments

Comments
 (0)