diff --git a/README.md b/README.md index 162f5904..94150775 100644 --- a/README.md +++ b/README.md @@ -125,6 +125,23 @@ st2 catalog apply --catalog "$CATALOG" --prepared ./prepared \ --expect-sha256 --json ``` +If the incumbent Agent Specs cannot be parsed, bind a one-time repair to their +exact structural declaration bytes instead: + +```sh +st2 catalog snapshot --catalog "$CATALOG" --output ./invalid-preimage \ + --raw-preimage --json +# Produce a fully valid ./prepared directory from that capture. +st2 catalog apply --catalog "$CATALOG" --prepared ./prepared \ + --expect-sha256 --raw-preimage --json +``` + +Raw-preimage mode has its own hash and receipt schemas. It refuses a +strictly-valid incumbent, still fully validates the prepared and applied +catalogs, and requires a readable external PTY-root declaration that remains +unchanged. It is a generic invalid-preimage transaction, not a validation +bypass or migration-policy engine. + To publish that exact snapshot as a new, absent catalog: ```sh @@ -580,7 +597,7 @@ evals retain their flat bus and completion semantics. `st2 agent publish --catalog ROOT (--spec FILE | --bundle DIR) --input-sha256 HEX (--expect-absent | --expect-sha256 HEX)` is the single-agent declaration writer. `st2 catalog apply --catalog ROOT -(--prepared DIR --expect-sha256 ROOT_HEX | --resume)` is the complete +(--prepared DIR --expect-sha256 ROOT_HEX [--raw-preimage] | --resume)` is the complete declaration-plane writer. Each admits the complete prospective catalog under a compare-and-swap lock before making one atomic change. `st2 catalog bootstrap --catalog ROOT --prepared DIR --input-sha256 ROOT_HEX` diff --git a/docs/vrs/requirements.md b/docs/vrs/requirements.md index 96b35792..b9336a13 100644 --- a/docs/vrs/requirements.md +++ b/docs/vrs/requirements.md @@ -192,8 +192,12 @@ accepted. facts. Whole-catalog apply accepts only that projection, rechecks the root digest under the exclusive lock, durably stages the desired bytes, and resumes after interruption solely from a closed marker and its content-addressed - stage. Version 1 requires one explicit external PTY root and rejects effective - PTY-root changes. Fresh-catalog bootstrap is a distinct create transaction, + stage. A distinct raw-preimage projection may bind a repair to the exact + structural declaration bytes of an invalid incumbent without interpreting + those bytes. It has a separate hash and receipt type, refuses a strictly valid + incumbent, admits only a fully valid prepared result, and persists its mode in + the recovery marker. Version 1 requires one explicit external PTY root and + rejects effective PTY-root changes. Fresh-catalog bootstrap is a distinct create transaction, not a catalog-apply mode: it binds an exact captured prepared projection to a caller-supplied digest, initializes the persistent authoring lock and first catalog generation before visibility, and publishes the complete catalog by diff --git a/docs/vrs/spec.md b/docs/vrs/spec.md index 4f015b9e..70bb1f01 100644 --- a/docs/vrs/spec.md +++ b/docs/vrs/spec.md @@ -292,6 +292,21 @@ identical retry is `unchanged`. Its domain-separated, path-sorted root SHA-256 covers normalized relative paths, file bytes, executable bits, and empty workspace directory facts. +`st2 catalog snapshot --catalog ROOT --output DIR --raw-preimage --json` +exists only to externalize CAS for an invalid incumbent. Under the same shared +lock it structurally captures `catalog.kdl`, canonical +`agents///agent.kdl` leaves, their bounded static bundle files, +the bounded `_templates` tree, and existing canonical `.workspace` directory +facts without parsing Agent Spec bytes. The ordinary state/control exclusions +still apply. Every captured input must be a safe real file or directory with no +symlink or hard-link alias. The incumbent catalog envelope must parse and name +an external PTY root. A catalog that passes strict projection, live workspace +validation, and full admission is refused. Its root uses the distinct +`st2.catalog-raw-preimage-root.v1` hash domain and the receipt schema is +`st2.catalog-raw-preimage-snapshot.v1`; it is not interchangeable with a strict +snapshot root. Create-only retry rechecks both the raw root and output link +counts. + `st2 catalog diff --catalog ROOT --prepared DIR --expect-sha256 HEX --json` holds the existing authoring lock in shared mode and performs no initialization or publication. It projects and fully admits the coherent live catalog, rejects @@ -428,6 +443,22 @@ the durable desired stage and original owned-leaf list without re-enforcing that stale precondition. Malformed or mismatched records remain fenced. External lock execution and bypass flags are not part of the contract. +`st2 catalog apply --catalog ROOT --prepared DIR --expect-sha256 HEX +--raw-preimage --json` is the only writer that accepts the raw-preimage root. +It first captures and fully admits `DIR` through the ordinary strict prepared +projection. Under EX it refuses a strictly valid incumbent, requires the +incumbent catalog envelope to parse, proves the effective external PTY root is +unchanged, structurally reprojects the invalid live declaration plane, and +checks its raw-domain root against `HEX` before any declaration, workspace, +state, writer-temporary, marker, or stage mutation. A successful CAS reuses the +ordinary durable stage, generation commit, leaf publication, strict live +verification, and fsync sequence. Its receipt schema is +`st2.catalog-raw-preimage-apply.v1`; its durable marker schema is +`st2.catalog-raw-preimage-apply-incomplete.v1`. The marker schema preserves the +projection type, so source-free `--resume` emits the truthful raw-preimage +receipt after converging from the strictly validated stage. This mode owns no +policy for interpreting or transforming invalid bytes. + `st2 catalog bootstrap --catalog ROOT --prepared DIR --input-sha256 HEX --json` is the create-only declaration transaction for an absent catalog. `ROOT` must be one absent final component below an existing canonical real parent. st2 diff --git a/src/catalog_transaction.rs b/src/catalog_transaction.rs index 9685d0ec..fa81514e 100644 --- a/src/catalog_transaction.rs +++ b/src/catalog_transaction.rs @@ -21,7 +21,11 @@ const DIFF_SCHEMA: &str = "st2.catalog-diff.v1"; const BOOTSTRAP_SCHEMA: &str = "st2.catalog-bootstrap.v1"; const APPLY_SCHEMA: &str = "st2.catalog-apply.v1"; const MARKER_SCHEMA: &str = "st2.catalog-apply-incomplete.v1"; +const RAW_SNAPSHOT_SCHEMA: &str = "st2.catalog-raw-preimage-snapshot.v1"; +const RAW_APPLY_SCHEMA: &str = "st2.catalog-raw-preimage-apply.v1"; +const RAW_MARKER_SCHEMA: &str = "st2.catalog-raw-preimage-apply-incomplete.v1"; const HASH_DOMAIN: &[u8] = b"st2.catalog-declaration-root.v1\0"; +const RAW_HASH_DOMAIN: &[u8] = b"st2.catalog-raw-preimage-root.v1\0"; const STAGE_PREFIX: &str = "catalog-apply-stage-"; const WRITER_TEMP_PREFIXES: [&str; 3] = [ ".agent.kdl.presentation-", @@ -37,6 +41,7 @@ const TEMPLATE_MAX_TOTAL_BYTES: u64 = 32 * 1024 * 1024; pub struct SnapshotRequest { pub catalog: PathBuf, pub output: PathBuf, + pub raw_preimage: bool, } #[derive(Debug)] @@ -191,6 +196,10 @@ pub enum ApplyMode { prepared: PathBuf, expect_sha256: String, }, + RawPreimage { + prepared: PathBuf, + expect_sha256: String, + }, Resume, } @@ -1015,8 +1024,22 @@ pub fn snapshot(request: SnapshotRequest) -> Result { ); let _lock = CatalogLock::shared(&catalog)?; - let projection = project(&catalog, ProjectionSource::Current, &catalog)?; - validate_live_workspace_facts(&catalog, &projection.workspace_dirs)?; + let projection = if request.raw_preimage { + anyhow::ensure!( + !catalog_is_strictly_valid(&catalog), + "raw-preimage snapshot refuses an already-valid catalog" + ); + let incumbent_config = crate::catalog::load(&catalog) + .context("raw-preimage snapshot requires a valid incumbent catalog envelope")?; + validate_external_pty_root(&catalog, &incumbent_config, "raw-preimage snapshot v1")?; + let projection = project_raw_current(&catalog)?; + validate_projection_link_counts(&catalog, &projection, "raw live catalog")?; + projection + } else { + let projection = project(&catalog, ProjectionSource::Current, &catalog)?; + validate_live_workspace_facts(&catalog, &projection.workspace_dirs)?; + projection + }; match fs::symlink_metadata(&output) { Ok(metadata) => { anyhow::ensure!( @@ -1024,7 +1047,13 @@ pub fn snapshot(request: SnapshotRequest) -> Result { "snapshot output is not a real directory: {}", output.display() ); - let existing = project(&output, ProjectionSource::Prepared, &catalog)?; + let existing = if request.raw_preimage { + let existing = project_raw_current(&output)?; + validate_projection_link_counts(&output, &existing, "raw snapshot output")?; + existing + } else { + project(&output, ProjectionSource::Prepared, &catalog)? + }; anyhow::ensure!( existing.root_sha256 == projection.root_sha256, "snapshot output already exists with root sha256 {}, expected {}", @@ -1032,7 +1061,11 @@ pub fn snapshot(request: SnapshotRequest) -> Result { projection.root_sha256 ); return Ok(SnapshotResult { - schema: SNAPSHOT_SCHEMA, + schema: if request.raw_preimage { + RAW_SNAPSHOT_SCHEMA + } else { + SNAPSHOT_SCHEMA + }, status: SnapshotStatus::Unchanged, catalog, output, @@ -1057,7 +1090,11 @@ pub fn snapshot(request: SnapshotRequest) -> Result { sync_dir(&parent)?; Ok(SnapshotResult { - schema: SNAPSHOT_SCHEMA, + schema: if request.raw_preimage { + RAW_SNAPSHOT_SCHEMA + } else { + SNAPSHOT_SCHEMA + }, status: SnapshotStatus::Created, catalog, output, @@ -1336,45 +1373,70 @@ pub fn apply(request: ApplyRequest) -> Result { let captured = tempfile::tempdir().context("create prepared-catalog capture root")?; capture_prepared_catalog(&prepared, captured.path())?; let desired = project(captured.path(), ProjectionSource::Prepared, &catalog)?; - Some((prepared, expect_sha256, desired)) + Some((prepared, expect_sha256, desired, false)) + } + ApplyMode::RawPreimage { + prepared, + expect_sha256, + } => { + validate_sha256(&expect_sha256)?; + let prepared = canonical_real_dir_no_alias(&prepared, "prepared catalog")?; + anyhow::ensure!( + !prepared.starts_with(&catalog), + "prepared catalog must be outside the live catalog: {}", + prepared.display() + ); + let captured = tempfile::tempdir().context("create prepared-catalog capture root")?; + capture_prepared_catalog(&prepared, captured.path())?; + let desired = project(captured.path(), ProjectionSource::Prepared, &catalog)?; + Some((prepared, expect_sha256, desired, true)) } ApplyMode::Resume => None, }; let lock = CatalogLock::exclusive_for_catalog_apply(&catalog)?; let control = retained_dir_path(lock.control())?; - cleanup_writer_temporaries(&catalog)?; let marker_path = control.join(APPLY_MARKER); let existing_marker = read_marker_optional(&marker_path)?; let recovered = existing_marker.is_some(); - let (prepared, expect_sha256, desired, marker) = match (prepared_input, existing_marker) { - (Some(_), Some(_)) => { - anyhow::bail!("catalog apply is incomplete; recover only with `catalog apply --resume`") - } - (Some((prepared, expect_sha256, desired)), None) => { - (Some(prepared), expect_sha256, desired, None) - } - (None, Some(marker)) => { - validate_marker(&marker)?; - let stage_path = control.join(&marker.stage_name); - let staged = project(&stage_path, ProjectionSource::Prepared, &catalog) - .context("validate durable recovery stage")?; - anyhow::ensure!( - staged.root_sha256 == marker.prepared_root_sha256, - "durable recovery stage hash mismatch: expected {}, found {}", - marker.prepared_root_sha256, - staged.root_sha256 - ); - ( - None, - marker.expected_root_sha256.clone(), - staged, - Some(marker), - ) - } - (None, None) => anyhow::bail!("catalog apply --resume requires an incomplete apply marker"), - }; + let (prepared, expect_sha256, desired, marker, raw_preimage) = + match (prepared_input, existing_marker) { + (Some(_), Some(_)) => { + anyhow::bail!( + "catalog apply is incomplete; recover only with `catalog apply --resume`" + ) + } + (Some((prepared, expect_sha256, desired, raw_preimage)), None) => { + (Some(prepared), expect_sha256, desired, None, raw_preimage) + } + (None, Some(marker)) => { + validate_marker(&marker)?; + let stage_path = control.join(&marker.stage_name); + let staged = project(&stage_path, ProjectionSource::Prepared, &catalog) + .context("validate durable recovery stage")?; + anyhow::ensure!( + staged.root_sha256 == marker.prepared_root_sha256, + "durable recovery stage hash mismatch: expected {}, found {}", + marker.prepared_root_sha256, + staged.root_sha256 + ); + let raw_preimage = marker.schema == RAW_MARKER_SCHEMA; + ( + None, + marker.expected_root_sha256.clone(), + staged, + Some(marker), + raw_preimage, + ) + } + (None, None) => { + anyhow::bail!("catalog apply --resume requires an incomplete apply marker") + } + }; + if recovered { + cleanup_writer_temporaries(&catalog)?; + } validate_live_workspace_facts(&catalog, &desired.workspace_dirs)?; // Admission reads exact durable/captured declaration bytes. Catalog-contained workspace facts // are mirrored as empty directories; their live content is never copied or hashed. @@ -1393,18 +1455,42 @@ pub fn apply(request: ApplyRequest) -> Result { ); (expect_sha256.clone(), marker.original_paths, None) } else { - let current = project_excluding( - &catalog, - ProjectionSource::Current, - &catalog, - &desired.workspace_dirs, - )?; - let live_config = crate::catalog::load(&catalog)?; + if raw_preimage { + anyhow::ensure!( + !catalog_is_strictly_valid(&catalog), + "raw-preimage apply refuses an already-valid catalog" + ); + } + let live_config = if raw_preimage { + crate::catalog::load(&catalog) + .context("raw-preimage apply requires a valid incumbent catalog envelope")? + } else { + crate::catalog::load(&catalog)? + }; let same_pty_root = effective_pty_root(&catalog, &live_config) == effective_pty_root(&catalog, &desired_config); + if !raw_preimage { + cleanup_writer_temporaries(&catalog)?; + } + let current = if raw_preimage { + let current = project_raw_current(&catalog)?; + validate_projection_link_counts(&catalog, ¤t, "raw live catalog")?; + current + } else { + project_excluding( + &catalog, + ProjectionSource::Current, + &catalog, + &desired.workspace_dirs, + )? + }; if current.root_sha256 == desired.root_sha256 && same_pty_root { return Ok(ApplyResult { - schema: APPLY_SCHEMA, + schema: if raw_preimage { + RAW_APPLY_SCHEMA + } else { + APPLY_SCHEMA + }, status: ApplyStatus::Unchanged, catalog, prepared, @@ -1424,12 +1510,20 @@ pub fn apply(request: ApplyRequest) -> Result { expect_sha256, current.root_sha256 ); + if raw_preimage { + cleanup_writer_temporaries(&catalog)?; + } let original_paths = current.files.keys().cloned().collect::>(); ensure_durable_stage(lock.control(), &catalog, &stage_name, &desired)?; write_marker( lock.control(), &ApplyMarker { - schema: MARKER_SCHEMA.to_string(), + schema: if raw_preimage { + RAW_MARKER_SCHEMA + } else { + MARKER_SCHEMA + } + .to_string(), stage_name: stage_name.clone(), expected_root_sha256: expect_sha256.clone(), prepared_root_sha256: desired.root_sha256.clone(), @@ -1468,7 +1562,11 @@ pub fn apply(request: ApplyRequest) -> Result { let _ = lock.control().sync_all(); Ok(ApplyResult { - schema: APPLY_SCHEMA, + schema: if raw_preimage { + RAW_APPLY_SCHEMA + } else { + APPLY_SCHEMA + }, status: ApplyStatus::Applied, catalog, prepared, @@ -1521,6 +1619,76 @@ fn format_issue(issue: &crate::validate::Issue) -> String { format!("{} [{}]: {}", issue.path, issue.code, issue.message) } +fn catalog_is_strictly_valid(root: &Path) -> bool { + project(root, ProjectionSource::Current, root) + .and_then(|projection| { + validate_live_workspace_facts(root, &projection.workspace_dirs)?; + validate_full_catalog(root) + }) + .is_ok() +} + +/// Project the declaration plane without interpreting declaration bytes. +/// +/// This exists solely to bind a repair transaction to the exact bytes of an invalid current +/// catalog. It deliberately has no policy for why those bytes are invalid. Mutable agent state is +/// excluded by the same structural boundaries as the strict projection; a prospective catalog is +/// never admitted through this path. +fn project_raw_current(root: &Path) -> Result { + let metadata = fs::symlink_metadata(root)?; + anyhow::ensure!( + metadata.is_dir() && !metadata.file_type().is_symlink(), + "raw projection root is not a real directory: {}", + root.display() + ); + let mut files = BTreeMap::new(); + add_optional_regular(root, &root.join(crate::catalog::CONFIG_FILE), &mut files)?; + let spec_paths = collect_canonical_specs(root, ProjectionSource::Current, &mut files)?; + let workspace_dirs = raw_workspace_dirs(root, &spec_paths)?; + for spec in &spec_paths { + let bundle = spec.parent().context("canonical spec has no bundle")?; + collect_bundle_files( + root, + bundle, + bundle, + ProjectionSource::Current, + &workspace_dirs, + &mut files, + )?; + } + collect_templates(root, ProjectionSource::Current, &mut files)?; + let root_sha256 = hash_raw_projection(&files, &workspace_dirs); + Ok(DeclarationProjection { + files, + workspace_dirs, + root_sha256, + }) +} + +fn raw_workspace_dirs(root: &Path, spec_paths: &[PathBuf]) -> Result> { + let mut workspace_dirs = BTreeSet::new(); + for spec in spec_paths { + let bundle = spec.parent().context("canonical spec has no bundle")?; + let workspace = bundle.join(".workspace"); + match fs::symlink_metadata(&workspace) { + Ok(metadata) => { + anyhow::ensure!( + metadata.is_dir() && !metadata.file_type().is_symlink(), + "canonical workspace fact is not a real directory: {}", + workspace.display() + ); + workspace_dirs.insert(normalized_relative(root, &workspace)?); + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => { + return Err(error) + .with_context(|| format!("inspect workspace fact {}", workspace.display())); + } + } + } + Ok(workspace_dirs) +} + fn project( root: &Path, source: ProjectionSource, @@ -2019,6 +2187,28 @@ fn hash_projection( format!("{:x}", hasher.finalize()) } +fn hash_raw_projection( + files: &BTreeMap, + workspace_dirs: &BTreeSet, +) -> String { + let mut hasher = Sha256::new(); + hasher.update(RAW_HASH_DOMAIN); + for (path, file) in files { + hasher.update([1]); + hasher.update((path.len() as u64).to_be_bytes()); + hasher.update(path.as_bytes()); + hasher.update([u8::from(file.executable)]); + hasher.update((file.bytes.len() as u64).to_be_bytes()); + hasher.update(&file.bytes); + } + for path in workspace_dirs { + hasher.update([2]); + hasher.update((path.len() as u64).to_be_bytes()); + hasher.update(path.as_bytes()); + } + format!("{:x}", hasher.finalize()) +} + fn materialize_projection(projection: &DeclarationProjection, root: &Path) -> Result<()> { for (relative, file) in &projection.files { let target = root.join(relative); @@ -2359,7 +2549,7 @@ fn read_marker_optional(path: &Path) -> Result> { fn validate_marker(marker: &ApplyMarker) -> Result<()> { anyhow::ensure!( - marker.schema == MARKER_SCHEMA, + matches!(marker.schema.as_str(), MARKER_SCHEMA | RAW_MARKER_SCHEMA), "unsupported catalog apply marker schema '{}'", marker.schema ); diff --git a/src/main.rs b/src/main.rs index 508a7158..af5b2d3a 100644 --- a/src/main.rs +++ b/src/main.rs @@ -461,6 +461,10 @@ enum CatalogCmd { /// Destination directory. It must be outside the live catalog. #[arg(long, value_name = "DIR")] output: PathBuf, + /// Hash and capture the declaration plane without parsing it. Only for repairing an + /// invalid catalog; the captured directory remains unvalidated. + #[arg(long)] + raw_preimage: bool, /// Emit the typed snapshot receipt as JSON. #[arg(long)] json: bool, @@ -483,6 +487,10 @@ enum CatalogCmd { conflicts_with = "resume" )] expect_sha256: Option, + /// Match the current declaration plane without parsing it. The prepared catalog is still + /// fully validated, and this mode refuses an already-valid current catalog. + #[arg(long, conflicts_with = "resume")] + raw_preimage: bool, /// Resume the durable incomplete marker and internal stage without the original source. #[arg(long, conflicts_with_all = ["prepared", "expect_sha256"])] resume: bool, @@ -1040,11 +1048,16 @@ fn main() -> Result<()> { println!("{}", serde_json::to_string_pretty(&result)?); Ok(()) } - Command::Catalog(CatalogCmd::Snapshot { output, json }) => { + Command::Catalog(CatalogCmd::Snapshot { + output, + raw_preimage, + json, + }) => { let result = st2::catalog_transaction::snapshot(st2::catalog_transaction::SnapshotRequest { catalog: catalog_arg(None)?, output, + raw_preimage, })?; if json { println!("{}", serde_json::to_string_pretty(&result)?); @@ -1064,6 +1077,7 @@ fn main() -> Result<()> { Command::Catalog(CatalogCmd::Apply { prepared, expect_sha256, + raw_preimage, resume, json, }) => { @@ -1073,9 +1087,16 @@ fn main() -> Result<()> { let prepared = prepared.context("clap requires --prepared unless --resume")?; let expect_sha256 = expect_sha256.context("clap requires --expect-sha256 unless --resume")?; - st2::catalog_transaction::ApplyMode::Prepared { - prepared, - expect_sha256, + if raw_preimage { + st2::catalog_transaction::ApplyMode::RawPreimage { + prepared, + expect_sha256, + } + } else { + st2::catalog_transaction::ApplyMode::Prepared { + prepared, + expect_sha256, + } } }; let result = st2::catalog_transaction::apply(st2::catalog_transaction::ApplyRequest { diff --git a/tests/catalog_apply.rs b/tests/catalog_apply.rs index ae426c68..17164f27 100644 --- a/tests/catalog_apply.rs +++ b/tests/catalog_apply.rs @@ -37,6 +37,16 @@ fn write_agent_for_host(catalog: &Path, host: &str, identity: &str, retired: boo .unwrap(); } +fn write_invalid_agent(catalog: &Path, identity: &str) { + write_agent(catalog, identity, false); + let path = agent_dir(catalog, identity).join("agent.kdl"); + let invalid = fs::read_to_string(&path).unwrap().replace( + " retired #false\n", + " desired-state \"running\" because=\"unsupported\"\n", + ); + fs::write(path, invalid).unwrap(); +} + fn ensure_external_pty_config(catalog: &Path) { let config = catalog.join("catalog.kdl"); if !config.exists() { @@ -70,6 +80,23 @@ fn snapshot(catalog: &Path, output: &Path) -> Value { serde_json::from_slice(&result.stdout).unwrap() } +fn raw_snapshot(catalog: &Path, output: &Path) -> Output { + ensure_external_pty_config(catalog); + st2() + .args([ + "catalog", + "snapshot", + "--catalog", + catalog.to_str().unwrap(), + "--output", + output.to_str().unwrap(), + "--raw-preimage", + "--json", + ]) + .output() + .unwrap() +} + fn apply(catalog: &Path, prepared: &Path, expected: &str) -> Output { st2() .args([ @@ -87,6 +114,24 @@ fn apply(catalog: &Path, prepared: &Path, expected: &str) -> Output { .unwrap() } +fn raw_apply(catalog: &Path, prepared: &Path, expected: &str) -> Output { + st2() + .args([ + "catalog", + "apply", + "--catalog", + catalog.to_str().unwrap(), + "--prepared", + prepared.to_str().unwrap(), + "--expect-sha256", + expected, + "--raw-preimage", + "--json", + ]) + .output() + .unwrap() +} + fn bootstrap(catalog: &Path, prepared: &Path, input_sha256: &str) -> Output { st2() .args([ @@ -119,11 +164,17 @@ fn resume(catalog: &Path) -> Output { } #[test] -fn catalog_apply_cli_exposes_exactly_the_two_closed_modes() { +fn catalog_apply_cli_exposes_exactly_the_three_closed_modes() { let help = st2().args(["catalog", "apply", "--help"]).output().unwrap(); assert!(help.status.success()); let help = String::from_utf8(help.stdout).unwrap(); - for flag in ["--prepared", "--expect-sha256", "--resume", "--json"] { + for flag in [ + "--prepared", + "--expect-sha256", + "--raw-preimage", + "--resume", + "--json", + ] { assert!(help.contains(flag), "catalog apply help omitted {flag}"); } assert!(!help.contains("--expect-absent")); @@ -145,6 +196,7 @@ fn catalog_apply_cli_exposes_exactly_the_two_closed_modes() { "--expect-sha256", "0000000000000000000000000000000000000000000000000000000000000000", ], + vec!["catalog", "apply", "--resume", "--raw-preimage"], ] { let rejected = st2().args(args).output().unwrap(); assert!( @@ -937,6 +989,272 @@ fn snapshot_is_typed_deterministic_and_excludes_state_and_workspaces() { assert_eq!(second["rootSha256"], first["rootSha256"]); } +#[test] +fn raw_preimage_repairs_an_invalid_catalog_and_preserves_mutable_state() { + let temp = tempfile::tempdir().unwrap(); + let catalog = temp.path().join("catalog"); + write_invalid_agent(&catalog, "worker"); + ensure_external_pty_config(&catalog); + let dir = agent_dir(&catalog, "worker"); + fs::create_dir_all(dir.join("resources/inbox")).unwrap(); + fs::write( + dir.join("resources/inbox/message.md"), + "keep resource state", + ) + .unwrap(); + fs::create_dir_all(dir.join("archive")).unwrap(); + fs::write(dir.join("archive/old.md"), "keep archive state").unwrap(); + fs::write(dir.join("status"), "busy").unwrap(); + + let strict_snapshot = st2() + .args([ + "catalog", + "snapshot", + "--catalog", + catalog.to_str().unwrap(), + "--output", + temp.path().join("strict-invalid").to_str().unwrap(), + "--json", + ]) + .output() + .unwrap(); + assert!(!strict_snapshot.status.success()); + + let raw_capture_dir = temp.path().join("raw-capture"); + let raw_capture = raw_snapshot(&catalog, &raw_capture_dir); + assert!( + raw_capture.status.success(), + "{}", + String::from_utf8_lossy(&raw_capture.stderr) + ); + let raw_capture: Value = serde_json::from_slice(&raw_capture.stdout).unwrap(); + assert_eq!( + raw_capture["schema"], + "st2.catalog-raw-preimage-snapshot.v1" + ); + assert!( + !raw_capture_dir + .join("agents/host/worker/resources") + .exists() + ); + assert!(!raw_capture_dir.join("agents/host/worker/archive").exists()); + assert!(!raw_capture_dir.join("agents/host/worker/status").exists()); + + let desired_source = temp.path().join("desired-source"); + write_agent(&desired_source, "worker", false); + let prepared = temp.path().join("prepared"); + snapshot(&desired_source, &prepared); + + let strict = apply( + &catalog, + &prepared, + raw_capture["rootSha256"].as_str().unwrap(), + ); + assert!(!strict.status.success()); + assert!( + fs::read_to_string(dir.join("agent.kdl")) + .unwrap() + .contains("because=\"unsupported\"") + ); + + let repaired = raw_apply( + &catalog, + &prepared, + raw_capture["rootSha256"].as_str().unwrap(), + ); + assert!( + repaired.status.success(), + "{}", + String::from_utf8_lossy(&repaired.stderr) + ); + let repaired: Value = serde_json::from_slice(&repaired.stdout).unwrap(); + assert_eq!(repaired["schema"], "st2.catalog-raw-preimage-apply.v1"); + assert_eq!(repaired["status"], "applied"); + assert_eq!(repaired["beforeSha256"], raw_capture["rootSha256"]); + assert_eq!( + fs::read_to_string(dir.join("resources/inbox/message.md")).unwrap(), + "keep resource state" + ); + assert_eq!( + fs::read_to_string(dir.join("archive/old.md")).unwrap(), + "keep archive state" + ); + assert_eq!(fs::read_to_string(dir.join("status")).unwrap(), "busy"); + assert!( + !fs::read_to_string(dir.join("agent.kdl")) + .unwrap() + .contains("because=\"unsupported\"") + ); +} + +#[test] +fn raw_preimage_refuses_valid_catalogs_and_wrong_cas_without_declaration_writes() { + let temp = tempfile::tempdir().unwrap(); + let valid = temp.path().join("valid"); + write_agent(&valid, "worker", false); + let valid_prepared = temp.path().join("valid-prepared"); + let valid_snapshot = snapshot(&valid, &valid_prepared); + let valid_raw_snapshot = raw_snapshot(&valid, &temp.path().join("valid-raw")); + assert!(!valid_raw_snapshot.status.success()); + assert!( + String::from_utf8_lossy(&valid_raw_snapshot.stderr) + .contains("refuses an already-valid catalog") + ); + let valid_raw_apply = raw_apply( + &valid, + &valid_prepared, + valid_snapshot["rootSha256"].as_str().unwrap(), + ); + assert!(!valid_raw_apply.status.success()); + assert!( + String::from_utf8_lossy(&valid_raw_apply.stderr) + .contains("refuses an already-valid catalog") + ); + + let invalid = temp.path().join("invalid"); + write_invalid_agent(&invalid, "worker"); + ensure_external_pty_config(&invalid); + let declaration = agent_dir(&invalid, "worker").join("agent.kdl"); + let context = agent_dir(&invalid, "worker").join("resources/context/now.md"); + fs::create_dir_all(context.parent().unwrap()).unwrap(); + fs::write(&context, "state before wrong CAS").unwrap(); + let writer_temporary = agent_dir(&invalid, "worker").join(".agent.kdl.publish-test"); + fs::write(&writer_temporary, "unfinished writer bytes").unwrap(); + let before = fs::read(&declaration).unwrap(); + let wrong = raw_apply( + &invalid, + &valid_prepared, + "0000000000000000000000000000000000000000000000000000000000000000", + ); + assert!(!wrong.status.success()); + assert!(String::from_utf8_lossy(&wrong.stderr).contains("precondition failed")); + assert_eq!(fs::read(&declaration).unwrap(), before); + assert_eq!( + fs::read_to_string(&context).unwrap(), + "state before wrong CAS" + ); + assert_eq!( + fs::read_to_string(&writer_temporary).unwrap(), + "unfinished writer bytes" + ); + assert!(!invalid.join(".st2/catalog-apply-incomplete").exists()); + assert!(fs::read_dir(invalid.join(".st2")) + .unwrap() + .all(|entry| !entry + .unwrap() + .file_name() + .to_string_lossy() + .starts_with("catalog-apply-stage-"))); +} + +#[test] +fn raw_preimage_rejects_hard_linked_declarations() { + let temp = tempfile::tempdir().unwrap(); + let catalog = temp.path().join("catalog"); + write_invalid_agent(&catalog, "worker"); + ensure_external_pty_config(&catalog); + let declaration = agent_dir(&catalog, "worker").join("agent.kdl"); + fs::hard_link(&declaration, temp.path().join("alias.kdl")).unwrap(); + let rejected = raw_snapshot(&catalog, &temp.path().join("capture")); + assert!(!rejected.status.success()); + assert!(String::from_utf8_lossy(&rejected.stderr).contains("hard-linked")); +} + +#[test] +fn raw_preimage_requires_a_readable_envelope_and_an_unchanged_pty_root() { + let temp = tempfile::tempdir().unwrap(); + let malformed_envelope = temp.path().join("malformed-envelope"); + write_invalid_agent(&malformed_envelope, "worker"); + fs::write(malformed_envelope.join("catalog.kdl"), "catalog {").unwrap(); + let rejected = raw_snapshot( + &malformed_envelope, + &temp.path().join("malformed-capture"), + ); + assert!(!rejected.status.success()); + assert!(String::from_utf8_lossy(&rejected.stderr) + .contains("requires a valid incumbent catalog envelope")); + + let catalog = temp.path().join("catalog"); + write_invalid_agent(&catalog, "worker"); + ensure_external_pty_config(&catalog); + let raw_capture = raw_snapshot(&catalog, &temp.path().join("raw-capture")); + assert!(raw_capture.status.success()); + let raw_capture: Value = serde_json::from_slice(&raw_capture.stdout).unwrap(); + + let desired_source = temp.path().join("desired-source"); + write_agent(&desired_source, "worker", false); + fs::write( + desired_source.join("catalog.kdl"), + "catalog { pty-root \"/tmp/st2-catalog-transaction-other-pty\" }\n", + ) + .unwrap(); + let prepared = temp.path().join("prepared"); + snapshot(&desired_source, &prepared); + let declaration = fs::read(agent_dir(&catalog, "worker").join("agent.kdl")).unwrap(); + let rejected = raw_apply( + &catalog, + &prepared, + raw_capture["rootSha256"].as_str().unwrap(), + ); + assert!(!rejected.status.success()); + assert!(String::from_utf8_lossy(&rejected.stderr) + .contains("refuses an effective pty-root change")); + assert_eq!( + fs::read(agent_dir(&catalog, "worker").join("agent.kdl")).unwrap(), + declaration + ); + assert!(!catalog.join(".st2/catalog-apply-incomplete").exists()); +} + +#[test] +fn raw_preimage_resume_uses_the_durable_validated_stage() { + let temp = tempfile::tempdir().unwrap(); + let catalog = temp.path().join("catalog"); + write_invalid_agent(&catalog, "worker"); + ensure_external_pty_config(&catalog); + let raw_capture = raw_snapshot(&catalog, &temp.path().join("raw-capture")); + assert!(raw_capture.status.success()); + let raw_capture: Value = serde_json::from_slice(&raw_capture.stdout).unwrap(); + + let desired_source = temp.path().join("desired-source"); + write_agent(&desired_source, "worker", false); + let prepared = temp.path().join("prepared"); + snapshot(&desired_source, &prepared); + let interrupted = st2() + .args([ + "catalog", + "apply", + "--catalog", + catalog.to_str().unwrap(), + "--prepared", + prepared.to_str().unwrap(), + "--expect-sha256", + raw_capture["rootSha256"].as_str().unwrap(), + "--raw-preimage", + "--json", + ]) + .env("ST2_TEST_CATALOG_APPLY_CRASH_AT", "marker-created") + .output() + .unwrap(); + assert!(!interrupted.status.success()); + + fs::remove_dir_all(&prepared).unwrap(); + let recovered = resume(&catalog); + assert!( + recovered.status.success(), + "{}", + String::from_utf8_lossy(&recovered.stderr) + ); + let recovered: Value = serde_json::from_slice(&recovered.stdout).unwrap(); + assert_eq!(recovered["schema"], "st2.catalog-raw-preimage-apply.v1"); + assert_eq!(recovered["recovered"], true); + assert!( + !fs::read_to_string(agent_dir(&catalog, "worker").join("agent.kdl")) + .unwrap() + .contains("because=\"unsupported\"") + ); +} + #[test] fn complete_template_library_survives_unused_apply_and_supports_a_later_reference() { let temp = tempfile::tempdir().unwrap();