sync: converge inherited catalog tombstones

Author: myobie

CHANGELOG.md

@@ -8,6 +8,11 @@ EXPERIMENTAL, so on-disk formats and the CLI may change without notice.
 
 ### Added
 
+- **Truthful sync status.** `fabric sync ls` now reports logical Present,
+  Tombstone, and observed-on-disk counts plus explicit missing, unexpected, and
+  content-mismatch drift. `fabric sync ls --json` provides the same state as a
+  stable machine-readable array.
+
 - **Safe st2 catalog sync recipe.** The README now provides a copy-pasteable,
   two-entry positive allow-list for declarations and bus data, with explicit
   machine-local PTY/runtime exclusions and an ordinary resource/message
@@ -55,6 +60,12 @@ EXPERIMENTAL, so on-disk formats and the CLI may change without notice.
 
 ### Fixed
 
+- **Inherited catalog tombstones no longer leave folders divergent.** Under
+  catalog policy, any surviving physical copy now advances an inherited
+  Tombstone to a higher Present version and supplies its content over the wire,
+  converging every materialized folder and persisted restart. Bus policy keeps
+  its existing delete semantics and removes the same stale bytes.
+
 - **macOS startup no longer blocks forever in CoreWLAN.** The optional Wi-Fi
   transmit-rate lookup is now bounded to 250 ms and cached per interface; a
   wedged synchronous CoreWLAN XPC request falls back to unknown speed without

README.md

@@ -764,6 +764,16 @@ sync only ever touches already-trusted peers — it adds no new trust surface.
   Peers that receive the tombstone remove the path. Tombstones are retained;
   automatic sweeping is not implemented.
 
+Catalog never originates tombstones, but persisted or wire state may inherit one
+from an older bus configuration. Fabric migrates that state using
+union-of-presence: if unchanged in-scope bytes still exist on any catalog node,
+that node advances them to a higher logical Present version and publishes their
+content, so every catalog manifest and materialized folder converges. Bus keeps
+the same tombstone authoritative and removes the stale bytes. If no catalog
+node retains a copy, Fabric cannot reconstruct deleted content; the inherited
+tombstone remains visible in `fabric sync ls` until an operator deliberately
+recreates the canonical file.
+
 Publication tools must treat every watcher-visible, included path as a durable
 logical sync key. Stage temporary, backup, and partial files **outside the
 configured sync folder** (on the same filesystem when an atomic rename is
@@ -799,6 +809,7 @@ rollback on the next reconcile.
 ```sh
 fabric sync add <folder> --name <name> [--peers "*"|a,b] [--policy catalog|bus] [--include "*.toml"]
 fabric sync ls
+fabric sync ls --json
 fabric sync rm <name-or-folder>
 fabric sync reload
 ```
@@ -809,6 +820,15 @@ the file to a running daemon, mirroring `reload-peers`. The daemon serves and
 dials sync over the reserved `fabric/sync/1` ALPN, gated by the same peer
 allow-list as every other fabric protocol.
 
+`fabric sync ls` reports `present` (logical files in the manifest),
+`tombstones` (retained logical deletions), and `observed` (included paths in the
+last durable local-disk receipt). `drift=clean` means the logical Present paths
+and observed bytes agree. A `drift=WARNING` names `missing` Present paths,
+`unexpected` observed paths whose manifest is tombstoned or absent, and
+`mismatched` paths whose observed content hash differs from the logical Present.
+`fabric sync ls --json` emits a stable array with those fields plus a Boolean
+`drift` for automation.
+
 ### Sync an st2 catalog safely
 
 An st2 catalog mixes declarative fleet data, durable bus data, and strictly
@@ -865,9 +885,11 @@ fabric status
 fabric ping <peer-name>
 ```
 
-`fabric sync ls` must show the same two logical names on every host, with the
-local catalog paths and intended peer selectors. `fabric status` and
-`fabric ping` must show the selected peers reachable.
+`fabric sync ls` must show the same two logical names and the same logical
+Present/Tombstone counts on every host, with the local catalog paths and
+intended peer selectors. In steady state `observed` equals `present` and
+`drift=clean`. `fabric status` and `fabric ping` must show the selected peers
+reachable.
 
 For the default Fabric home, inspect the effective include lists and fail if
 machine-local paths were added:

src/control.rs

@@ -131,7 +131,21 @@ pub struct SyncEntryStatus {
     pub folder: String,
     pub policy: String,
     pub peers: String,
+    /// Legacy logical-Present count retained on the local control wire so an
+    /// older client can still decode a newer daemon response.
     pub files: usize,
+    #[serde(default)]
+    pub present: usize,
+    #[serde(default)]
+    pub tombstones: usize,
+    #[serde(default)]
+    pub observed: usize,
+    #[serde(default)]
+    pub missing: usize,
+    #[serde(default)]
+    pub unexpected: usize,
+    #[serde(default)]
+    pub mismatched: usize,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]

src/daemon.rs

@@ -2444,7 +2444,13 @@ async fn process_control_request(
                         folder: status.folder.display().to_string(),
                         policy: status.policy.to_string(),
                         peers: peers_display(&status.peers),
-                        files: status.files,
+                        files: status.present,
+                        present: status.present,
+                        tombstones: status.tombstones,
+                        observed: status.observed,
+                        missing: status.missing,
+                        unexpected: status.unexpected,
+                        mismatched: status.mismatched,
                     })
                     .collect(),
                 None => Vec::new(),

src/main.rs

@@ -249,7 +249,11 @@ enum SyncCommands {
         include: Option<String>,
     },
     /// List configured sync entries and their live state.
-    Ls,
+    Ls {
+        /// Emit a stable JSON array for scripts.
+        #[arg(long)]
+        json: bool,
+    },
     /// Remove a sync entry by name or folder and reload the daemon.
     Rm { name_or_folder: String },
     /// Re-read syncs.toml into the running daemon (like reload-peers).
@@ -690,16 +694,42 @@ async fn run_sync(home: &FabricHome, command: SyncCommands) -> Result<()> {
             let _ = send_control(home, ControlRequest::SyncReload).await;
             println!("sync {name:?} written to {}", home.syncs_path().display());
         }
-        SyncCommands::Ls => match send_control(home, ControlRequest::SyncStatus).await? {
+        SyncCommands::Ls { json } => match send_control(home, ControlRequest::SyncStatus).await? {
             ControlResponse::SyncStatus { entries } => {
+                if json {
+                    let entries: Vec<_> = entries.iter().map(SyncLsJsonEntry::from).collect();
+                    println!("{}", serde_json::to_string_pretty(&entries)?);
+                    return Ok(());
+                }
                 if entries.is_empty() {
                     println!("no sync entries");
                 }
                 for entry in entries {
-                    println!(
-                        "{}\t{}\t{}\tpeers={}\t{} files",
-                        entry.name, entry.folder, entry.policy, entry.peers, entry.files
-                    );
+                    let present = logical_present(&entry);
+                    if entry.missing == 0 && entry.unexpected == 0 && entry.mismatched == 0 {
+                        println!(
+                            "{}\t{}\t{}\tpeers={}\tpresent={present}\ttombstones={}\tobserved={}\tdrift=clean",
+                            entry.name,
+                            entry.folder,
+                            entry.policy,
+                            entry.peers,
+                            entry.tombstones,
+                            entry.observed,
+                        );
+                    } else {
+                        println!(
+                            "{}\t{}\t{}\tpeers={}\tpresent={present}\ttombstones={}\tobserved={}\tdrift=WARNING missing={} unexpected={} mismatched={}",
+                            entry.name,
+                            entry.folder,
+                            entry.policy,
+                            entry.peers,
+                            entry.tombstones,
+                            entry.observed,
+                            entry.missing,
+                            entry.unexpected,
+                            entry.mismatched,
+                        );
+                    }
                 }
             }
             response => bail!("unexpected daemon response: {response:?}"),
@@ -721,6 +751,98 @@ async fn run_sync(home: &FabricHome, command: SyncCommands) -> Result<()> {
     Ok(())
 }
 
+#[derive(serde::Serialize)]
+struct SyncLsJsonEntry<'a> {
+    name: &'a str,
+    folder: &'a str,
+    policy: &'a str,
+    peers: &'a str,
+    present: usize,
+    tombstones: usize,
+    observed: usize,
+    drift: bool,
+    missing: usize,
+    unexpected: usize,
+    mismatched: usize,
+}
+
+impl<'a> From<&'a fabric::control::SyncEntryStatus> for SyncLsJsonEntry<'a> {
+    fn from(entry: &'a fabric::control::SyncEntryStatus) -> Self {
+        Self {
+            name: &entry.name,
+            folder: &entry.folder,
+            policy: &entry.policy,
+            peers: &entry.peers,
+            present: logical_present(entry),
+            tombstones: entry.tombstones,
+            observed: entry.observed,
+            drift: entry.missing != 0 || entry.unexpected != 0 || entry.mismatched != 0,
+            missing: entry.missing,
+            unexpected: entry.unexpected,
+            mismatched: entry.mismatched,
+        }
+    }
+}
+
+fn logical_present(entry: &fabric::control::SyncEntryStatus) -> usize {
+    if entry.present == 0 {
+        entry.files
+    } else {
+        entry.present
+    }
+}
+
+#[cfg(test)]
+mod sync_ls_tests {
+    use super::*;
+    use fabric::control::SyncEntryStatus;
+
+    fn status() -> SyncEntryStatus {
+        SyncEntryStatus {
+            name: "catalog".to_string(),
+            folder: "/catalog".to_string(),
+            policy: "catalog".to_string(),
+            peers: "*".to_string(),
+            files: 40,
+            present: 40,
+            tombstones: 3,
+            observed: 42,
+            missing: 0,
+            unexpected: 2,
+            mismatched: 0,
+        }
+    }
+
+    #[test]
+    fn sync_ls_json_schema_exposes_counts_and_drift() {
+        let status = status();
+        let json = serde_json::to_value(SyncLsJsonEntry::from(&status)).unwrap();
+        assert_eq!(
+            json,
+            serde_json::json!({
+                "name": "catalog",
+                "folder": "/catalog",
+                "policy": "catalog",
+                "peers": "*",
+                "present": 40,
+                "tombstones": 3,
+                "observed": 42,
+                "drift": true,
+                "missing": 0,
+                "unexpected": 2,
+                "mismatched": 0
+            })
+        );
+    }
+
+    #[test]
+    fn sync_ls_accepts_legacy_control_present_count() {
+        let mut status = status();
+        status.present = 0;
+        assert_eq!(logical_present(&status), 40);
+    }
+}
+
 fn absolutize(folder: &str) -> Result<PathBuf> {
     let path = PathBuf::from(folder);
     if path.is_absolute() {

src/sync/config.rs

@@ -25,11 +25,12 @@ use crate::config::FabricHome;
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
 #[serde(rename_all = "snake_case")]
 pub enum SyncPolicy {
-    /// Union + newer-wins + NEVER delete on a peer + no sweep + no tombstones.
+    /// Union + newer-wins + NEVER originate a delete + no sweep.
     ///
     /// Safe for a job catalog: a file present on any peer is present on all
-    /// peers, and nothing ever removes a file. Decommissioning is expressed as
-    /// an edit (e.g. `retired = true`), never a file deletion.
+    /// peers, and nothing ever removes a file. An inherited tombstone is
+    /// superseded when any physical copy survives. Decommissioning is expressed
+    /// as an edit (e.g. `retired = true`), never a file deletion.
     Catalog,
     /// Union + newer-wins + no delete + tombstone sweep. Reserved for the
     /// smalltalk bus; not yet fully implemented (see [`PolicyRules`]).