sync: absent from my scan does not mean deleted - #75
Merged
Conversation
Shipping delete propagation removed THIRTEEN LIVE FILES from three machines on 2026-08-25. Every one was recoverable from git, which is the only reason this is a bad night and not a disaster. This is the fix and the test that reproduces it. WHAT HAPPENED. Earlier that evening `plans/**` was removed from an entry's `include` as part of a migration. That left about twenty paths recorded in the entry's manifest and observed set while its scan could no longer see them. Inert under a policy that does not propagate deletes. Then propagation was turned on. To the entry those paths read as "in my records, absent from my scan", which is precisely what a local delete looks like, so it tombstoned them and the real files were removed everywhere. THE TWO CASES ARE INDISTINGUISHABLE FROM INSIDE THAT LOOP, and only one of them is a delete. Narrowing an include is a SCOPE change. Nobody deleted anything. THE FIX. `scan_into_node_observed` no longer calls `local_remove` for a path the entry does not include. One condition. WHAT I DELIBERATELY DID NOT DO. The tidier half of the rule is to drop excluded paths from the manifest as well. That needs its own change: a peer whose config still selects the path sends it back on every reconcile, this side drops it again, and the whole index is rewritten each time. Fixing a delete by inventing a write loop is not a fix. Filtering on adopt is the likelier answer and it changes what crosses the wire. The comment says so at the site. THE TEST FAILED FIRST, on `main`: DROPPING A PATH FROM THE INCLUDE DELETED THE FILE. `a_path_dropped_from_include_is_forgotten_not_deleted` has a positive control, because a fixture that never recorded the file would prove nothing when the file survives. THE SECOND TEST IS A GUARD AND I SAY SO IN ITS DOC. `enabling_delete_propagation_does_not_delete_what_is_still_there` was written to reproduce a proposed cause, that enabling propagation replayed a backlog of old deletes. IT DOES NOT REPRODUCE, because that was not the cause. It passes before and after. I kept it rather than manufacture a failure: it proves the switch itself is not the dangerous part, so nobody has to wonder again. WHAT THE EVIDENCE SAID. Exactly the excluded set died. `_templates/**` and `agents/**/agent.kdl` survived, both still included, and there were zero deleted tracked files anywhere outside `plans/`. A replayed backlog would not land that precisely on paths excluded three hours earlier. MY ACTUAL ERROR, since it is the reusable part: I wrote a test for that migration, gave it a positive control, and ran it against the OLD policy, then shipped it with the new one. The right experiment against the wrong world. Agent: Silber.fabric
The second door into the same defect, and this one is already live. It
needs no configuration change and no release. It has been reachable the
whole time.
IF A WATCHED ROOT STOPS EXISTING, THE ENTRY IS DELETED ON EVERY PEER.
`scan_folder` opens with `if !root.exists() { return Ok(out) }`. A missing
root returns an EMPTY scan rather than an error. Every tracked path is
then in the observed set and absent from the scan, which is the shape of a
local delete, so a delete-propagating policy tombstones the whole entry
and sends that to everyone.
`st2-bus-default` is bus policy and bus has ALWAYS propagated deletes. That
entry carries 17,689 files across three machines.
A ROOT VANISHES WITHOUT ANYBODY DELETING ANYTHING: an unmounted volume, a
directory renamed while a pass runs, a mount not ready at boot, a
rename-swap of the directory by another process.
WHY THE BLAST RADIUS IS NOT ABOUT THE FILE COUNT. The bus entry carries
agent context and message history and git holds NONE of it. The catalog
repository tracks `agent.kdl`, `plans/` and `_templates/` only. When
thirteen files were lost earlier tonight they came back because git had
them. A bus mass delete has no such floor.
THE FIX. `scan_into_node_observed` returns early and changes nothing when
the root does not exist. Deleting the CONTENTS of a folder still
propagates normally, because the root survives that and the scan runs.
THE TEST FAILED FIRST. `a_vanished_root_is_not_a_mass_delete` records
three files under bus policy, removes the root directory, scans again, and
every file is tombstoned on `main`. It asserts against the policy that
actually propagates, not a hypothetical one.
I FOUND THIS BY FIXING MY OWN MISTAKE, which is the honest provenance. A
test suite written after an incident inherits the shape of that incident.
The class is "the thing I am watching stopped existing", and a narrowed
include and a vanished root are two doors into it. There may be a third.
Agent: Silber.fabric
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two doors into one defect. A path that is absent from an entry's scan is not necessarily a path anybody deleted, and treating the two as the same thing deletes live files on every peer.
Door one caused a real incident on 2026-08-25. Door two is already live on
mainand needs no configuration change or release to reach.Door one — a path removed from an
includeplans/**was taken out of an entry's include as part of a migration. That left ~20 paths recorded in the entry's manifest and observed set while its scan could no longer see them. Inert under a policy that does not propagate deletes.Then propagation was turned on. Those paths read as "in my records, absent from my scan" — exactly what a local delete looks like. Thirteen live files were removed from three machines. All were recoverable from git, which is the only reason it was a bad night rather than a disaster.
Narrowing an include is a scope change. Nobody deleted anything.
Fix:
scan_into_node_observedno longer callslocal_removefor a path the entry does not include.Test failed first on
main:Door two — the root itself stops existing
scan_folderopens withif !root.exists() { return Ok(out) }. A missing root returns an empty scan, not an error. Every tracked path is then absent from the scan, so a delete-propagating policy tombstones the whole entry and ships it to every peer.st2-bus-defaultis bus policy, and bus has always propagated deletes. That entry carries 17,689 files across three machines. A root vanishes without anyone deleting anything: an unmounted volume, a directory renamed mid-pass, a mount not ready at boot, a rename-swap by another process.The blast radius is not about the file count. The bus entry carries agent context and message history, and git holds none of it — the catalog repo tracks
agent.kdl,plans/and_templates/only. Tonight's thirteen files came back because git had them. A bus mass delete has no such floor.Fix:
scan_into_node_observedreturns early and changes nothing when the root does not exist. Deleting the contents of a folder still propagates normally, because the root survives that and the scan runs.Test failed first on
main:What I deliberately did not do
The tidier half of door one is to drop excluded paths from the manifest too. That needs its own change: a peer whose config still selects the path sends it back on every reconcile, this side drops it again, and the whole index is rewritten each time. Fixing a delete by inventing a write loop is not a fix. Filtering on
adoptis the likelier answer and it changes what crosses the wire. The comment says so at the site.A third test that is a guard, not a reproduction
enabling_delete_propagation_does_not_delete_what_is_still_therewas written to reproduce a proposed cause of the incident — that enabling propagation replayed a backlog of old deletes. It does not reproduce, because that was not the cause. It passes before and after. I kept it rather than manufacture a failure: it proves the switch itself is not the dangerous part.The discriminator that settled the real cause: exactly the excluded set died.
_templates/**andagents/**/agent.kdlsurvived, both still included, and there were zero deleted tracked files outsideplans/.Provenance, and the lesson
I found door two while fixing door one, which is the right way round. But: a test suite written after an incident inherits the shape of that incident. I built the folder-sync matrix around the failure I had just seen. The class is "the thing I am watching stopped existing", and these are two doors into it. There may be a third.
Green: lib 242, folder_sync 5, sync_slice 3.
Agent: Silber.fabric