Skip to content

sync: absent from my scan does not mean deleted - #75

Merged
myobie merged 2 commits into
mainfrom
sync/forget-excluded-paths
Aug 25, 2026
Merged

sync: absent from my scan does not mean deleted#75
myobie merged 2 commits into
mainfrom
sync/forget-excluded-paths

Conversation

@myobie

@myobie myobie commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Two doors into one defect. A path that is absent from an entry's scan is not necessarily a path anybody deleted, and treating the two as the same thing deletes live files on every peer.

Door one caused a real incident on 2026-08-25. Door two is already live on main and needs no configuration change or release to reach.


Door one — a path removed from an include

plans/** was taken out of an entry's include as part of a migration. That left ~20 paths recorded in the entry's manifest and observed set while its scan could no longer see them. Inert under a policy that does not propagate deletes.

Then propagation was turned on. Those paths read as "in my records, absent from my scan" — exactly what a local delete looks like. Thirteen live files were removed from three machines. All were recoverable from git, which is the only reason it was a bad night rather than a disaster.

Narrowing an include is a scope change. Nobody deleted anything.

Fix: scan_into_node_observed no longer calls local_remove for a path the entry does not include.

Test failed first on main:

DROPPING A PATH FROM THE INCLUDE DELETED THE FILE.

Door two — the root itself stops existing

scan_folder opens with if !root.exists() { return Ok(out) }. A missing root returns an empty scan, not an error. Every tracked path is then absent from the scan, so a delete-propagating policy tombstones the whole entry and ships it to every peer.

st2-bus-default is bus policy, and bus has always propagated deletes. That entry carries 17,689 files across three machines. A root vanishes without anyone deleting anything: an unmounted volume, a directory renamed mid-pass, a mount not ready at boot, a rename-swap by another process.

The blast radius is not about the file count. The bus entry carries agent context and message history, and git holds none of it — the catalog repo tracks agent.kdl, plans/ and _templates/ only. Tonight's thirteen files came back because git had them. A bus mass delete has no such floor.

Fix: scan_into_node_observed returns early and changes nothing when the root does not exist. Deleting the contents of a folder still propagates normally, because the root survives that and the scan runs.

Test failed first on main:

a root that vanished was read as a delete of every file in the entry,
and that tombstone set goes to every peer

What I deliberately did not do

The tidier half of door one is to drop excluded paths from the manifest too. That needs its own change: a peer whose config still selects the path sends it back on every reconcile, this side drops it again, and the whole index is rewritten each time. Fixing a delete by inventing a write loop is not a fix. Filtering on adopt is the likelier answer and it changes what crosses the wire. The comment says so at the site.

A third test that is a guard, not a reproduction

enabling_delete_propagation_does_not_delete_what_is_still_there was written to reproduce a proposed cause of the incident — that enabling propagation replayed a backlog of old deletes. It does not reproduce, because that was not the cause. It passes before and after. I kept it rather than manufacture a failure: it proves the switch itself is not the dangerous part.

The discriminator that settled the real cause: exactly the excluded set died. _templates/** and agents/**/agent.kdl survived, both still included, and there were zero deleted tracked files outside plans/.

Provenance, and the lesson

I found door two while fixing door one, which is the right way round. But: a test suite written after an incident inherits the shape of that incident. I built the folder-sync matrix around the failure I had just seen. The class is "the thing I am watching stopped existing", and these are two doors into it. There may be a third.

Green: lib 242, folder_sync 5, sync_slice 3.

Agent: Silber.fabric

myobie added 2 commits August 25, 2026 00:18
Shipping delete propagation removed THIRTEEN LIVE FILES from three
machines on 2026-08-25. Every one was recoverable from git, which is the
only reason this is a bad night and not a disaster. This is the fix and
the test that reproduces it.

WHAT HAPPENED. Earlier that evening `plans/**` was removed from an
entry's `include` as part of a migration. That left about twenty paths
recorded in the entry's manifest and observed set while its scan could no
longer see them. Inert under a policy that does not propagate deletes.

Then propagation was turned on. To the entry those paths read as "in my
records, absent from my scan", which is precisely what a local delete
looks like, so it tombstoned them and the real files were removed
everywhere.

THE TWO CASES ARE INDISTINGUISHABLE FROM INSIDE THAT LOOP, and only one
of them is a delete. Narrowing an include is a SCOPE change. Nobody
deleted anything.

THE FIX. `scan_into_node_observed` no longer calls `local_remove` for a
path the entry does not include. One condition.

WHAT I DELIBERATELY DID NOT DO. The tidier half of the rule is to drop
excluded paths from the manifest as well. That needs its own change: a
peer whose config still selects the path sends it back on every
reconcile, this side drops it again, and the whole index is rewritten
each time. Fixing a delete by inventing a write loop is not a fix.
Filtering on adopt is the likelier answer and it changes what crosses the
wire. The comment says so at the site.

THE TEST FAILED FIRST, on `main`:
  DROPPING A PATH FROM THE INCLUDE DELETED THE FILE.
`a_path_dropped_from_include_is_forgotten_not_deleted` has a positive
control, because a fixture that never recorded the file would prove
nothing when the file survives.

THE SECOND TEST IS A GUARD AND I SAY SO IN ITS DOC.
`enabling_delete_propagation_does_not_delete_what_is_still_there` was
written to reproduce a proposed cause, that enabling propagation replayed
a backlog of old deletes. IT DOES NOT REPRODUCE, because that was not the
cause. It passes before and after. I kept it rather than manufacture a
failure: it proves the switch itself is not the dangerous part, so nobody
has to wonder again.

WHAT THE EVIDENCE SAID. Exactly the excluded set died. `_templates/**`
and `agents/**/agent.kdl` survived, both still included, and there were
zero deleted tracked files anywhere outside `plans/`. A replayed backlog
would not land that precisely on paths excluded three hours earlier.

MY ACTUAL ERROR, since it is the reusable part: I wrote a test for that
migration, gave it a positive control, and ran it against the OLD policy,
then shipped it with the new one. The right experiment against the wrong
world.

Agent: Silber.fabric
The second door into the same defect, and this one is already live. It
needs no configuration change and no release. It has been reachable the
whole time.

IF A WATCHED ROOT STOPS EXISTING, THE ENTRY IS DELETED ON EVERY PEER.

`scan_folder` opens with `if !root.exists() { return Ok(out) }`. A missing
root returns an EMPTY scan rather than an error. Every tracked path is
then in the observed set and absent from the scan, which is the shape of a
local delete, so a delete-propagating policy tombstones the whole entry
and sends that to everyone.

`st2-bus-default` is bus policy and bus has ALWAYS propagated deletes. That
entry carries 17,689 files across three machines.

A ROOT VANISHES WITHOUT ANYBODY DELETING ANYTHING: an unmounted volume, a
directory renamed while a pass runs, a mount not ready at boot, a
rename-swap of the directory by another process.

WHY THE BLAST RADIUS IS NOT ABOUT THE FILE COUNT. The bus entry carries
agent context and message history and git holds NONE of it. The catalog
repository tracks `agent.kdl`, `plans/` and `_templates/` only. When
thirteen files were lost earlier tonight they came back because git had
them. A bus mass delete has no such floor.

THE FIX. `scan_into_node_observed` returns early and changes nothing when
the root does not exist. Deleting the CONTENTS of a folder still
propagates normally, because the root survives that and the scan runs.

THE TEST FAILED FIRST. `a_vanished_root_is_not_a_mass_delete` records
three files under bus policy, removes the root directory, scans again, and
every file is tombstoned on `main`. It asserts against the policy that
actually propagates, not a hypothetical one.

I FOUND THIS BY FIXING MY OWN MISTAKE, which is the honest provenance. A
test suite written after an incident inherits the shape of that incident.
The class is "the thing I am watching stopped existing", and a narrowed
include and a vanished root are two doors into it. There may be a third.

Agent: Silber.fabric
@myobie myobie changed the title sync: a path that leaves an include is not a file anybody deleted sync: absent from my scan does not mean deleted Aug 24, 2026
@myobie
myobie merged commit 2e36734 into main Aug 25, 2026
2 checks passed
@myobie
myobie deleted the sync/forget-excluded-paths branch August 25, 2026 06:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant