dofs, rpc: track renames in sync and make fetch progress a resumable cursor - #2
Closed
ndisidore wants to merge 3 commits into
Closed
dofs, rpc: track renames in sync and make fetch progress a resumable cursor#2ndisidore wants to merge 3 commits into
ndisidore wants to merge 3 commits into
Conversation
ndisidore
force-pushed
the
nathan/feat/track-rename
branch
from
June 10, 2026 17:14
ea3997e to
99d421f
Compare
Move provider renames through a filesystem primitive so local rename behavior and sync output share one implementation. A rename stamps the moved inode subtree with one revision and records tombstones for the old paths, letting the existing change stream represent moves without a new wire opcode. Directory renames are O(subtree) in database writes and wire entries. That cost is explicit in the protocol docs. The apply path also resolves type conflicts by replacing the local node tree with the upstream entry, and rm now unlinks final symlinks without following them to their targets. rename and rm resolve symlinked parents to a real path before they mutate, so the read-only mount guard is re-checked against that resolved source and destination. The earlier guard only saw the unresolved request, which let a symlink into a read-only mount carry a delete or a move past it. Structural replacement and removal now unlink one dirent at a time and reap the inode only once its last link is gone, so a sibling hardlink survives a type change at another name. That refcount-gated unlink lives in one helper shared by rm, rename, and the apply path. The directory self-move guard is inode-based: it tests the resolved destination parent against the source subtree, so a destination that traverses a symlink out of the source is allowed while one that lands back inside is rejected. A textual prefix test on the unresolved destination could do neither and is gone.
Large directory renames can produce thousands of entries at one revision. A scalar fetch watermark can only resume at rev boundaries, so a crash in the middle of one of those streams forces the next pull to replay the whole rev. Store fetch progress as a rev/path cursor and checkpoint committed batches inside a rev. fetchChanges now advertises a current cursor and streams only entries at or before that cursor, which keeps retry behavior deterministic while materialized entries read current data. This changes the RPC fetch shape from scalar revs to cursors. The durable object and wsd are deployed as a matched pair, so the protocol is updated in lockstep rather than negotiated across mixed versions. coalesceChanges emits one entry per name of a touched inode rather than the single name pathOf returns, so a hardlinked file reaches the wire under every name and a rename of such a file no longer drops its new path. pullOnce owns the fetchChanges result envelope in a try/finally, disposing it on the cross-side invariant trip and on an apply error as well as on the clean drain, so a failing pull no longer leaks the stream stub for the life of the session. A cursor is a resume point, not a point-in-time snapshot handle. coalesceChanges materializes each path's current state and the store keeps no history, so a path that races past the advertised cursor is deferred to a later pull rather than frozen at the cursor's rev. The docs say so explicitly: a path=null cursor means every change committed through that rev has been offered, and convergence holds because the cursor never advances past the rev that would redeliver a deferred path.
Fetch progress is now a rev/path cursor, but the exported watermark helper still accepted scalar fetchRev writes. That left two public write paths for one logical cursor. Restrict the scalar watermark API to pushRev, move fetch progress callers to readFetchCursor and writeFetchCursor, and normalize equal-rev partial cursors when a peer push proves the full rev was applied.
ndisidore
force-pushed
the
nathan/feat/track-rename
branch
from
June 11, 2026 15:55
99d421f to
dea9c3c
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Collaborator
|
This is amazing, thank you 馃槏 Tested it locally, everything feels solid. A great improvement. I've merged the code in 4e51a6c after resolving the conflicts. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change reworks how the durable object filesystem represents moves and how the container daemon resumes a fetch. It spans three commits: renames now flow through a real filesystem primitive and into the sync stream, fetch progress becomes a revision-and-path cursor instead of a single revision, and the now-redundant scalar write path for fetch progress is removed.
Overview. Renames move out of the provider and into a
renamefilesystem primitive alongside the existingrm,link,mkdir, andsymlinkprimitives. A rename stamps the moved inode subtree with one revision and records tombstones for the old paths, so a move is represented on the wire as old-path deletes plus live entries for the new paths, with no new message type. On the sync side, fetch progress changes from a single revision watermark to a{ rev, path }cursor, keyed per backend, so a pull can resume in the middle of a large revision. The fetch RPC is updated to advertise and echo cursors rather than scalar revisions.Why. Three problems motivate the change. First, rename lived inline in the provider and produced no entry in the change stream, so a peer never learned that a file or directory had moved; the inline implementation also carried an in-code note that it should become a real primitive with its own tests. Second, a large directory rename can produce thousands of entries at a single revision, and a scalar fetch watermark can only resume at revision boundaries, so a crash partway through streaming one of those revisions forced the next pull to replay the entire revision. Third, once cursors existed, the exported watermark helper still accepted scalar fetch-revision writes, leaving two public write paths for one logical cursor and an easy way to persist the revision and path components inconsistently.
What this unlocks is that moves, including directory moves, now converge across peers through the same change stream as every other mutation, retries become deterministic and bounded because a pull resumes at the last committed
{ rev, path }rather than the start of the revision, per-backend keying lets a single workspace host more than one backend with independent cursors, and fetch progress has a single source of truth.Alternatives considered. A dedicated move opcode on the wire was rejected because it would require both sides to understand and special-case a new message type in the apply path, whereas tombstones plus live entries reuse the change representation that already exists. Keeping rename inline in the provider was rejected because it is the only operation that mutates directory entries that was not already a primitive recording its own tombstones, and inlining would let local rename behavior and sync output drift apart. Keeping the scalar fetch watermark and carrying a per-entry cursor on the wire was rejected as heavier than having the server advertise a snapshot cursor and stream only the entries at or before it.
Tradeoffs. This is a breaking wire change. The fetch RPC moves from scalar revisions to cursors:
fetchChangesadvertises a snapshot cursor and echoes anappliedPushCursor, andwatermarksreturns afetchCursor. The durable object and the container daemon are deployed as a matched pair, so the protocol is updated in lockstep and there is no compatibility shim for older clients. Directory renames are linear in the size of the moved subtree for both database writes and wire entries, which is an accepted cost documented in the protocol notes. Fetch progress gains a small companion table for the in-revision path while the revision component stays in the existing watermark table for schema compatibility; the table and its per-backend column need no migration because they are introduced fresh and the baseline schema runs on every open. Rename deliberately leaves the old and new parent directory modification times untouched, a small divergence from POSIX rename, so the stream does not treat the parents as content changes.Testing. The
dofs,rpc, andworkspacesuites pass locally withnpm testin each package. New tests pin the rename overwrite matrix across files, directories, and symlinks, the hardlink-onto-hardlink case that removes only the source name, the no-op rename through a symlinked path, tombstones recorded at the resolved real path through intermediate symlinks, fetch resume from a{ rev, path }cursor, and independent cursors across backends. The two real-FUSE tests in the container daemon package require the system FUSE library and do not run in environments without it.Manual verification