dofs: perf/bench and quickwins - #14
Merged
Merged
Conversation
Measure the filesystem operations against a real Durable Object SqlStorage via vitest-pool-workers, not the node SQLiteTestStorage fixture (which caches prepared statements and would understate per-statement cost). Each scenario reports wall-clock ns/op and a deterministic statement and row count from a CountingStorage decorator over the real backend. Scenarios cover both stat surfaces across a path-depth sweep, present and missing exists, small and positional reads, readdir versus ls, recursive delete, a write-heavy create/edit/delete burst, single and subtree renames, and a dedup guard on database size. A signature gate asserts the deterministic counts so drift fails the run. Runs only through vitest.config.bench.ts, so it stays out of the regular suite; src/bench is excluded from the build as it imports the workers-only test module. Invoke with npm run bench.
statSync resolved the path twice — once through statImpl for mode/size/mtime and again for the inode behind the link count. stat now returns the resolved inode, so the provider reuses it for both. nlink for hardlinked files and the inode 0 / nlink 1 of a pending-create file are unchanged. Recursive delete threads the parent inode and name the post-order walk already knows into unlinkDirent instead of re-resolving each parent from the root; the single-entry path reuses the parent it just resolved. Deleted dirents, reap-on-last-link, and tombstones are unchanged. writeSync stats only when appending, the only case that needs the offset. A non-append write is validated by writeRangeSyncImpl (ENOENT/EISDIR); a zero-length write keeps an explicit existence check since it short-circuits before that resolve.
Cluster the two composite-key tables on their primary keys so a lookup reads from the b-tree leaf with no rowid indirection: child_inode lives in the vfs_dirents PK leaf, so a path segment resolves with a single covering read, and a file's chunks sit in (inode, idx) order. Both are eligible — composite PK, no AUTOINCREMENT, not part of any foreign key. vfs_blob_bytes stays a rowid table; it holds the large blob payloads and the schema's only foreign key. Fresh databases get the shape from the baseline DDL; a v4 to v5 migration rebuilds existing ones, copying each table into a WITHOUT ROWID replacement and recreating its secondary index (the rename and drop destroy it, and the baseline CREATE INDEX does not re-run). The rebuild runs inside initializeSchema's transaction. A migration test stages a v4 database with nested dirs, a hardlink, multi-chunk files and dedup, and asserts the version bump, both tables WITHOUT ROWID, both indexes present, byte-for-byte data survival, and that a fresh install matches a migrated one. The direct-range-write test watches vfs_blobs.last_seen instead of a chunk rowid, which no longer exists.
Resolve the read path with a single recursive-CTE statement and add a per-Database path-to-inode cache, so repeat stat/exists/read of a path cost one statement, and one node re-read once warm, independent of depth. Both fast paths serve only follow-symlinks resolutions outside a transaction. A cache hit re-reads just the node row, so mode/size/mtime stay current — only the path-to-inode mapping is cached. A miss runs one recursive walk of vfs_dirents joined to vfs_nodes; it descends only through directories, so a missing segment or non-directory intermediate yields null exactly as the loop, and any symlink on the path falls back to the per-component loop (which follows links and enforces ELOOP). Symlink resolutions are not cached. lstat/readlink and every resolve inside a mutation keep the loop: those are shallow and hot, and the cache must not be populated mid-transaction so a rollback leaves no stale entry. The cache is invalidated at every dirent move — create, unlink, delete, rename, and the sync-apply structural cleanup — dropping one entry for a leaf and the whole subtree for a directory or symlink change.
A directory rename stamps the shared rev across the moved inode subtree and records an old-path tombstone per edge with two set-based statements over one recursive-CTE walk: an UPDATE keyed on the subtree inodes and an INSERT ... SELECT of the old paths. The self-move guard reuses the same walk. The tombstone rows and rev stamps form the set the sync log expects, coalesced by rev and path independently of insertion order. File and symlink renames take the single-row path. buildManifest encodes the chunk list once and hashes those bytes, so the manifest hash and the stored encoding stay in step by construction.
Reads resolve chunk bytes by hash and leave vfs_blobs.last_seen untouched. A chunk being read is already linked to a node, so gc's orphan gate retains its blob independent of the timestamp and a read never needs to stamp it. last_seen remains the timer that guards blobs staged but not yet linked — the sync receiver, the streaming write, and manifests — and those write paths still stamp it. The read clock argument that fed the removed touch is dropped.
Directory listing resolves the prefix to an inode and seeds its recursive walk there, so it visits only the listed subtree rather than the whole store. The prefix walk follows dirents structurally and does not resolve symlinks, so a symlink prefix lists nothing. The output is unchanged: the same absolute file paths in path order. The sync object probe matches the requested hashes against vfs_blobs through an IN (…) list on the raw hash blobs, so the lookup rides the primary-key index instead of scanning; a probe larger than the window splits into a few bounded index lookups. The present subset comes back in input order with duplicates preserved. A positional read fetches its overlapping chunks in one indexed range scan ordered by index instead of a lookup per chunk. Missing indices are absent from the result, so the assembly compacts around them just as the per-chunk walk did, and the blob byte cache is consulted in the same order.
aron-cf
approved these changes
Jul 6, 2026
aron-cf
left a comment
aron-cf
left a comment
Collaborator
There was a problem hiding this comment.
Nice! Thank you so much. This is rad. Great improvements all round and thanks for presenting it in a way that was super easy to review 🙇
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Makes
@cloudflare/dofs(the Durable Object SQLite virtual filesystem) fast on the path-heavy operations that dominate agent workloads —stat,exists,read, directory listing, delete, and rename — while preserving every design goal: content-addressed dedup, chunked buffered writes, hardlinks, small on-disk footprint, and sync convergence. A reproducible micro-benchmark harness (against real DO SqlStorage) lands first, so every change is measured, not asserted.Net effect: operations that previously cost
O(path-depth)SQL round-trips per call are now a single statement cold and ~O(1)warm; whole-subtree operations (delete, directory rename) are set-based; and reads no longer emit writes.Motivation
dofsis used by agents to work on git repos, so it is overwhelmingly filesystem-op heavy: constantstat/exists/readagainst deep paths (node_modules, nestedsrc/…), plus bursts of edits, deletes, and renames. The hot path was structurally expensive:1 + 2·depthSQL statements per lookup, no single-query resolve, no cache. Astat20 levels deep cost 41 statements;provider.statSyncdoubled that to 83 (it resolved twice and added a link-count query).2Nround-trips).last_seentouch), sogrep -rwrote proportional to bytes scanned.ls <dir>scanned the entire filesystem, and the sync object-probe ran one query per hash.On Durable Objects each SQL statement is a billed row op and they run serially, so statement count is the cost that matters — and it was scaling with tree depth and size.
Commit-by-commit
72fc614add filesystem micro-benchmark harness — depth-parameterized bench over real DO SqlStorage; deterministic statement counts + wall-clock, with a signature gate so future changes must consciously update the fingerprint.72a9ffedrop redundant path resolves in statSync, rm, and writeSync —statSyncresolves once and folds in link-count (3+4·depth → 2+2·depth); recursive delete reuses the parent inode the walk already knows; non-appendwriteSyncskips an unused stat.3fa0566store vfs_dirents and vfs_chunks WITHOUT ROWID — clusters those tables on their real PKs, sochild_inodelives in the dirents leaf and a path-segment resolve is a single covering read. Includes the v4→v5 migration (recreating both secondary indexes the rebuild drops) with a lossless-migration test;vfs_blob_bytesintentionally untouched.1e52048resolve paths in one statement and cache path lookups — single recursive-CTE resolve (→ 1statement) plus a per-DO positive/negative path→inode cache; falls back to the component walk on symlinks. Invalidated on every local mutation and on the sync-apply path.49a48c0batch directory-rename subtree writes and encode manifests once — directory rename applies the whole subtree with set-based SQL (2N → ~3statements), producing a byte-identical change-log; manifest encodes once.a3a73f2resolve reads without restamping last_seen — removes the per-chunk write on the read path;last_seenstays as the crash-window timer for staged-but-unlinked blobs.60ab69cscope ls to its subtree and batch hasObjects and range reads —lsseeds its walk at the target inode instead of scanning the store; the sync object-probe matches raw hash blobs through an index-backedIN (…)list (bounded windows) instead of a query per hash; positional reads fetch chunks in one indexed range scan.Real-world benchmarks
How we measure. The harness (
packages/dofs/src/bench/) runs against a real Durable ObjectSqlStorageunder workerd via@cloudflare/vitest-pool-workers— not a node SQLite stand-in, whose statement caching would understate cost. Two signals per operation, across path depths 1–20:Reproduce:
cd packages/dofs && npm run bench.Before → after (real DO SqlStorage; stmts = SQL statements = billed row ops):
stata file 20 dirs deep —fs.stat(editor open,git status)stata file 20 dirs deep —provider.statSyncstatthe same deep path (build hot loop)existsa missing file 16 deep (module resolution)cat, file open)grep -rscanning N chunks (read amplification)rm -rfa 2000-entry treemva directory with 500 descendantsls <dir>Dedup and footprint are preserved: 100 × 1 MiB identical files → 0.64 MiB on disk (content-addressed sharing intact), and
WITHOUT ROWIDmade the DB marginally smaller (667,648 → 659,456 bytes).Risks
WITHOUT ROWIDmigration rebuildsvfs_dirents/vfs_chunksin one atomictransactionSync, recreating both secondary indexes and converging with the fresh-install DDL; covered by a lossless-migration test (data byte-identical, fresh == migrated). Caveat for large existing tenants: this is the first migration whose cost scales with filesystem size, and the consumer runs schema init synchronously on DO construction — so a very large store pays a one-time whole-table copy on first wake post-deploy. Worth measuring worst-case copy time / peak storage before GA; fresh installs getWITHOUT ROWIDdirectly with no migration.last_seen: reads no longer refresh it, so a read-then-deleted blob is reclaimable up to ~1 h sooner — strictly better for footprint; the staging crash-window guarantee is unchanged. (gc()is not yet wired to a scheduler, so this is latent regardless.)All changes are behavior-preserving; no public API or wire-format change.
Testing
tsc --noEmitand biome clean.lssubtree-independence + symlink prefix,hasObjectsorder/duplicates,readRangemulti-chunk/sparse/EIO parity, and the "reads don't restamplast_seen" invariant.exit=1("2 errors") is a known vitest-pool-workers teardown race in two node-only files — reproduced on the unchanged base, not introduced here.Notes & follow-ups