Skip to content

Commit 4727b44

Browse files
rejuvenileclaude
andcommitted
Performance, reliability, and observability improvements for NativeLink
Major changes: - MokaEvictingMap: lock-free TinyLFU cache replacing Mutex+LRU EvictingMap, migrated FilesystemStore, scheduler, ExistenceCacheStore, and MemoryStore - Streaming blob pipeline: StreamingBlob primitive for concurrent read-while-write with sliding window, InFlightBlobMap, Phase 1 complete - io_uring integration: fork of tokio-epoll-uring with metadata ops (linkat, renameat, mkdirat, unlinkat, symlinkat), hybrid I/O mode - QUIC transport: BBR congestion, jumbo MTU discovery, connection pooling, TLS support, dual TCP+QUIC transport with automatic fallback - Server graceful shutdown: HTTP/2 GOAWAY drain, QUIC shutdown, flush writes, scheduler drain with configurable timeouts - Worker improvements: blob mirroring, worker proxy store, peer sharing, locality-aware scheduling, prefetch, directory cache optimizations - Store optimizations: scatter-gather MemoryStore, zero-copy ByteStream read/write codecs, batch operations, write dedup with in-flight coalescing, parallel BFS tree resolution - Observability: pprof CPU profiling, stall detector with stack dumps, comprehensive write path and eviction logging, action lifecycle logging - GetTree improvements: subtree caching, coalescing concurrent requests, batch ExistenceCache insert, Arc zero-copy cache entries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 0807153 commit 4727b44

99 files changed

Lines changed: 23870 additions & 3818 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.cargo/config.toml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[build]
2+
rustflags = ["-C", "target-cpu=native", "-C", "force-frame-pointers=yes", "--cfg", "tokio_unstable"]
3+
4+
[target.x86_64-unknown-linux-gnu]
5+
rustflags = ["-C", "target-cpu=native", "-C", "link-arg=-fuse-ld=mold", "-C", "force-frame-pointers=yes", "--cfg", "tokio_unstable"]
6+
7+
[profile.release]
8+
lto = "thin"
9+
codegen-units = 8

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,4 @@ nativelink.bazelrc
2626
buck-out/
2727
nativelink_config.schema.json
2828
.cargo/config.toml
29+
.cargo/config.toml

.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "tokio-epoll-uring"]
2+
path = tokio-epoll-uring
3+
url = forgejo@optimus.m0n0.space:rejuvenile/tokio-epoll-uring.git

CLAUDE.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,5 +53,44 @@
5353
- `#[cfg(target_os = "...")]` for OS-specific code (Linux vs macOS)
5454

5555
## Tests
56+
- **Test-first development**: when implementing any new feature, write tests first
57+
(unit, integration, and cross-component interaction tests). Verify they fail before
58+
implementing the feature, then make them pass. Include fakes/mocks for
59+
hardware-interaction tests where needed.
60+
- **Bug fixes require a failing test first**: when fixing a bug, write a test that
61+
reproduces the failure, verify it fails, then implement the fix and show the test
62+
passes. Never fix a bug without a regression test.
5663
- Integration tests in `tests/` directory; minimal inline `#[cfg(test)]` modules
5764
- Use `nativelink-macro` test harness (`#[nativelink_test]`)
65+
66+
## Change Process
67+
- **Chesterton's Fence**: before modifying or removing any behavior, always check
68+
`git log`, `git blame`, and `git log -S` to understand *why* the code exists.
69+
If a commit message or comment explains the reason, evaluate whether that reason
70+
still applies before making the change.
71+
72+
## Code Review
73+
- **Before committing any change**, send the changes to a code review agent and a
74+
performance review agent. Work to obtain their sign-off before committing. Fix
75+
any issues they identify. Only commit after both reviews pass with no blocking
76+
issues.
77+
78+
## Git Journal
79+
- **Journal all git operations**: append every `git commit`, `git push`, `git revert`,
80+
`git stash`, and any other state-changing git command to `.claude/git-journal.md`
81+
in the working directory. Each entry should include the timestamp, command, and a
82+
one-line description. This prevents losing track of what was done across context
83+
compressions.
84+
85+
## Working Directory Discipline
86+
- **Always verify `pwd` before git operations.** Agent worktrees (`.claude/worktrees/`)
87+
have separate git branches. Commits in a worktree do NOT go to `main`. The Bash tool
88+
may silently `cd` into a worktree after an agent runs. Always `cd /path/to/nativelink`
89+
before any `git commit`, `git push`, or `git status`.
90+
- **Never use `git stash pop`** — it can cause merge conflicts that `git checkout --` resolves
91+
by reverting uncommitted edits. Use `git stash apply` + `git stash drop` separately.
92+
- **Commit early, commit often.** After each logical change compiles, commit immediately.
93+
Don't accumulate multiple uncommitted changes across a session — context compression
94+
or worktree confusion can lose them.
95+
- **After editing files, verify with `git diff --stat HEAD`** that the expected changes
96+
appear before moving on to the next task.

0 commit comments

Comments
 (0)