Commit 4727b44
Performance, reliability, and observability improvements for NativeLink
Major changes:
- MokaEvictingMap: lock-free TinyLFU cache replacing Mutex+LRU EvictingMap,
migrated FilesystemStore, scheduler, ExistenceCacheStore, and MemoryStore
- Streaming blob pipeline: StreamingBlob primitive for concurrent
read-while-write with sliding window, InFlightBlobMap, Phase 1 complete
- io_uring integration: fork of tokio-epoll-uring with metadata ops
(linkat, renameat, mkdirat, unlinkat, symlinkat), hybrid I/O mode
- QUIC transport: BBR congestion, jumbo MTU discovery, connection pooling,
TLS support, dual TCP+QUIC transport with automatic fallback
- Server graceful shutdown: HTTP/2 GOAWAY drain, QUIC shutdown, flush
writes, scheduler drain with configurable timeouts
- Worker improvements: blob mirroring, worker proxy store, peer sharing,
locality-aware scheduling, prefetch, directory cache optimizations
- Store optimizations: scatter-gather MemoryStore, zero-copy ByteStream
read/write codecs, batch operations, write dedup with in-flight
coalescing, parallel BFS tree resolution
- Observability: pprof CPU profiling, stall detector with stack dumps,
comprehensive write path and eviction logging, action lifecycle logging
- GetTree improvements: subtree caching, coalescing concurrent requests,
batch ExistenceCache insert, Arc zero-copy cache entries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 0807153 commit 4727b44
99 files changed
Lines changed: 23870 additions & 3818 deletions
File tree
- .cargo
- benches
- docs
- nativelink-config/src
- nativelink-proto
- com/github/trace_machina/nativelink/remote_execution
- genproto
- nativelink-scheduler
- src
- tests
- utils
- nativelink-service
- src
- tests
- nativelink-store
- src
- tests
- nativelink-util
- benches
- examples
- src
- tests
- nativelink-worker
- src
- tests
- utils
- packaging/macos
- src/bin
- tests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
53 | 53 | | |
54 | 54 | | |
55 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
56 | 63 | | |
57 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
0 commit comments