Any plans for FUSE passthrough (kernel 6.9+) to cut per-op overhead on cached reads?

[charlesbmi]

We're evaluating JuiceFS to serve a large build cache — a big tree of build outputs (100+ GB, hundreds of thousands of mostly-small files) kept in S3. We mount it read-only with OverlayFS on top so builds write locally and never modify the cache, pull content lazily, and use juicefs warmup to pre-load the working set into the local cache before/while the build runs.

One thing stood out in benchmarking: even once the working set is fully cached locally, reads are noticeably slower than reading the same files from a plain local ext4 copy of the tree — about 25% slower end-to-end for the identical workload. Since the data is already on local disk at that point, the gap looks like the FUSE userspace round-trip on every open/read rather than actual I/O or network.

Kernel 6.9 added FUSE passthrough, which lets read/write/mmap go straight to a backing file and skip the userspace hop, and go-fuse (which JuiceFS uses) already has the primitives — FOPEN_PASSTHROUGH and backing-fd registration. Is using passthrough for fully-cached files something that's been considered, or on the roadmap? I'm guessing the chunked block cache makes it tricky, since a cached file isn't a single contiguous backing file on disk — is that the main blocker, or is there more to it?

More broadly, if there's a recommended way to reduce per-op overhead for read-heavy, mostly-cached workloads (short of the Python SDK — that doesn't fit here since the reader is an ordinary build tool using normal file syscalls), I'd be glad to hear it. Thanks!

[jiefenghuang]

1. The performance issue may be related to FUSE overhead. What is the average file size in your workload? Is the read pattern sequential? If so, increasing --max-fuse-io could improve performance. It would still be best to use tools such as stats, profile, and accesslog to identify the actual read pattern and bottleneck.
2. Another community user has raised a similar issue and submitted a PR for FOPEN_PASSTHROUGH. We will consider it for version 1.5.

[charlesbmi]

1. The files are mostly small. Let me benchmark and check, thanks for the recommendation!

2. Neat! Cross-linking the PR for my own visibility: fuse: experimental write-path FUSE passthrough (WIP) #7202

[charlesbmi]

Thanks, this was really helpful — profile/accesslog pointed straight at the bottleneck.

Context on the workload: it's a build-output tree, ~740k files, median size ~2.6 KB (very small-file heavy), though ~95% of the bytes are in the 7.6% of files over 128 KB. Reads during a build are a mix — lots of small random opens plus some large sequential reads during image packaging.

I tried --max-fuse-io=4M on a fully-warm build (everything already in the local cache): no change, 171.6s vs 170.7s. The accesslog op-mix showed why — content reads are only ~1% of FUSE op-time:

op        count      total_s   avg_us   %op-time
lookup    430,811    1052.7    2443     98.3
read       35,652      11.6     326      1.1
open       40,268       4.2     104      0.4

So it's almost entirely lookup (path resolution) — ~430k at ~2.4ms avg. Since our mount is read-only against immutable content, I raised the metadata caches (--entry-cache/--dir-entry-cache/--attr-cache/--negative-entry-cache=3600s, plus --readdir-cache). That cut the warm build 170.7s → 146.2s and lookups 430k → 123k (repeats become kernel dcache hits). For reference the same build off a plain local ext4 copy is ~136s, so we went from ~25% over native to ~7%.

Two questions if you have a moment:

1. The remaining ~123k lookups still average ~2.3ms — is that expected for the SQLite engine, or is there a knob (or a way to pre-warm the metadata/dentry cache) to bring first-touch lookups down further?
2. This is really the same "cached access still pays the FUSE round-trip" theme as the passthrough thread — but since lookup dominates rather than read/write, would passthrough even apply here, or is the metadata-cache path the only lever?

Thanks again!

[jiefenghuang]

Is 123K the result after multiple runs, or from the first run?

To warm up the metadata cache, you can simply traverse the entire directory. However, this depends on the amount of available memory, and you may need to adjust Linux’s cache reclaim settings accordingly.

[charlesbmi]

👍🏼 I was mostly interested in a first-run, as we'll run this as a cache for cold CI build machines. That makes sense to warm up the metadata cache for multi-run settings. Maybe that will come in handy a different way. Thanks for the help!