[Bug] FSCache on demand loading not working #1769
Description
Problem Description
When running nydus-snapshotter with --fs-driver fscache, fscache appears to initialize and cache data, but container file reads never hit the cache.
This internally shows
INFO[2025-09-18T12:04:31.708240749Z] Start nydus-snapshotter. Version: v0.15.3, PID: 7849, FsDriver: fscache, DaemonMode: multiple
INFO[2025-09-18T12:04:31.710484604Z] Run daemons monitor...
INFO[2025-09-18T12:04:31.710717679Z] initializing shared nydus daemon for fscache
INFO[2025-09-18T12:04:31.713075176Z] nydusd command: /usr/local/bin/nydusd singleton --fscache /var/lib/containerd/io.containerd.snapshotter.v1.nydus/cache --apisock /var/lib/containerd/io.containerd.snapshotter.v1.nydus/socket/d35vajrt64uh53eit6q0/api.sock --log-level info --log-rotation-size 100
[2025-09-18 12:04:31.714596 +00:00] INFO Program Version: v2.3.6, Git Commit: "cb50bd934426d3532534432426c4a701ecaec4a1", Build Time: "2025-09-09T10:40:34.901064076Z", Profile: "release", Rustc Version: "rustc 1.84.0 (9fc6b4312 2025-01-07)"
Observations
- Fscache infrastructure appears functional: Cache writes (167k operations) prove fscache subsystem is working
- EROFS mounts successfully: Kernel logs show erofs mounting with fscache detection
- Zero cache reads: Despite cache writes, no read operations hit the cache
- Likely Result: All container file reads go to backend storage instead of cache (unsure)
Expected Behavior
Container file reads should hit fscache, showing cache read statistics
This should result in performance better than Overlay/FUSE mode
Actual Behavior
All container file reads bypass cache, going directly to backend storage.
Performance worse than Nydus+FUSE and equal to Overlay.
cat /proc/fs/fscache/stats
Reads : DR=0 RA=0 RF=0 RS=0 WB=0 WBZ=0
Writes : BW=0 WT=0 DW=0 WP=0 2C=0
ZeroOps: ZR=0 sh=0 sk=0
DownOps: DL=0 ds=0 df=0 di=0
CaRdOps: RD=0 rs=0 rf=0
UpldOps: UL=0 us=0 uf=0
CaWrOps: WR=0 ws=0 wf=0
Retries: rq=0 rs=0 wq=0 ws=0
Objs : rr=0 sr=0 foq=0 wsc=0
WbLock : skip=0 wait=0
-- FS-Cache statistics --
Cookies: n=0 v=0 vcol=0 voom=0
Acquire: n=100 ok=100 oom=0
LRU : n=0 exp=0 rmv=0 drp=100 at=0
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=100 rtr=0 drop=100
NoSpace: nwr=0 ncr=0 cull=73
IO : rd=46058 wr=169765 mis=0
Kernel Logs Show Fscache Detection
kernel: netfs: Cache "CacheFiles" added (type cachefiles)
kernel: CacheFiles: File cache on CacheFiles registered
kernel: erofs: [deprecated] fscache-based on-demand read feature in use.
How to reproduce
- Run
sudo containerd-nydus-grpc --nydusd-config /etc/nydus/nydusd-config.fscache.json --fs-driver fscache --log-to-stdoutwith the config I provided - Run
sudo nerdctl run --rm -it --snapshotter=nydus docker.io/hsiangkao/ubuntu:20.04-rafs-v6 /bin/bash - Check /proc/fs/fscache/stats
Environment Details
- Nydus-snapshotter version: v0.15.3
- Nydus version: v2.3.6
- Container runtime:
- Operating System: Ubuntu 24.04.3 LTS
- Kernel version: 6.14.11
- Mount shows: erofs ... (ro,relatime,user_xattr,acl,cache_strategy=readaround,fsid=...)
Nydusd config:
{
"type": "bootstrap",
"config": {
"backend_type": "registry",
"backend_config": {},
"cache_type": "fscache",
"prefetch_config": {
"enable": false,
"threads_count": 16,
"merging_size": 1048576
}
}
}
Additional Information
- Fscache provides no performance benefit (0% cache hit rate)
- All container file reads go to backend storage
- On-demand loading not working as expected
Are you willing to submit PR?
- Yes I am willing to submit a PR!