[Improvement] EROFS: Migrate from deprecated fscache on-demand read to fanotify pre-content hooks

[gabrielstedman]

Problem Summary

The Linux kernel is deprecating the fscache-based on-demand read feature in EROFS. This feature is marked as deprecated since Linux 6.12 and is scheduled for removal after fanotify pre-content hooks land in the kernel.

Currently, Nydus uses fscache (CONFIG_EROFS_FS_ONDEMAND) to handle on-demand loading of container image data. This requires:

• Kernel fscache subsystem (CONFIG_FSCACHE)
• Cachefiles driver (CONFIG_CACHEFILES_ONDEMAND)
• Special device interface (/dev/cachefiles)

As the kernel moves away from this approach in favor of fanotify-based pre-content hooks, Nydus must migrate to remain compatible with future Linux versions.

Why This Matters

• Deprecation Timeline:

  • Linux 6.12 (October 2024): CONFIG_EROFS_FS_ONDEMAND marked deprecated
  • Current status: Still functional up to Linux 6.18
  • Future removal: Scheduled after fanotify pre-content hooks merge

• Kernel References:

  • Deprecation commit: 0d442ce0b302 - erofs: mark experimental fscache backend deprecated
  • Removal branch: https://kernel.googlesource.com/pub/scm/linux/kernel/git/xiang/linux/+/refs/heads/erofs/remove_fscache

Current State

Nydus currently implements on-demand read via fscache in:

• service/src/fs_cache.rs - FsCacheHandler
• storage/src/cache/fscache/mod.rs - FsCacheMgr
• Requires kernel config: CONFIG_FSCACHE, CONFIG_CACHEFILES, CONFIG_CACHEFILES_ONDEMAND, CONFIG_EROFS_FS_ONDEMAND

What Needs to Happen

Nydus needs to implement fanotify pre-content hook support as an alternative or replacement to the fscache backend, allowing it to:

• Work with future Linux kernels after fscache on demand is removed
• Reduce kernel dependency footprint (fanotify is lighter weight)
• Support both old (fscache) and new (fanotify) systems during transition period

Related Issues

• [Bug] FSCache on demand loading not working #1769 - FSCache on demand loading not working

Resources

• Kernel deprecation: [Bug] FSCache on demand loading not working #1769 (comment)
• EROFS removal branch: https://kernel.googlesource.com/pub/scm/linux/kernel/git/xiang/linux/+/refs/heads/erofs/remove_fscache

Proposed Improvement

Implement a new fanotify-based cache backend driver alongside the existing fscache driver:

1. Create service/src/fs_fanotify.rs - New FanotifyHandler that:

   • Listens for fanotify permission events (FAN_PRE_ACCESS, FAN_OPEN_PERM, FAN_ACCESS_PERM)
   • Fetches chunks from backend when needed
   • Writes data to file via pwrite() to populate page cache
   • Responds with FAN_ALLOW to unblock access

2. Update service/src/fs_service.rs - Route cache backend based on config:

   • cache_type = "fscache" → FsCacheHandler (existing)
   • cache_type = "fanotify" → FanotifyHandler (new)

3. Configuration - Simplify kernel requirements:

   • Remove: CONFIG_FSCACHE, CONFIG_CACHEFILES, CONFIG_CACHEFILES_ONDEMAND, CONFIG_EROFS_FS_ONDEMAND
   • Keep: CONFIG_EROFS_FS, CONFIG_FANOTIFY

This allows Nydus to support both backends during the transition, with fanotify as the new standard for future Linux versions.

Additional Context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[bergwolf]

/cc @hsiangkao what's your opinion?

[hsiangkao]

/cc @hsiangkao what's your opinion?

I don’t have any resource to implement it.

[bergwolf]

@hsiangkao Is there any architecture design doc for erofs + fanotify + userspace on demand in the kernel community?

[hsiangkao]

@hsiangkao Is there any architecture design doc for erofs + fanotify + userspace on demand in the kernel community?

No, that is why fscache is still here.
btw, I'm not sure if erofs + fanotify will be implemented as a part of containerd + erofs snapshotter if anyone has extra development resource, but currently no on my side.

[hsiangkao]

@bergwolf @imeoer
Anyway, I really think the highest priority for Nydus to leverge erofs in a good state is to make virtual block device approach first for both runC and Kata (either using nbd, tcmu, ublk or vhost-user-blk) as I said years ago, but it seems no one really cares.
Fanotify design currently is under discussion, as long as Nydus developpers are actively developing RAFS v6 Nydus, we could have a brainstorming community meeting to move forward all potential direction including erofs + fanotify, otherwise if it's still like the previous years, I'm not sure if I could help on any case.

[hsiangkao]

@gabrielstedman I just talked with @bergwolf offline, my opinion is that

1. I reiterate that erofs + pre-content hooks will supersede the deprecated erofs+fscache approach in the product roadmaps;
2. Personally I don't have time to work on (even give a detailed design) of erofs + fanotify pre-content hooks, but I could give you a rough design as below:

• Prepare a directory to contain rafs v6 blobs, and create sparse files if any blob needs to pull on-demand;
• In the nydusd, use fanotify hooks to hook these blobs so that every I/O operation of these blobs will trap into nydusd;
• Then nydusd trigger mount to mount erofs filesystem using the following mount format:
  mount(source=bootstrap, options=[device=blob0, device=blob1, ...., device=blobn-1]
  I think that is all nydusd fanotify userspace work, if you have any interest, you could give a try on the detailed design and code implementation.

[bergwolf]

@gabrielstedman Given that fscache is deprecated in kernel, I would propose to deprecate it in nydus as well. Meanwhile, according to @hsiangkao, erofs + fanotify + nydusd should just work. Would you like to help implement it as a new way to integrate erofs with nydus?

[gabrielstedman]

@gabrielstedman Given that fscache is deprecated in kernel, I would propose to deprecate it in nydus as well. Meanwhile, according to @hsiangkao, erofs + fanotify + nydusd should just work. Would you like to help implement it as a new way to integrate erofs with nydus?

Hi @bergwolf Unfortunately I don't have the capacity nor the required rust knowledge to implement something like this.
I opened the issue to help the project keep track of this need.
I would suggest, that if you're happy with the wording, keep this open as is until such time that someone can pick it up.