Enable CONFIG_CACHEFILES_ONDEMAND and CONFIG_EROFS_FS_ONDEMAND in AL2023 kernel for nydus/EROFS fscache support #2569
Description
What would you like to be added?
Enable the following kernel configuration options in the AL2023 EKS-optimized AMI kernel:
CONFIG_CACHEFILES_ONDEMAND=y
CONFIG_EROFS_FS_ONDEMAND=y
Why is this needed?
The nydus-snapshotter project (a containerd non-core subproject) supports an in-kernel fscache mode that provides significantly better performance than FUSE for lazy-loading container images. This feature requires Linux 5.19+ with the above kernel options enabled.
AL2023 ships with kernel 6.1/6.12, which is new enough to support this feature, but the required config options are not enabled:
$ cat /boot/config-6.1.158-180.294.amzn2023.x86_64 | grep -E "(CACHEFILES_ONDEMAND|EROFS_FS_ONDEMAND)"
# CONFIG_CACHEFILES_ONDEMAND is not set
(CONFIG_EROFS_FS_ONDEMAND is not present at all, indicating it's disabled)
The base EROFS and CACHEFILES modules are already enabled:
CONFIG_FSCACHE=m
CONFIG_CACHEFILES=m
CONFIG_EROFS_FS=m
Container image lazy-loading/streaming is increasingly important for reducing cold-start times, especially for:
- Large ML/AI workloads with multi-GB images
- Serverless/FaaS platforms requiring fast pod startup
- High-density clusters with frequent scaling events
Nydus with fscache mode provides the best performance for this use case by handling image data fetching in-kernel rather than through FUSE userspace context switches. Alibaba/Ant Group runs this at massive scale in production. Without these kernel options, users are limited to FUSE mode, which works but has measurably higher overhead due to kernel/userspace context switching on every file access.
This would complement existing container acceleration efforts in the EKS ecosystem (SOCI snapshotter, etc.) and provide users with more options for optimizing container startup performance.
Happy to provide any additional information or testing if helpful.
References
- EROFS paper (USENIX ATC '19): Gao et al., "EROFS: A Compression-friendly Readonly File System for Resource-scarce Devices" - https://www.usenix.org/conference/atc19/presentation/gao
- Nydus fscache documentation: https://github.com/dragonflyoss/nydus/blob/master/docs/nydus-fscache.md
- EROFS over fscache merged in Linux 5.19: https://kernelnewbies.org/Linux_5.19#EROFS
- CNCF blog on Nydus evolution: https://www.cncf.io/blog/2022/11/15/the-evolution-of-the-nydus-image-acceleration/