incus: tmpfs-overlay feature not working #477619
Description
Nixpkgs version
- Stable (25.11)
Describe the bug
Incus 6.16 allows tmpfs and tmpfs-overlay disks for containers. But tmpfs-overlay disk type does not work on NixOS. This problem is NixOS only because of its violation of FHS. There is the full discussion: Incus on NixOS: Failed to start tmpfs-overlay device. Let me have a brief summary here:
When incus launches container with overlay fs, it needs to know the location of liblxc temporary rootfs mount point. In order to generate lxc.conf like this:
lxc.mount.entry = none proc tmpfs size=50000000,defaults 0 0
lxc.mount.entry = none proc/upper invalid defaults,create=dir,optional 0 0
lxc.mount.entry = none proc/work invalid defaults,create=dir,optional 0 0
lxc.mount.entry = none sys overlay userxattr,lowerdir=/opt/incus/lib/lxc/rootfs/etc,upperdir=/opt/incus/lib/lxc/rootfs/proc/upper,workdir=/opt/incus/lib/lxc/rootfs/proc/work 0 0
lxc.mount.entry = /opt/incus/lib/lxc/rootfs/proc etc none move 0 0
lxc.mount.entry = /opt/incus/lib/lxc/rootfs/sys etc none move 0 0
(In this case the temporary rootfs mount point is /opt/incus/lib/lxc/rootfs, and I mount a tmpfs-overlay on /etc. This is a working example on Incus-OS, in which liblxc is located in /opt/incus/lib)
Currently incus hard coded three locations. They correspond to where liblxc.so usually lies in normal systems. It find a existing one and use that (not ideal, but reasonable):
https://github.com/lxc/incus/blob/aafe09569764457b716e5c65b2d38212efd49ec5/internal/server/device/disk.go#L988
On my NixOS installation, temporary mount point is /nix/store/m25jfcwkcp16hwlq5dx5mgk57hq8j1g9-lxc-6.0.5/lib/lxc/rootfs. Which breaks incus tmpfs-overlay device setup.
Proposed solutions
- lxc uses a relative path 'lxc/rootfs' (relative to liblxc itself) as default temporary rootfs mount point. But it offers a compile option to change that https://github.com/lxc/lxc/blob/f51937348d9a1495e768fc409246e935cad8d435/meson_options.txt#L99. We could compile Nix version of lxc with temporary rootfs mount point set to
/opt/incus/lib/lxc/rootfs(imitate Incus-OS). - Let incus allow different liblxc locations. I personally prefer compile option over config option, because it changes whenever package upgrade. Or even automatically detect liblxc location (Is that possible, for system like NixOS?). This will involve more work on incus.
Steps to reproduce
Enable incus(6.16 or later) on NixOS, and then
$ incus create images:debian/13 c1
Creating c1
$ incus config device add c1 tmpfs-overlay disk path=/etc size=50MiB source=tmpfs-overlay:
Device tmpfs-overlay added to c1
$ incus start c1
Error: Failed to start device "tmpfs-overlay": Cannot find rootfs path for container
Try `incus info --show-log c1` for more info
Expected behaviour
container with tmpfs-overlay device should start and tmpfs-overlay device should be mounted.
Screenshots
No response
Relevant log output
Additional context
Leave the original discussion for reference: Incus on NixOS: Failed to start tmpfs-overlay device.
System metadata
- system:
"x86_64-linux" - host os:
Linux 6.12.63, NixOS, 25.11 (Xantusia), 25.11.20251229.89dbf01 - multi-user?:
yes - sandbox:
yes - version:
nix-env (Nix) 2.31.2 - nixpkgs:
/nix/store/rv4z0vd7v0pkadinxd1c1xmd4b850ldr-source
Notify maintainers
@aanderse @adamcstephens @megheaiulian @mkg20001
Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)
I assert that this issue is relevant for Nixpkgs
- I assert that this is a bug and not a support request.
- I assert that this is not a duplicate of an existing issue.
- I assert that I have read the NixOS Code of Conduct and agree to abide by it.
Is this issue important to you?
Add a 👍 reaction to issues you find important.