Native `overlay` storage has significant impact on rootless containers creation time

[travier]

Native overlay storage has significant impact on container creation time and podman command responsiveness (podman images).

Switching back to fuse-overlayfs mitigates this issue, trading it off for some runtime cost. Note that for most rootless containers such as toolboxes, the performance overhead of fuse-overlayfs is likely negligible as the home directory is bind mounted and thus not going through fuse. Containers that also store their data in volumes are not penalized either.

# Record / backup your containers first as the command below will remove them all
$ podman systemd reset

# Create the storage config file with the content below
$ cat ~/.config/containers/storage.conf 
[storage]
driver = "overlay"
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

# Pull your containers again

See:

• https://gitlab.com/fedora/ostree/sig/-/issues/57

---

Original issue: Dependency on fuse-overlayfs in containers-common moved to Suggests

fuse-overlayfs has been moved from a Recommends to a Suggests in Fedora 40 and later:

• Rawhide: https://src.fedoraproject.org/rpms/containers-common/c/558d2ade830a984a7ec5f7068c4826c597566457
• F40: https://src.fedoraproject.org/rpms/containers-common/c/447945e59a01cb6715ed2a21877d45bf0b91ef67

I manually added it back in Fedora Atomic Desktops for F40: https://pagure.io/workstation-ostree-config/pull-request/526

In Fedora CoreOS we have it explicitly listed in https://github.com/coreos/fedora-coreos-config/blob/testing-devel/manifests/fedora-coreos-base.yaml#L106.

We should reach out to the podman folks to figure out if we should also remove it with the update to F41.

[debarshiray]

The Podman folks want to move off fuse-overlayfs and encourage people to use the Linux kernel's overlay file system. I think the question for Fedora CoreOS is: how important is backwards compatibility for old containers?

Backwards compatibility is important for Toolbx. So, I am considering adding at least a Recommends on fuse-overlayfs to the toolbox RPM, until we can figure out a decent way to migrate users than some cryptic string hidden in some logs.

[debarshiray]

I tried to upstream the changes to unbreak Fedora 40:
containers/common#2203

[travier]

We should look into this again as we're nearing the F41 release/rebase so that would be a good time to remove it. Otherwise, we might have to carry it until F42.

[dustymabe]

@travier @debarshiray - trying to tease out some details for the meeting discussion today:

• does having fuse-overlayfs installed cause it to be preferred?

ideally if it's installed it's just there for backwards compat, but newly created machines (or even possibly new containers on upgrading machines) would use the preferred in kernel overlayfs.

[jlebon]

@travier @debarshiray - trying to tease out some details for the meeting discussion today:

* does having `fuse-overlayfs` installed cause it to be preferred?

At least on my f40 Silverblue, it's there (which I guess is thanks to @travier) but my backend is still overlay:

podman info | grep -i graphDriver
  graphDriverName: overlay

I think this is one of those things that only gets updated if you do e.g. podman system reset to reset storage defaults.

I'd be surprised if a lot of people are still using it, so the impact should be minimal. Though OTOH we also didn't mention it in our f41 communications and it feels a bit late for a potentially breaking change.

[c4rt0]

This was discussed today in #meeting-1:fedoraproject.org,

AGREED: We won't remove the package for F41 but will work with the podman team to determine the appropriate time to do the deprecation. This may be on a Fedora major boundary with an associated Fedora change request.

[eriksjolund]

Maybe not so important but good to know:
fuse-overlayfs is faster than native overlay
at creating a new container when these conditions are all met:

• rootless Podman is used
• a modified UID/GID mapping is used
• no container has yet been created with the specified container image and UID/GID mapping

Reference: https://github.com/containers/podman/blob/b65f3b19a59e4af17c1649461e9366f39f97e51b/docs/tutorials/performance.md?plain=1#L68-L84

[dustymabe]

Nice @eriksjolund. @giuseppe I assume containers/podman#16541 (comment) is still the state?

[giuseppe]

Is there any particular reason to drop it? It is a quite small binary (~112kb on Fedora 40) and there are still cases where it is needed. For example, running on NFS. That is not possible for rootless using native overlay.