Value of $XDG_RUNTIME_DIR persists across reboots

[WhyNotHugo]

Issue Description

Previously, XDG_RUNTIME_DIR=/run/user-1000.

My system setup has changed, and after a reboot, XDG_RUNTIME_DIR points to a different location.

podman seems to have persisted the value of XDG_RUNTIME_DIR somewhere, and now fails, because it's trying to access a path which no longer exists:

> podman ps
WARN[0000] RunRoot is pointing to a path (/run/user-1000/containers) which is not writable. Most likely podman will fail. 
Error: creating events dirs: mkdir /run/user-1000: permission denied

Note that the value of XDG_RUNTIME_DIR has been stable since the system rebooted: the value that podman is using was only ever exposed before the system went offline.

Steps to reproduce the issue

Steps to reproduce the issue

1. Set XDG_RUNTIME_DIR to some value.
2. podman run … (any container)
3. Reconfigure the system so that the next boot uses a different XDG_RUNTIME_DIR (e.g.: by changing the system component that initialises it).
4. Reboot
5. podman ps

Describe the results you received

Podman tries to access a path based on an environment variable's value from a previous boot.

Describe the results you expected

Podman should not cache environment variables or values derived from them; it should read the current XDG_RUNTIME_DIR.

podman info output

Podman 5.8.1-r1, on Alpine Linux.

> podman info
WARN[0000] RunRoot is pointing to a path (/run/user-1000/containers) which is not writable. Most likely podman will fail. 
Error: creating events dirs: mkdir /run/user-1000: permission denied

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

[WhyNotHugo]

In libpod/runtime.go, mergeDBConfig explicitly overwrite the value of r.storageConfig.RunRoot with the value persisted in the database.

In all cases, podman would use a stale value of XDG_RUNTIME_DIR.
This only works if the database value is equal to the current value (in which case the assignment is a no-op).

[Luap99]

That is expected, you cannot safely change paths later because full paths may be stored in the container db as json values, so if we would use the new path we might end up in a very bad state.

You would have to podman system reset to start fresh.

[WhyNotHugo]

That is expected

As an end-user, this is most definitely not expected.

The xdg basedir spec is quite clear about this:

The lifetime of the directory MUST be bound to the user being logged in

podman assumes the contrary: it assumes that the directory persists across reboots, and fails if its path changes.

---

you cannot safely change paths later because full paths may be stored in the container db as json values, so if we would use the new path we might end up in a very bad state.

The XDG_RUNTIME_DIR is, by definition, ephemeral. All of its contents are flushed when the system goes offline. Any path that points to directories inside of it become invalid at that point.

The current implementation only works because most implementations use a stable path across reboots.

You would have to podman system reset to start fresh.

Destroying all my data every reboot is absolutely unacceptable.

[Luap99]

That is expected

As an end-user, this is most definitely not expected.

I am not claiming otherwise, I am saying this is expected behavior of podman as of today adn as such not something I consider a bug. That does not rule out changes in the future.

The XDG_RUNTIME_DIR is, by definition, ephemeral. All of its contents are flushed when the system goes offline. Any path that points to directories inside of it become invalid at that point.

Well the files are gone sure, we handle that fine and recreate paths. The problem is we have many db options who may commit such tmp paths into the db and all these paths would somehow be updated each boot which is not exactly trival to maintain or alternative figure out how to not store them or store relative paths.

These are technical problems to solve before we could remove such restrictions
cc @mheon

Also the other reason why we have such paths checks is that many user run with inconsistent envs, i.e. no one session with no XDG_RUNTIME_DIR while another with. IF we do not catch this we end up with inconsistent states which is just really bad.

Destroying all my data every reboot is absolutely unacceptable.

Why would you change that value on every reboot?

[WhyNotHugo]

Well the files are gone sure, we handle that fine and recreate paths. The problem is we have many db options who may commit such tmp paths into the db and all these paths would somehow be updated each boot which is not exactly trival to maintain or alternative figure out how to not store them or store relative paths.

Can we store the paths relative to $XDG_RUNTIME_DIR?

The paths can be made absolute when reading them and be made relative when saving them, providing a clear barrios where this happens.

Also the other reason why we have such paths checks is that many user run with inconsistent envs, i.e. no one session with no XDG_RUNTIME_DIR while another with.

I think it's safe to exit-error if $XDG_RUNTIME_DIR; the user can deal with fixing their environment in such a case. I think pretty much anything else would fail in such a scenario anyway.

Why would you change that value on every reboot?

That a bit tangential, but instead of using a suid helper to create the directory in a location only writable by root, it's created without suid in a location where all users can write. The location is not stable across reboots in order to prevent races. I wrote an article with the rationale behind this if you're really interested.

A longer-term goal is to mount / with nosuid.

[mheon]

I was planning to include auto-rewrite for paths in the database (if and only if no containers, pods, and volumes are present) in my 6.0 SQLite removal, but I ended up deferring it for reasons I've honestly forgotten.

Changing storage for paths in containers to be relative - to allow moves without removing all your containers - is also possible, but adds complications, the potential for bugs, and introduces a lot of migration problems (what about containers created prior to the patch?). Further, it doesn't really solve the problem that our cached paths in the database seek to prevent. If XDG_RUNTIME_DIR changes but Podman container storage is not moved at the same time, you now have containers that exist in the database but whose backing storage no longer exists. Fixed paths guarantees that we're insulated from environment variable changes, which can and do occur on a regular basis (e.g. /tmp vs /var/run for temporary files based on how a user logged in is a fairly common one we see).

[Luap99]

Fixed paths guarantees that we're insulated from environment variable changes, which can and do occur on a regular basis (e.g. /tmp vs /var/run for temporary files based on how a user logged in is a fairly common one we see).

I think that is fair for as long as the user is logged in, as pointed out here though the spec says XDG_RUNTIME_DIR can change afterwards so we should be able to deal with it somehow.

We already have special state reset logic to detect reboots, and of course to recreate the tmpdirs we use there. So we could freeze the XDG_RUNTIME_DIR on each boot but allow changes between reboots.

But I do agree implementation wise making the code ready for such things is non trival considering we need to keep old containers working.
I do not see this as a high priority thing for us to fix but I would certainly welcome contributions.

[WhyNotHugo]

Changing storage for paths in containers to be relative - to allow moves without removing all your containers - is also possible, but adds complications, the potential for bugs

Note that nothing is moved. The path to which $XDG_RUNTIME_DIR points (both in my setup and yours) ceases to exist upon reboot. It gets re-created and re-populated.

Further, it doesn't really solve the problem that our cached paths in the database seek to prevent. If XDG_RUNTIME_DIR changes but Podman container storage is not moved at the same time, you now have containers that exist in the database but whose backing storage no longer exists.

$XDG_RUNTIME_DIR must not change while the user has running processes; this is expressly prohibited by the basedir spec. Such a scenario would be the result of a wildly broken setup, and all sorts of other things would break (or note: wayland, pipewire, dbus).

I'm not sure that this realistically needs to be supported.

Fixed paths guarantees that we're insulated from environment variable changes, which can and do occur on a regular basis (e.g. /tmp vs /var/run for temporary files based on how a user logged in is a fairly common one we see).

I guess this refers to $TMPDIR, and this protection can remain, since it's orthogonal to caching the value of XDG_RUNTIME_DIR.

---

TLDR: I think the main value of storing paths is protecting against $TMPDIR and others changing, but realistically, protecting against change in XDG_RUNTIME_DIR is unnecessary.

[mheon]

Changing storage for paths in containers to be relative - to allow moves without removing all your containers - is also possible, but adds complications, the potential for bugs

Note that nothing is moved. The path to which $XDG_RUNTIME_DIR points (both in my setup and yours) ceases to exist upon reboot. It gets re-created and re-populated.

Further, it doesn't really solve the problem that our cached paths in the database seek to prevent. If XDG_RUNTIME_DIR changes but Podman container storage is not moved at the same time, you now have containers that exist in the database but whose backing storage no longer exists.

$XDG_RUNTIME_DIR must not change while the user has running processes; this is expressly prohibited by the basedir spec. Such a scenario would be the result of a wildly broken setup, and all sorts of other things would break (or note: wayland, pipewire, dbus).

And yet, it happens. Particularly when dealing with headless systems where you have a user starting sessions directly via SSH, while also having cronjobs or similar starting containers. One of these creates a proper login session and gets a /var/run/$UID created; one does not. If the containers created by these jobs overlap, things can start to break quickly. Caching paths ensures we're at least all using the same source of truth. If we cache /var/run/$UID and it doesn't exist for the timer-initiated containers, things still break, but it's at least immediately obvious to folks what went wrong.

[WhyNotHugo]

podman is trying to accomodate broken environments where XDG_RUNTIME_DIR is not consistent for concurrent sessions. This behaviour violates the specification, and software that sets up environments in this way needs to be fixed.

Obviously I'm biased because this affects me, but scenarios which don't violate the spec take precedence over scenarios which violate the spec.

The big issue now is that folks have come to rely on this band-aid, and removing it is a breaking change.

[WhyNotHugo]

Okay, new pitch:

When reading values from the database, check if they exist and area readable/writeable. If they are override with these values, otherwise ignore them.