Podman Kube Play with --userns=auto - How to chown volumes?

[giuseppe]

Discussed in #22848

^{Originally posted by MadSpindel May 30, 2024}
I'm trying to understand how I can use rootfull containers with --userns=auto. I have a Kubernetes YAML spec generated with podman kube generate. The problem is, with the hostUsers set to false, the HUSER could change UID and then it will not be possible for the pod to change files in the volume mount. How can I update the Kubernetes YAML spec to tell Podman to chown (change owner) of the volume to new UID created with hostUsers: false?

My current Kubernetes YAML spec:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-05-15T07:29:14Z"
  labels:
    app: dns-server-pod
  name: dns-server-pod
spec:
  hostUsers: false
  containers:
  - args:
    - /etc/dns
    env:
    - name: DNS_SERVER_DOMAIN
      value: dns-server
    image: docker.io/technitium/dns-server:latest
    name: dns-server
    ports:
    - containerPort: 53
      hostPort: 53
    - containerPort: 5380
      hostPort: 5380
    - containerPort: 53
      hostPort: 53
      protocol: UDP
    volumeMounts:
    - mountPath: /etc/dns
      name: technitium_config-pvc
  volumes:
  - name: technitium_config-pvc
    persistentVolumeClaim:
      claimName: technitium_config

```</div>

[giuseppe]

to follow what Kubernetes does upstream, then we need to enforce idmap on each volume when hostUsers: false is used

[Luap99]

But idmap only works as root with the right filesystem's? So what should podman do if idmap is not available?

[giuseppe]

yes idmap works only with root.

Not sure how this should be handled. Is it better to not follow what Kubernetes does in this case?

Maybe we could expect idmap to be an explicit option for the volume

[Luap99]

Not sure but it would be confusing to have different behaviour depending on system and root vs rootless.

Setting mount options explicitly should already work AFAIK although not sure how the syntax looks today. Also for my understanding is hostUsers: false mapped to --userns=auto?

[MadSpindel]

From my perspective, as a simple user, I think it probably should work like Kubernetes if the right conditions are met. The criterias should be defined in the documentation. Then it's clear for everyone. I don't think it need to support all the different edge cases.

[darrellenns]

I have also run into this issue. According to this Kubernetes documentation, they are solving it by using ID-mapped mounts rather than chown. Perhaps podman could do something similar?

[rhatdan]

Podman has support for this in rootful containers, the issue is rootless containers are not allowed access by the linux kernel. We would do it, if we were allowed.

[MadSpindel]

Podman has support for this in rootful containers, the issue is rootless containers are not allowed access by the linux kernel. We would do it, if we were allowed.

Now it was some time ago I tried this, but I'm quite sure I was using this with rootful containers, as described in the first post of this issue. If I recall correctly, I was running podman-kube@.service with the root user. I think it worked the first time but after reboot, the UID could change and then it didn't have access to the volumes anymore.

[darrellenns]

I'm running into this with rootful containers as well.

I was testing yesterday with containers lauched as root with systemd and podman-kube@.service and hostUsers:false. The first time a pod is started, it assigns the lowest available available UID block and used that for both the container and the volume. On subsequent runs, if a different UID block was the lowest available then the containers would run as the different UID but the volume would not be chown'ed or otherwise id mapped. This resulted in broken file permissions in the container.

Perhaps there is some storage config or yaml option needed to enable this id mapping? Maybe it's only supported on certain storage backends (not the default local storage)? I haven't been able to find any documentation on this yet - but maybe I'm just missing it? I'm testing on Debian trixie with the distribution-provided packages (podman 5.3.1). I've made no changes to the default storage config, and I'm using a BTRFS filesystem.

I'm not 100% sure at this point if this is expected behavior (not implemented feature), a bug, or just me missing something.

@rhatdan Is there some documentation or config/yaml settings example you can direct me to for idmap volume mounts with rootful containers?

[MadSpindel]

@darrellenns Which podman version did you try it with? I tried now again with podman version 4.9.3 on an Ubuntu 24.04.2 LTS virtual machine. It looks like it might work without the issues I experienced before.

Edit: Nvm, Podman 5.3.1 on Debian Trixie.

Did you also do this:

The --userns=auto flag requires that the user name containers be specified in the /etc/subuid and /etc/subgid files, with an unused range of subordinate user IDs that Podman containers are allowed to allocate. See subuid(5).

Example: containers:2147483647:2147483648.

[MadSpindel]

Maybe it worked for me because I was only testing with one pod. Now when I added some more it stopped working.

[darrellenns]

@MadSpindel It grabs the lowest possible chunk of the available UID range. If you only ever have 1 pod, then you won't run into the issue. If you have multiple pods and use persistent volumes, then depending on the order they are started the volume ownership gets broken.

[Daskan]

Something i currently have problems with too when running multiple pods.

On my host the volume files may be root:root but in the container i'd like to have them as root:somethingelse without setting UserNS=auto:gidmapping=25:somethingelse.
My current (not beautiful) fix for that is to set the files to
'root:(grep '^containers:' /etc/subgid | cut -d: -f2 + whatEverINeed)' and
UserNS=auto:gidmapping=25:(grep '^containers:' /etc/subgid | cut -d: -f2 + whatEverINeed)
It works but is neither intuitive nor beautiful.