Missing --group-add in "podman pod create"?

[Lotendan]

Issue Description

Attempting to create a pod when subject to linux group permissions results in an OCI permission denied error.

Steps to reproduce the issue

Environment

$ ls -alh /mnt
drwxr-x--- 11 root disk 4.0K Nov  2 19:01 disk

$ ls -alh /mnt/disk/
drwxr-x--- 10 lotendan lotendan 4.0K Nov 8 09:39 lotendan

$ cd /mnt/disk/lotendan
$ mkdir data
$ ls -alh data
drwxr-xr-x 2 lotendan lotendan 4.0K Nov 16 16:59 data

We setup the environment as follows:

$ cd /mnt/disk/lotendan
$ export XDG_DATA_HOME=$(pwd)/.local
$ export XDG_CACHE_HOME=$(pwd)/.cache
$ export XDG_CONFIG_HOME=$(pwd)/.config
$ export XDG_RUNTIME_DIR=/run/user/$(id -u)

$ id
uid=1001(lotendan) gid=1001(lotendan) groups=1001(lotendan),1002(disk)

$ cat /etc/subuid
lotendan:100000:65535
$ cat /etc/subgid
lotendan:100000:65535

We are using this setup because the rest of the filesystem needs to be read-only.

Podman run

First we try with podman run to start an individual container.
There is a requirement that the same user that runs the pod is mapped in the container (i.e. `--userns=keep-id is a requirement).

# Reset
$ podman system reset

# Obviously this fails:
$ podman run --rm -it alpine sh
Error: crun: open `/mnt/disk/lotendan/.local/containers/storage/overlay/4204dc48db42dd03d081987f3cf77268e9c39d0fc3203ce8a9c174a7bc0ea89e/merged`: Permission denied: OCI permission denied

# Adding keep-id because we want to keep the local user
$ podman run --rm -it --userns=keep-id alpine sh
Error: crun: open `/mnt/disk/lotendan/.local/containers/storage/overlay/881f9140b07a3c526bb7002b8587a64ad0491d2a95d1400234413aa7c1458b08/merged`: Permission denied: OCI permission denied

# Need to specify the user to use
podman run --rm -it --userns=keep-id --user=1001:1001 alpine sh
Error: crun: open `/mnt/disk/lotendan/.local/containers/storage/overlay/143fc71fdf06fdc9877ba70e0dede351d28f849fa57c81a96ba8669d8a8dd61c/merged`: Permission denied: OCI permission denied

# Now we are missing some groups, let's add them
$ podman run --rm -it --userns=keep-id --user=1001:1001 --group-add=keep-groups alpine sh
~ $ id
uid=1001(lotendan) gid=1001(lotendan) groups=65534(nobody),1001(lotendan),65534(nobody)

# Volumes also work
$ podman run --rm -it --userns=keep-id --user=1001:1001 --group-add=keep-groups -v $(pwd)/data:/data alpine sh
~ $ touch /data/foo.txt
~ $

Note: --privileged did not help.
Ok, with this set-up we managed to start a container.

Podman pod (with infra container)

Let's try with a pod now:

# Reset
$ podman system reset

# Create the pod with userns like before
$ podman pod create --userns=keep-id mypod
92d7e1d6107df88bb07099c30abe46a90eb83b3d294df5ac3ea8942a7bc6cc73

# Add a container
# Note: cannot specify --userns here since we are using --pod -> OK
$ podman run -it --rm --pod=mypod --user=1001:1001 --group-add=keep-groups -v $(pwd)/data:/data alpine sh
ERRO[0000] Starting some container dependencies
ERRO[0000] "crun: open `/mnt/disk/lotendan/.local/containers/storage/overlay-containers/a3db3412dd13a991e2ca8aed4f9e78fbfcfa67cb56d59ce2c0a51f5c081adadf/rootfs/merge`: Permission denied: OCI permission denied"
Error: starting some containers: internal libpod error

# The error means that it is the infra container failing to start
$ podman pod start mypod
Error: starting container a3db3412dd13a991e2ca8aed4f9e78fbfcfa67cb56d59ce2c0a51f5c081adadf: crun: open `/mnt/disk/lotendan/.local/containers/storage/overlay-containers/a3db3412dd13a991e2ca8aed4f9e78fbfcfa67cb56d59ce2c0a51f5c081adadf/rootfs/merge`: Permission denied: OCI permission denied

# Trying with "other" permissions and bypass group checks
$ sudo chmod 0755 /mnt/disk

# Now it works, which indicates that we were missing the group "disk" while creating the infra container
$ podman pod start mypod
mypod

# Restore permissions
$ sudo chmod 0750 /mnt/disk

This means that, if we were able to pass groups when creating the infra container using podman pod create, there would be no issue.

Podman pod (without infra container)

For the sakes of testing now let's try without infra container:

# Reset
$ podman system reset

# With or without --userns leads to the same result
$ podman pod create --infra=false mypod
23901c7953fe5e11447b968c761a0aa7bc3aa0f4ad77aabe00a4d1de7ec73953

# Pod starts, but getting permissions denied as the users and groups are not properly mapped
podman run --rm -it -v $(pwd)/data:/data --user=1001:1001 --pod=mypod --group-add=keep-groups alpine sh
~ $ touch /data/bar.txt
touch: /data/bar.txt: Permission denied
~ $ id
uid=1001(lotendan) gid=1001(1001) groups=65534(nobody),0(root),65534(nobody)   # Note we have "root" group, this is undesirable.

# Let's cheat a bit and emulate --userns in the container:
$ podman run --rm -it -v $(pwd)/data:/data --user=1001:1001 --uidmap=0:1:1001 --uidmap=1001:0:1 --uidmap=1002:1002:64535 --gidmap=0:1:1001 --gidmap=1001:0:1 --gidmap=1002:1002:64535 --pod=mypod --group-add=keep-groups alpine sh
~ $ id
uid=1001(lotendan) gid=1001(1001) groups=65534(nobody),1001(1001),65534(nobody)
~ $ groups
1001 nobody nobody
~ $ touch /data/bar.txt
~ $

That being said, running without infra container is not always desirable, because we lose many of the functionalities it provides.

Issues

We have no ability to add groups to podman pod create to be passed down to the infra container for creation.
Therefore we cannot satisfy filesystem permission checks around linux groups when the command creates the infra container.
Or, for specific cases like these, does it make sense to manually add an infra container with podman run using the pause image and adding the relevant groups with --group-add?
In such a deployment, do we still benefit from the functionalities that an infra container would natively provide?

How do you recommend to proceed with this setup? When willing to create a pod with an infra container in rootless mode, and where groups matter?
Do I miss something in /etc/subuid and /etc/subgid?
Anything else I could do to make it work in pod mode?

Thanks a lot for the help.

Describe the results you received

Receiving OCI permission denied error in this specific setup when we are subject to linux group permissions on folders.

Describe the results you expected

Expect the pod to be created with an infra container in place.

podman info output

host:
  arch: arm64
  buildahVersion: 1.41.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 99.94
    systemPercent: 0.03
    userPercent: 0.03
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: archarm
    version: unknown
  eventLogger: file
  freeLocks: 2046
  hostname: lotendan
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.17.7-2-aarch64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 2950484032
  memTotal: 19148365696
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-1.24-1
    path: /usr/bin/crun
    version: |-
      crun version 1.24
      commit: 54693209039e5e04cbe3c8b1cd5fe2301219f0a1
      rundir: /run/user/1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_09_19.623dbf6-1
    version: |
      pasta 2025_09_19.623dbf6
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1001/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 209h 52m 14.00s (Approximately 8.71 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /mnt/disk/lotendan/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /mnt/disk/lotendan/.local/containers/storage
  graphRootAllocated: 1081101176832
  graphRootUsed: 36088987648
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1001/containers
  transientStore: false
  volumePath: /mnt/disk/lotendan/.local/containers/storage/volumes
version:
  APIVersion: 5.6.2
  Built: 1759423108
  BuiltTime: Thu Oct  2 18:38:28 2025
  GitCommit: 9dd5e1ed33830612bc200d7a13db00af6ab865a4
  GoVersion: go1.25.1 X:nodwarf5
  Os: linux
  OsArch: linux/arm64
  Version: 5.6.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

N/A

Additional information

Thanks a lot for the help!

[mheon]

To start: --user is unnecessary when --userns=keep-id is passed. Podman will set it automatically to ensure that the container starts as the appropriate UID/GID.

For the rest: I don't think we've ever encountered an environment where the use of supplemental groups was necessary simply for Podman to function. I am a little reluctant to add --group-add for the infra container because, aside from your use-case, I can think of no legitimate usage for it; it should never be necessary for the infra container to have access to groups within the container, so the only legitimate usage is keeping external groups for unusual storage situations like your own. I suppose we could add the flag but disallow every argument except keep-groups? Unfortunately, I don't think there's a way to determine it's necessary in an automated fashion, so we can't entirely rid ourselves of it.

I'm going to retag this as an RFE because of the narrow scope of this (again, never encountered a similar storage setup), but if someone wants to add the option to podman pod create I don't think there will be a problem.

[Lotendan]

Thank you for the detailed response.
I understand the concerns.

To start: --user is unnecessary when --userns=keep-id is passed. Podman will set it automatically to ensure that the container starts as the appropriate UID/GID.

Indeed that is true for a container. For a pod I tested and since we cannot combine --pod and --userns we need to specify a user (even if the pod is created with podman pod create --userns).

For the rest is that possible that I’m simply missing mappings from the subuid/subgid?
Or as a workaround, would that work to create a pod without infra container and then add a pause image to it? (How to reference the pause image from podman run?)

Again thanks 👍

[Luap99]

@giuseppe I know I asked this before but wouldn't this be solved if we always use the intermediate storage mount path?
49eb5af

I really think given all these storage access issues we should just do that.

[giuseppe]

@Luap99 I've no strong opinion, and I usually like the idea of having just one way of doing things, but there are some drawbacks.

We were always using the intermediate mount namespace in the past, but we had some weird issues with mount propagations (we probably still have a race condition if a new mount namespace is created between the intermediate mount creation and setting it MS_SLAVE) so I just avoided the issue by not requiring the intermediate mount namespace and limiting it only to cases when it is really necessary.
One thing we could play with now, and that wasn't available when we first added this feature, is the new mount API to create the bind mount without the race condition.

The additional mount namespace won't probably have any visible impact on performance and memory usage.

[Luap99]

yeah given we are on the 6.0 cycle now I think it would be a good time to start using the new mount api where it makes sense and not even worry about the fall backs. If I reading this right fsopen and the related calls were added in kernel 5.2 so I think it is fair to say we don't need to worry about older kernels assuming we don't need even newer features of that api.

I guess in general it would be nice to have a kernel baseline version properly defined and tracked that we know podman will work on.

[giuseppe]

It has been around long enough that I think it is safe to assume it is always available