UBI9/RHEL PAM account checks fail in rootless Podman on Ubuntu host, causing crontab/pam_unix to reject a valid container user

[saramonzon]

Issue Description

Inside a rootless Podman container based on registry.access.redhat.com/ubi9/ubi, PAM account management fails for a valid normal container user when using RHEL/UBI9 PAM stacks such as crond and login.

The failure appears when running commands such as:

crontab -l

as a non-root user inside the container. crontab fails with a PAM error indicating that the user is not allowed to access crontab.

The same UBI9 image/PAM setup works on a RedHat-like Podman host with SELinux enabled. An Ubuntu container image using Debian cron/PAM also works on the same Ubuntu host where UBI9 fails.

This looks like a possible interaction between:

• UBI9/RHEL userspace PAM/cronie
• rootless Podman
• Ubuntu 24.04 host, kernel 6.8.0-111-generic

Testing points specifically to pam_unix.so failing during the PAM account phase. pam_access.so alone succeeds, pam_permit.so succeeds, but pam_unix.so fails.

The failing operation is equivalent to:

pam_start("crond", "iskylims", ...)
pam_acct_mgmt(...)

On the failing Ubuntu/rootless Podman host, this returns:

pam_acct_mgmt=9 Authentication service cannot retrieve authentication info

On the working RedHat-like/SELinux Podman host, the same test returns:

pam_acct_mgmt=0 Success

This does not appear to be caused by the application, Django, django-crontab, UID/GID mismatch, broken /etc/shadow, or a generic setuid failure. A custom setuid-root test binary can successfully read /etc/shadow on the failing Ubuntu host.

Steps to reproduce the issue

Run the following on the Ubuntu host using rootless Podman:

podman run --rm -it registry.access.redhat.com/ubi9/ubi bash -lc '
set -eux
dnf -y install cronie pam shadow-utils gcc

useradd -m -u 1212 -s /sbin/nologin iskylims || true

echo "User:"
id iskylims
getent passwd iskylims
grep "^iskylims:" /etc/shadow

echo "PAM crond:"
cat /etc/pam.d/crond
cat /etc/pam.d/system-auth

echo "crontab as iskylims:"
su -s /bin/bash iskylims -c "crontab -l"
echo "exit=$?"
'

Observe that crontab -l fails for the valid container user iskylims.

Replace the crond PAM account stack with pam_permit.so:

cat > /etc/pam.d/crond <<EOF
auth       include    system-auth
account    required   pam_access.so
account    required   pam_permit.so
session    required   pam_loginuid.so
session    include    system-auth
EOF

Run again:

su -s /bin/bash iskylims -c "crontab -l"

Observe that crontab -l now works.

Describe the results you received

Running crontab -l as the valid container user fails:

You (iskylims) are not allowed to access to (crontab) because of pam configuration.

The default UBI9 /etc/pam.d/crond is:

auth       include    system-auth
account    required   pam_access.so
account    include    system-auth
session    required   pam_loginuid.so
session    include    system-auth

system-auth includes:

account     required      pam_unix.so

Testing showed:

pam_access.so alone succeeds.
pam_permit.so succeeds.
pam_unix.so fails.

Replacing account include system-auth with account required pam_permit.so makes crontab -l work.

A direct setuid-root PAM test calling:

pam_start("crond", "iskylims", ...)
pam_acct_mgmt(...)

fails on the Ubuntu host:

as root:
uid=0 euid=0 gid=0 egid=0
pam_start=0 Success
pam_acct_mgmt=9 Authentication service cannot retrieve authentication info

as iskylims:
uid=1212 euid=0 gid=1212 egid=1212
pam_start=0 Success
pam_acct_mgmt=9 Authentication service cannot retrieve authentication info

The same direct PAM test succeeds on a RedHat-like/SELinux Podman host:

as root:
uid=0 euid=0 gid=0 egid=0
pam_start=0 Success
pam_acct_mgmt=0 Success

as iskylims:
uid=1004 euid=0 gid=1004 egid=1004
pam_start=0 Success
pam_acct_mgmt=0 Success

Other PAM services on the failing Ubuntu host with the UBI9 image:

as root:
crond pam_acct_mgmt=9 Authentication service cannot retrieve authentication info
su    pam_acct_mgmt=0 Success
login pam_acct_mgmt=9 Authentication service cannot retrieve authentication info
other pam_acct_mgmt=7 Authentication failure

as iskylims:
crond pam_acct_mgmt=9 Authentication service cannot retrieve authentication info
su    pam_acct_mgmt=9 Authentication service cannot retrieve authentication info
login pam_acct_mgmt=9 Authentication service cannot retrieve authentication info
other pam_acct_mgmt=7 Authentication failure

The following were equivalent or checksum-identical between the failing Ubuntu host container and the working RedHat-like/SELinux host container:

pam-1.5.1-28.el9.x86_64
cronie-1.5.7-16.el9.x86_64
shadow-utils-4.9-16.el9.x86_64
/usr/bin/crontab
/usr/sbin/unix_chkpwd
/usr/lib64/security/pam_unix.so
/lib64/libpam.so.0
/lib64/libcrypt.so.2
/lib64/libc.so.6
/lib64/libnss_files.so.2

The following files are effectively the same except for UID/GID values:

/etc/passwd
/etc/shadow
/etc/pam.d/crond
/etc/pam.d/system-auth
/etc/nsswitch.conf

A setuid-root test binary can successfully read /etc/shadow on the failing Ubuntu host, so generic setuid and generic shadow access appear to work.

On the same Ubuntu host/rootless Podman runtime, an Ubuntu image works:

podman run --rm -it ubuntu:24.04 bash -lc '
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y cron passwd
useradd -m -u 1212 -s /usr/sbin/nologin iskylims || true
su -s /bin/bash iskylims -c "crontab -l"
'

Output:

no crontab for iskylims

So this seems specific to UBI9/RHEL PAM/cronie behavior under this Ubuntu rootless Podman runtime.

Describe the results you expected

expected crontab -l to succeed for the valid container user.

For a user without a crontab, the expected output would be something like:

no crontab for iskylims

or an equivalent successful crontab listing if a crontab already exists.

I did not expect pam_unix.so account checks to fail with:

pam_acct_mgmt=9 Authentication service cannot retrieve authentication info

especially because:

• The same UBI9 PAM/cronie setup works on a RedHat-like/SELinux Podman host.
• A Debian/Ubuntu cron/PAM setup works on the same Ubuntu rootless Podman host.
• The relevant UBI9 PAM/cronie binaries and libraries are checksum-identical between the failing and working containers.
• Generic setuid and generic /etc/shadow access via a custom setuid-root test binary work on the failing host.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 96.69
    systemPercent: 0.98
    userPercent: 2.33
  cpus: 16
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2044
  hostname: PC-M007021
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1212
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1212
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.8.0-111-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 852488192
  memTotal: 33480753152
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: runc
    package: containerd.io_1.7.25-1_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.2.4
      commit: v1.2.4-0-g6c52b3f
      spec: 1.2.0
      go: go1.22.10
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1212/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 237568
  swapTotal: 2147479552
  uptime: 503h 32m 13.00s (Approximately 20.96 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/smonzon/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.13-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /podman_data/smonzon
  graphRootAllocated: 982820896768
  graphRootUsed: 532148633600
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 34
  runRoot: /run/user/1212
  transientStore: false
  volumePath: /podman_data/smonzon/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3


Security: {"apparmorEnabled":false,"capabilities":"CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT","rootless":true,"seccompEnabled":true,"seccompProfilePath":"/usr/share/containers/seccomp.json","selinuxEnabled":false} 

ID mappings: 
{"gidmap":[{"container_id":0,"host_id":1212,"size":1},{"container_id":1,"host_id":165536,"size":65536}],"uidmap":[{"container_id":0,"host_id":1212,"size":1},{"container_id":1,"host_id":165536,"size":65536}]} 
subuid/subgid: /etc/subuid:smonzon:165536:65536 /etc/subgid:smonzon:165536:65536

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Failing environment:

Host OS: Ubuntu 24.04.3 LTS
Kernel: 6.8.0-111-generic
Podman: 4.9.3
OCI runtime: runc 1.2.4
Storage: overlay using fuse-overlayfs 1.13-dev
Rootless: true
SELinux: false
AppArmor enabled according to podman info: false
Seccomp: enabled

Working comparison environment:

RedHat-like Podman host
SELinux enabled
Container SELinux context similar to:
system_u:system_r:container_t:s0:c947,c967
Same UBI9/RHEL PAM/cronie userspace setup works there

Additional information

An Ubuntu 24.04 container image using Debian cron/PAM works on the same failing Ubuntu host/rootless Podman runtime.

The main question is whether this is expected behavior for UBI9/RHEL PAM account checks under rootless Podman on an Ubuntu host. If not, I would appreciate guidance on whether this is more likely to be a Podman/runtime issue, a UBI9/Linux-PAM issue, or an Ubuntu packaging/kernel/runtime issue.

[Luap99]

This does not sounds like podman bug, if the applications inside misbehave then it is likely a specific problem with that code so I move this to a discussion.

We also only support the latest upstream release here, so you would need to check if the problem even happens with the current release, otherwise it is best to report this trough the distro bug channels for the package