rootless podman not working inside podman --runtime krun

[krzsztf]

While running $ podman --runtime krun run --userns nomap --rm -it localhost/podman-in-krun:latest and trying to launch rootless podman inside, I get error:

Error: unable to copy from source docker://alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:6a0ac1617861a677b045b7ff88545213ec31c0ff08763195a70a4a5adda577bb"/""/"sha256:29df493baa13de438d6d2ece3a8333032e0b7b9b9d8cce4ee82194da255f61e1":
unpacking failed (error: exit status 1; output: setting up pivot dir: mkdir ./.pivot_root737267738: permission denied)

Full output:

tty: ttyname error: Inappropriate ioctl for device
[root@7091acef9b50 ~]# podman run --net=host --mount type=bind,source=/dev/null,target=/dev/mqueue alpine id
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 6a0ac1617861 done   | 
Copying config 3cb067eab6 done   | 
Writing manifest to image destination
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
[root@7091acef9b50 ~]# su - podman
tty: ttyname error: Inappropriate ioctl for device
[podman@7091acef9b50 ~]$ podman run --net=host --mount type=bind,source=/dev/null,target=/dev/mqueue alpine id
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob 6a0ac1617861 done   | 
Error: unable to copy from source docker://alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:6a0ac1617861a677b045b7ff88545213ec31c0ff08763195a70a4a5adda577bb"/""/"sha256:29df493baa13de438d6d2ece3a8333032e0b7b9b9d8cce4ee82194da255f61e1": unpacking failed (error: exit status 1; output: setting up pivot dir: mkdir ./.pivot_root737267738: permission denied)
[podman@7091acef9b50 ~]$ podman system info
host:
  arch: amd64
  buildahVersion: 1.43.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.2.1-2.fc44.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.2.1, commit: '
  cpuUtilization:
    idlePercent: 99.86
    systemPercent: 0.11
    userPercent: 0.04
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: container
    version: "44"
  eventLogger: file
  freeLocks: 2048
  hostname: 7091acef9b50
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 1
      size: 999
    - container_id: 1000
      host_id: 1001
      size: 64535
  kernel: 6.12.76
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 925675520
  memTotal: 1068728320
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    defaultNetwork: podman
    dns:
      package: aardvark-dns-1.17.1-1.fc44.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.17.1
    package: netavark-1.17.2-1.fc44.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.17.2
  ociRuntime:
    name: crun
    package: crun-1.27.1-1.fc44.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.27.1
      commit: 3ec076b3b6714ec2f1a10533cf18d5605a6de637
      rundir: /tmp/storage-run-1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20260120.g386b5f5-1.fc44.x86_64
    version: |
      pasta 0^20260120.g386b5f5-1.fc44.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /tmp/storage-run-1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 0h 5m 55.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/podman/.local/share/containers/storage
  graphRootAllocated: 1022488477696
  graphRootUsed: 827270365184
  graphStatus:
    Backing Filesystem: <unknown>
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/storage-run-1000/containers
  transientStore: false
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 5.8.2
  BuildOrigin: Fedora Project
  Built: 1776124800
  BuiltTime: Tue Apr 14 00:00:00 2026
  GitCommit: 5b263b5f5b48004a87caac44e67349a8266d9ef4
  GoVersion: go1.26.1-X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.8.2

Containerfile:

FROM quay.io/podman/stable

WORKDIR /root

COPY --chmod=755 <<EOF /entrypoint.sh 
setcap cap_setuid+ep /usr/bin/newuidmap
setcap cap_setgid+ep /usr/bin/newgidmap
bash
EOF

ENTRYPOINT /entrypoint.sh

[mg1986jp]

The mkdir ./.pivot_root...: permission denied error is from crun trying to set up the container rootfs inside the krun microVM. The issue is that rootless podman inside krun can't perform pivot_root because the overlay storage's upper dir doesn't have the right permissions in the VM context.

The rootful case works because root can write to the overlay upperdir. The rootless podman user fails because inside the krun VM, the overlay mount created by the outer container's storage doesn't allow the inner user namespace to create files.

Two fixes:

1. Use fuse-overlayfs for the inner podman (most reliable for nested rootless):

Add to your Containerfile:

FROM quay.io/podman/stable

RUN dnf install -y fuse-overlayfs && dnf clean all

# Configure rootless podman to use fuse-overlayfs
RUN mkdir -p /home/podman/.config/containers && \
    echo '[storage]' > /home/podman/.config/containers/storage.conf && \
    echo 'driver = "overlay"' >> /home/podman/.config/containers/storage.conf && \
    echo '[storage.options.overlay]' >> /home/podman/.config/containers/storage.conf && \
    echo 'mount_program = "/usr/bin/fuse-overlayfs"' >> /home/podman/.config/containers/storage.conf && \
    chown -R podman:podman /home/podman/.config

2. Use vfs storage driver (simpler but slower, uses more disk):

# As the podman user inside the krun container:
podman --storage-driver vfs run --net=host alpine id

Or set it permanently:

# /home/podman/.config/containers/storage.conf
[storage]
driver = "vfs"

The root cause: krun uses libkrun to run a minimal VM with virtiofs for the filesystem. Native overlay inside virtiofs + user namespaces has known issues with directory creation in the upperdir. fuse-overlayfs bypasses this because it operates entirely in userspace via FUSE rather than relying on kernel overlay support.

Also, the setcap approach in your entrypoint might not persist correctly inside krun since the VM's filesystem is virtiofs-backed. You might want to move those to the Containerfile's RUN layer instead.