Buildah 1.44.0 fails to pull images with opaque whiteouts when running in a privileged user namespace

[chmeliik]

Issue Description

We run buildah under an unshare --map-root-user --map-auto --mount namespace. In version 1.44.0, this causes buildah to fail when pulling images whose layers contain opaque whiteouts:

Error: creating build container: unable to copy from source docker://quay.io/devfile/python@sha256:54924a2ee4a2ef17028ae076ce38e59b3f4054353a5c9f9318dfaee60377532c: writing blob: adding layer with blob "sha256:778656c04542093db6d3b6e07bffbcf6ec4b24709276be7cdf177fcb3666663a"/""/"sha256:d326469892d974408d96f1e02d64dce10d20f88613688af11e99e3e22523beeb": unpacking failed (error: exit status 1; output: time="2026-06-09T19:55:43Z" level=warning msg="Failed to read CAP_SYS_ADMIN presence for the current process"
time="2026-06-09T19:55:43Z" level=warning msg="Failed to read current user namespace mappings"
operation not permitted)

The same thing occurs when running buildah in a podman --privileged container

According to my debugging, the root cause is a bug in go.podman.io/storage that got exposed by ce7811f:

• The storage-untar re-exec handler calls chroot() first before calling unshare.IsRootless() to decide what xattr name to use for the opaque whiteout (user.overlay.opaque vs trusted.overlay.opaque)
• chroot() does something that causes IsRootless() to fail both of the secondary checks (CAP_SYS_ADMIN presence, full user mappings), so it returns false
• Attempts to set trusted.overlay.opaque even though we're not real root => operation not permitted

Prior to v1.44.0, it worked fine, because something in init() called unshare.IsRootless() before the chroot happened, so the correct value was already cached by then. But now it's not called until after the chroot.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a Containerfile to install buildah and dependencies:

expand

FROM registry.access.redhat.com/ubi10/go-toolset:1.26.3 AS go-build

ARG BUILDAH_TAG=v1.44.0

USER 0

# Install dependencies for compiling buildah
RUN dnf -y install \
        bzip2 \
        glib2-devel \
        gpgme-devel \
        libassuan-devel \
        libseccomp-devel

WORKDIR /build

RUN git clone --branch=$BUILDAH_TAG https://github.com/podman-container-tools/buildah && \
    cd buildah && \
    make bin/buildah


FROM registry.access.redhat.com/ubi10/ubi-minimal:10.1

USER 0

RUN microdnf -y install \
        containers-common-extra \
        gpgme \
        libassuan \
        libseccomp \
        util-linux-core

COPY --from=go-build /build/buildah/bin/buildah /usr/local/bin/buildah

COPY <<EOF /etc/containers/storage.conf
[storage]
driver = "overlay"

[storage.options.overlay]
mountopt = "nodev"
EOF

# Set up mapping for unshare
RUN echo "root:1:65535" | tee /etc/sub{uid,gid}

ENV BUILDAH_ISOLATION=chroot

VOLUME /var/lib/containers/storage
# work around https://github.com/podman-container-tools/buildah/issues/6892
VOLUME /var/tmp

2. build it: buildah build -t buildah:1.44.0 .
3. try to pull an image with opaque whiteouts:

   podman run --rm --privileged buildah:1.44.0 buildah pull quay.io/devfile/python:slim@sha256:54924a2ee4a2ef17028ae076ce38e59b3f4054353a5c9f9318dfaee60377532c
   # results in error message above

4. optionally, repeat without --privileged but with unshare:

   podman run --rm buildah:1.44.0 unshare --map-root-user --map-auto --mount -- buildah pull quay.io/devfile/python:slim@sha256:54924a2ee4a2ef17028ae076ce38e59b3f4054353a5c9f9318dfaee60377532c
   # same error

Describe the results you received

Error when pulling image

Describe the results you expected

Successfully pull image

buildah version output

Version:         1.44.0
Go Version:      go1.26.3 (Red Hat 1.26.3-4.el10_2)
Image Spec:      1.1.1
Runtime Spec:    1.3.0
image Version:   5.40.0
Git Commit:      30a4189415e5d0a3a0c5971307a6432d8ed1a097
Built:           Tue Jun  9 21:20:31 2026
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

buildah info output

{
    "host": {
        "Distribution": {
            "distribution": "\"rhel\"",
            "version": "10.1"
        },
        "MemFree": 2014158848,
        "MemTotal": 33055121408,
        "OCIRuntime": "crun",
        "SwapFree": 8571138048,
        "SwapTotal": 8589930496,
        "arch": "amd64",
        "cpus": 14,
        "hostname": "b0b279f01167",
        "kernel": "7.0.11-100.fc43.x86_64",
        "os": "linux",
        "rootless": true,
        "uptime": "38h 6m 32.3s (Approximately 1.58 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphImageStore": "",
        "GraphOptions": null,
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "btrfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Supports shifting": "false",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "GraphTransientStore": false,
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

Provide your storage.conf

[storage]
driver = "overlay"

Upstream Latest Release

Yes

Additional environment details

Happens only when running as root in the user namespace, i.e. unshare --map-root-user or podman run --privileged --user=0

Additional information

Note on why it only happens under podman run --privileged or unshare --map-root-user: I believe it's due to MaybeReexecUsingUserNamespace deciding that it's not necessary to re-exec. In an unprivileged namespace, the re-exec sets the special environment variables that make unshare.IsRootless() return true right away, without having to fall back to the secondary checks. In a namespace that already has CAP_SYS_ADMIN, this doesn't happen.

[chmeliik]

As a workaround until this is fixed, I'm thinking of setting the _CONTAINERS_USERNS_CONFIGURED=done env var to make IsRootless() return true. As far as I can tell, it shouldn't break our use case. Though obviously we don't want this to be more than a short-term workaround since it's an internal implementation detail and may have unexpected side effects.

[Luap99]

I guess that should be moved to storage then, we need to call unshare.IsRootless() before we chroot there.

[chmeliik]

I guess that should be moved to storage then, we need to call unshare.IsRootless() before we chroot there.

Makes sense to me. Should I create a similar issue in https://github.com/podman-container-tools/container-libs?

[Luap99]

Not needed, I can handle it from here.

[Luap99]

podman-container-tools/container-libs#908 should fix it

[TomSweeneyRedHat]

Thanks @Luap99. So I'm hearing that we'll need to backport this into the Podman-6.0 branch here, bump storage, then vendor into Buildah, create Buildah v1.44.1, then vendor that into Podman, hopefully in time to vendor Buildah v1.44.1 into Podman for an RC of Podman 6. Holler if I missed anything.

[Luap99]

Thanks @Luap99. So I'm hearing that we'll need to backport this into the Podman-6.0 branch here, bump storage, then vendor into Buildah, create Buildah v1.44.1, then vendor that into Podman, hopefully in time to vendor Buildah v1.44.1 into Podman for an RC of Podman 6. Holler if I missed anything.

I have a better idea.

First this does not affect podman as there should be unshare.IsRootless() on its init path somewhere still so no podman updates needed.

As for storage well what I proposed is the correct thing for all library users. However the most simple fix is adding a

init() {
_ = unshare.IsRootless()
}

function into buildah again, I can do that against the the 1.44 branch then no extra vendor from container-libs needed which should be much simpler and less work for us.

[TomSweeneyRedHat]

Works for me, thanks @Luap99