Encountered error `[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted` when build the docker image

[ponkio-o]

Hi, I'm using rootless docker for use Docker in Docker (DinD). I need a hand with an error like the titile.

What happened?

We are using the following Dockerfile (and entrypoint), but the build has not been going through in the last week or so.
(In other words, the build was working correctly until last week)

FROM ubuntu:22.04

RUN apt-get update && apt-get install -y --no-install-recommends \
  kmod \
  dbus-user-session \
  curl \
  ca-certificates \
  uidmap \
  openssl \
  gnupg \
  gpg \
  iproute2 \
  && rm -rf /var/lib/apt/lists/* \
  && rm -rf /tmp/*

RUN install -m 0755 -d /etc/apt/keyrings; \
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg; \
    chmod a+r /etc/apt/keyrings/docker.gpg; \
    echo \
      "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
      "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
     tee /etc/apt/sources.list.d/docker.list > /dev/null
RUN apt-get update && apt-get upgrade -y; \
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras

RUN echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf; \
    echo "user.max_user_namespaces=28633" >> /etc/sysctl.conf; \
    sysctl --system

RUN set -eux; \
    addgroup --system dockremap; \
    adduser --system --no-create-home --ingroup=dockremap dockremap; \
    echo 'dockremap:165536:65536' >> /etc/subuid; \
    echo 'dockremap:165536:65536' >> /etc/subgid

RUN set -eux; \
    addgroup --system docker; \
    addgroup --system --gid=1000 rootless; \
    adduser --disabled-password --uid=1000 --ingroup=rootless rootless; \
    adduser rootless docker

COPY rootless_entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

RUN mkdir -p /home/rootless/.local/share/docker
RUN chown -R rootless:rootless /home/rootless/.local/share/docker
RUN mkdir /certs /certs/client && chmod 1777 /certs /certs/client
USER rootless
RUN dockerd-rootless-setuptool.sh install --skip-iptables

CMD ["/entrypoint.sh"]

The rootless_entrypoint.sh is below.

#!/bin/bash

echo "Generating Docker TLS certificate and key..."
mkdir -p /certs
openssl genrsa -out /certs/key.pem 4096
openssl req -new -x509 -sha256 -key /certs/key.pem -out /certs/cert.pem -days 365 -subj '/CN=localhost'

echo "Starting Docker..."
dockerd-rootless.sh -H ${DOCKER_HOST} --tlsverify --tlscacert=/certs/cert.pem --tlscert=/certs/cert.pem --tlskey=/certs/key.pem

The error message is as follows:

$ ls
Dockerfile              rootless_entrypoint.sh

$ docker build -t rootless-docker .
 => ERROR [13/13] RUN dockerd-rootless-setuptool.sh install --skip-iptables                                                                    0.1s
------
 > [13/13] RUN dockerd-rootless-setuptool.sh install --skip-iptables:
0.104 [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
0.104 [ERROR] RootlessKit failed, see the error messages and https://rootlesscontaine.rs/getting-started/common/ .
------
Dockerfile:49
--------------------
  47 |     RUN mkdir /certs /certs/client && chmod 1777 /certs /certs/client
  48 |     USER rootless
  49 | >>> RUN dockerd-rootless-setuptool.sh install --skip-iptables
  50 |
  51 |     CMD ["/entrypoint.sh"]
--------------------
ERROR: failed to solve: process "/bin/sh -c dockerd-rootless-setuptool.sh install --skip-iptables" did not complete successfully: exit code: 1

Environment

Build machine: Mac Studio Apple M2 Ultra
OS: macOS 13.6
Docker: Docker version 25.0.3, build 4debf41

What do I do to solve this problem?

[ponkio-o]

Commonly, this problem ocurre when the value of /proc/sys/kernel/unprivileged_userns_clone is set to 0.
https://docs.docker.com/engine/security/rootless/#errors-when-starting-the-docker-daemon

But I did set the kernel.unprivileged_userns_clone=1 to /etc/sysct.conf in Dockerfile.

[sebthom]

I am facing exactly the same issue trying to build rootless dind container (with Docker version 24.0.9, build 2936816) on Github Actions (ubuntu-22.04) runners using debian-stable base image.

Things were working fine until beginning of last week, now suddenly same build scripts fail with this error.

Running sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone" or sudo sysctl -w kernel.unprivileged_userns_clone=1 before the docker build on the host system has no effect. the error persists.

[sebthom]

I tried different versions of rootlesskit (2.0.2, 2.0.1, 1.1.1) all fail now with [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted despite kernel.unprivileged_userns_clone=1.

I also read that AppArmor can be the reason for the error: https://discuss.linuxcontainers.org/t/rootless-docker-on-new-ubuntu-kernels-does-not-work/18708 However when I disabled and even when I removed AppArmor the build still fails with [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted

[ponkio-o]

@sebthom Thank you for your response.

I tried different versions of rootlesskit (2.0.2, 2.0.1, 1.1.1) all fail now with [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted despite kernel.unprivileged_userns_clone=1.

I reproduced the same behavior when I tried docker build use with rootlesskit (v2.0.2, v2.0.1).

Upon investigation I noticed that a new version of docker-ce-rootless-extras:5:25.0.4-1~ubuntu.22.04~jammy has been released.
The build was successful with the following changes to the Dockerfile I initially presented.

diff --git a/rootless-docker/Dockerfile b/rootless-docker/Dockerfile
index be8105d..fc7cacd 100644
--- a/rootless-docker/Dockerfile
+++ b/rootless-docker/Dockerfile
@@ -30,7 +30,9 @@ RUN install -m 0755 -d /etc/apt/keyrings; \
       "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
      tee /etc/apt/sources.list.d/docker.list > /dev/null
 RUN apt-get update && apt-get upgrade -y; \
-    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
+    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+
+RUN apt-get install -y docker-ce-rootless-extras=5:25.0.3-1~ubuntu.22.04~jammy --allow-downgrades

rootlesskit version is v2.0.1.

$ docker build --no-cache -f Dockerfile --progress=plain -t rootless-docker-test .
$ docker run --rm -it rootless-docker-test rootlesskit --version
rootlesskit version 2.0.1

I assume from the above that there has been some change in package of docker-ce-rootless-extras. However, it is not yet known which dependencies were changed.

[sebthom]

Pinning the previous version also works on Debian apt-get install --no-install-recommends -y docker-ce-rootless-extras=5:25.0.3-1~debian.12~bookworm --allow-downgrades.
Thanks for the workaround!

[ponkio-o]

Yes, I would also try to avoid the error by this workaround.
But I don't want to keep using an old version, so I need to investigate the root case.

[DenisBalan]

This might be related to AppArmor (at least in my case)
Encountered the same when running containerd-rootless-setuptool.sh check and rootlesskit bash

so need to add an exclusion for AppArmor service (stopping/disabling the service didnt worked out for me)

https://github.com/rootless-containers/rootlesskit/blob/e860cf8b531740e57c687f96bf6a3442a5081f3d/pkg/parent/warn.go

########## BEGIN ##########
cat <<EOT | sudo tee "/etc/apparmor.d/home.ubuntu.bin.rootlesskit"
# ref: https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
abi <abi/4.0>,
include <tunables/global>

/home/ubuntu/bin/rootlesskit flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/home.ubuntu.bin.rootlesskit>
}
EOT
# ref: https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
abi <abi/4.0>,
include <tunables/global>

/home/ubuntu/bin/rootlesskit flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/home.ubuntu.bin.rootlesskit>
}
########## END ##########

[gcooney]

We are also seeing this issue. Started seeing it on March 7th shortly after 2.0.2 was released.

It looks like 2.0.2 added a check that rootlesskit works to dockerd-rootless-setuptool.sh: https://github.com/moby/moby/blob/b32cfc3b3a9d791ed3924ab168f27962dceec949/contrib/dockerd-rootless-setuptool.sh#L273C2-L278C4. Is it possible that rootlesskit can't run successfully outside a privileged container, e.g. in a docker build?

[AkihiroSuda]

dockerd-rootless-setuptool.sh install is designed for systemd environments and it does not make sense to run the script inside Dockerfile.

[OJFord]

@DenisBalan's suggestion fixed it for me, but now I have a different error presumably needing a different AppArmor permission?

time="2024-09-03T13:20:34Z" level=warning msg="failed to get disk size: no such file or directory"
time="2024-09-03T13:20:34Z" level=warning msg="failed to get disk size: no such file or directory"
buildkitd: mkdir /var/run/user/1001: permission denied
[rootlesskit:child ] error: command [buildkitd --addr=unix:///var/run/user/1001/buildkit/rootless.sock] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1

Is there a complete list somewhere @AkihiroSuda (I see you authored the hint Denis linked to above)?

This is running:

socket="unix:///var/run/user/$(id -u)/buildkit/rootless.sock"
rootlesskit --net=slirp4netns --copy-up=/etc --disable-host-loopback buildkitd --addr="$socket" &

This worked fine in Github's ubuntu-24.04 runners, but with 'Ubuntu 24.04 by Arm Limited' for a larger (and aarch64) runner it needed the app armor policy above and slirp4netns & uidmap explicitly installed to get as far as the error above.