singularity in docker <23.0.0: Requesting capability set 0x000001ffffffffff while permitted capability set is 0x0000003fffffffff

[jpfeuffer]

Version of Singularity
What version of Singularity are you using?
3.11.0 on GitHub Actions using the following Dockerfile: https://github.com/singularityhub/singularity-docker/blob/v3.11.0/Dockerfile

Describe the bug
see title.

To Reproduce
Steps to reproduce the behavior:
clone https://github.com/OpenMS/dockerfiles/blob/master/library/Dockerfile
then

  spython recipe dockerfiles/library/Dockerfile &> dockerfiles/library/Singularity.def
  sed -i "s/(^From:.*:)(.*)/\1latest/" dockerfiles/library/Singularity.def
  sed -i "s%(^OPENMS_BRANCH=)(.*)%\1nightly%" dockerfiles/library/Singularity.def
  sed -i "s/(^OPENMS_TAG=)(.*)/\1latest/" dockerfiles/library/Singularity.def
  cat dockerfiles/library/Singularity.def
  sudo singularity build library.sif dockerfiles/library/Singularity.def

see https://github.com/OpenMS/OpenMS/actions/runs/4495601964/jobs/7909390460

Expected behavior
Build the container, as with 3.10

OS / Linux Distribution
Host (GitHub runner):

• Image: ubuntu-22.04
• Version: 20230317.1

Docker:

• FROM golang:1.20.0-alpine
• options: --privileged

Installation Method
wget https://github.com/sylabs/singularity/releases/download/v${SINGULARITY_VERSION}/singularity-ce-${SINGULARITY_VERSION}.tar.gz

Additional context
Anything else which might be relevant. E.g. if the bug only occurs on a specific filesystem, or kernel version etc.

[dtrudg]

Please could you try to provide a reproducer that doesn't depend on using GitHub actions. i.e. what local docker usage and commands lead to the same issue?

A complex GitHub actions flow as a reproducer is a bit of a stretch for us to work with. Thanks.

[dtrudg]

Ahhh... what version of Docker is being used in GitHub actions? I think this may be caused by Docker < v23.0.0 on hosts with a new enough kernel to provide some new caps.

Singularity 3.11 introduced handling of the new CAP_PERFMON, CAP_BPF, and CAP_CHECKPOINT_RESTORE capabilities.

For a while, Docker didn't set these even with --privileged... due to an issue in runc versions that didn't know about them.

Singularity can tell, from the host information visible in the Docker container... that the host provides all those new capabilities.. so it is trying to set the full range. However, Docker has denied the 3 new caps due to this runc workaround.

I'm betting that if you use Docker v23.0.0 or later, which has this patch... moby/moby@c1c973e ... then it will work.

[jpfeuffer]

This?
Ubuntu 22.04 Github Runner software:

Docker Compose v1 1.29.2
Docker Compose v2 2.16.0+azure-2
Docker-Buildx 0.10.4
Docker-Moby Client 20.10.23+azure-2
Docker-Moby Server 20.10.23+azure-2

[dtrudg]

This? Ubuntu 22.04 Github Runner software:

Docker Compose v1 1.29.2 Docker Compose v2 2.16.0+azure-2 Docker-Buildx 0.10.4 Docker-Moby Client 20.10.23+azure-2 Docker-Moby Server 20.10.23+azure-2

Yes... arguably we could handle this better in Singularity, so that we gracefully accept the permitted set. We are hitting the same issue containerd had when running inside Docker:

containerd/containerd#5017

I can't promise that would be implemented quickly, though.

[jpfeuffer]

Ok I think we can wait until the runners are updated.

Any other workarounds, e.g. by listing all necessary permissions instead of using privileged? Not sure if this would make sense, I am not very familiar with linux privileges.

[dtrudg]

Any other workarounds, e.g. by listing all necessary permissions instead of using privileged? Not sure if this would make sense, I am not very familiar with linux privileges.

I don't think so. That version of docker isn't aware of the existence of the newer capabilities... so you won't be able to ask it to provide them.