Add `docker buildx` compatible Dockerfile

[matthewfeickert]

On the IRIS-HEP Slack @aperloff has created a minimal docker buildx Dockerfile that will build:

I figured out how to do this same thing with docker buildx and still have aperloff/cms-cvmfs-docker as the base image. I haven’t figured out all of the security implications though, so keep that in mind.

First I created a new Dockerfile, just so that I could test a simple build. It looks like:

# syntax=docker/dockerfile:1.3-labs

FROM aperloff/cms-cvmfs-docker:latest

USER root

ARG ARG_CVMFS_MOUNTS
ARG ARG_MY_UID
ARG ARG_MY_GID

ENV CVMFS_MOUNTS=$ARG_CVMFS_MOUNTS
ENV MY_UID=$ARG_MY_UID
ENV MY_GID=$ARG_MY_GID

RUN --security=insecure source /mount_cvmfs.sh  && \
    mount_cvmfs && \
    ls /cvmfs/cms.cern.ch && \
    source /home/cmsusr/.bashrc && \
    cmsrel CMSSW_12_0_0 && \
    ls -alh

ENTRYPOINT ["/run.sh"]

A few things to note. One is the leading line (# syntax=docker/dockerfile:1.3-labs), which allows you to use some experimental syntax features. I’m not sure yet if this is still needed, but it was needed for the references I was looking at. The other thing is that the run command where CVMFS is mounted needs --security=insecure. Another thing is that CVFMS must be mounted as the root user and then you can lower the permissions later. So you can’t set USER cmsusr and still mount CVMFS. Finally, the mount only lasts for the RUN command in which it was started. So you can’t start the mount in one layer and use it in another (at least not with these commands).

Then I executed the build using the following commands:

docker buildx create --driver-opt image=moby/buildkit:master --use --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure'
docker buildx use insecure-builder
docker buildx build --load --allow security.insecure --build-arg ARG_CVMFS_MOUNTS="cms.cern.ch oasis.opensciencegrid.org" --build-arg ARG_MY_UID=$(id -u) --build-arg ARG_MY_GID=$(id -g) -t cms-cvmfs-docker:test .
docker buildx rm insecure-builder

Notice that similar build arguments are passed to the build command as you would use to start a container using the base image. The other important pieces are --load to save the output image into the local database. You could use --push to send the image directly to a registry. Then there is --allow security.insecure , which is needed to allow for the mounting of CVMFS.
Once the build is done you can start a container using the same commands as before:

> docker run --rm -it --device /dev/fuse --cap-add SYS_ADMIN -e CVMFS_MOUNTS="cms.cern.ch oasis.opensciencegrid.org" -e MY_UID=$(id -u) -e MY_GID=$(id -g) cms-cvmfs-docker:test

Mounting the filesystem "cms.cern.ch" ... DONE
Mounting the filesystem "oasis.opensciencegrid.org" ... DONE
Checking CVMFS mounts ... DONE
	The following CVMFS folders have been successfully mounted:
		cms.cern.ch
		oasis.opensciencegrid.org
[cmsusr@0f937708359b ~]$ ll
total 8
drwxr-xr-x 1 cmsusr games 4096 Jun 16 22:45 CMSSW_12_0_0

Note, it doesn’t matter that CMSSW was checked out as the root user since /run.sh chowns all of the files in /home/cmsusr. I suppose to be on the safe side I should have specified my WORKDIR, but that’s a detail I leave for you.

Hope this helps you simplify your build approach.

I'm having some trouble getting a more complex case to work in PR #2, but I think that is because of lines like

...
    cmsenv && \
    python3 -m pip --no-cache-dir --verbose install --upgrade --user pip setuptools wheel && \
    python3 -m pip --no-cache-dir --verbose install --ignore-installed --upgrade --user 'pyhf[xmlio,minuit,contrib]' && \
...

when the USER is still root but we need the USER to be cmsusr. I'll debug more.

[aperloff]

The problem is you need to be root in order to mount CVMFS, but you can't split into multiple RUN command and still have CVMFS mounted on both (each RUN command is a new layer/shell). What about using su as in https://github.com/aperloff/cms-cvmfs-docker/blob/master/cvmfs/run.sh#L34 when you want to lower permission and be USER?