Reproducible Docker Builds

[chris-smith-zocdoc]

Is your feature request related to a problem? Please describe.
Currently the docker_image target does not create a reproducible image, even when using pant's remote caching This creates problems for any dependent target that is referencing the docker digest, as those targets inputs will always change, preventing them from being catchable in our CI system.

The simplest version of this problem is related to timestamps of files as they are copied into the docker context. Since these files are copied into the pants sandbox first, the timestamps are always changing, which produces a new docker digest.

Describe the solution you'd like
Buildkit 0.13.0 has made it very easy to re-write these timestamps in the image using the new rewrite-timestamp=true option

Setting the env var SOURCE_DATE_EPOCH also needs to occur so that the dates in the image metadata are stable. It would be nice if pants could derive this from the sources somehow, so that builds across different git commits are stable.

Additional context

Heres a script that showcases this behavior

#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
set -x trace


cat > Dockerfile.test <<EOF
FROM scratch

COPY test.txt /test.txt
EOF

if [ ! -f test.txt ]; then
    touch test.txt
fi

# buildx 0.13 is required for rewrite-timestamp
# https://github.com/moby/buildkit/blob/master/docs/build-repro.md
docker buildx create --use --driver-opt image=moby/buildkit:v0.13.1

build_opts="--tag=test:latest --output=type=docker,rewrite-timestamp=true --file=Dockerfile.test"

SOURCE_DATE_EPOCH=1710990413 docker buildx build $build_opts --iidfile=one.txt .

# remove all local caches/images to force a rebuild
docker system prune --all --force

sleep 1

SOURCE_DATE_EPOCH=1710990413 docker buildx build $build_opts --iidfile=two.txt .

if ! diff one.txt two.txt; then
  echo "Builds are not reproducible"
  exit 1
else
  echo "Builds are reproducible!"
fi

[kaos]

I imagine SOURCE_DATE_EPOCH could be equal to the commit date, in case the working dir is clean.

[lvets]

I'm not sure what the question is in the original description, but these are the steps I needed to take to create reproducible Docker images in Pants on my local dev machine (macOS). I'm fairly new to Pants in general, so there might be settings below that are not necessarily needed or redundant. Additionally, I'm using git locally and Colima instead of Docker.

• Create a .pants.bootstrap file with something like export SOURCE_DATE_EPOCH=$(git log -1 --format='%ct').
• In pants.toml, set:

  [docker]
  use_buildx = true
  env_vars = [
      "SOURCE_DATE_EPOCH",
      "BUILDX_NO_DEFAULT_ATTESTATIONS=1"
  ]

• In your BUILD file, set the following option in your docker_image() target: output={"type":"docker","rewrite-timestamp":"true"},
• Finally, you can only fetch immutable images thus you need to use digests in FROM lines.

With this in place, you should be able to recreate the same docker_image() based on the digests in the Dockerfile and git commit.

[rdeknijf]

@lvets Thanks, this helped me along a lot.
I have it somewhat working too. Locally (Linux, normal Docker), it uses the cached Pex file and therefore recreates the same image every time. But the Pex file still just gets the timestamp of initial creation; not 0 if I set SOURCE_DATE_EPOCH=0
In Gitlab CICD the Pex file is recreated every time (which I didn't expect because remote caching is working fine there), therefore a new image sha is created every time.