Load just file(s) needed in build context rather than full directory tree

[berney]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Setting an additional context that is from another directory to where the Dockerfile is, will copy the whole directory, rather than just the files referenced in the COPY commands.

I logged this regarding the docs, docker/docs#17214.

Expected behaviour

docker buildx build --build-arg stage3_file=stage3-amd64-hardened-nomultilib-openrc-20230423T164653Z.tar.xz --build-context tarball=$HOME/.kubler/downloads -t berne/stage3 .

Will copy only the 247MiB stage3-amd64-hardened-nomultilib-openrc-20230423T164653Z.tar.xz from $HOME/.kubler/downloads as the context.

In the docs https://github.com/docker/docs/blob/4d4a7352d1a60df0af09d74683ae16f2de84a5a6/build/building/context.md?plain=1#L62

It says:

The legacy builder sends the entire directory to the daemon, including bar
and node_modules directories, even though the Dockerfile does not use
them. When using BuildKit, the client only sends the
files required by the COPY instructions, in this case foo.

Actual behaviour

Buildx copies the entire 9.5GiB directory.

Buildx version

github.com/docker/buildx v0.10.4 c513d34

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.4)
  compose: Docker Compose (Docker Inc., v2.17.2)
  dev: Docker Dev Environments (Docker Inc., v0.1.0)
  extension: Manages Docker extensions (Docker Inc., v0.2.19)
  init: Creates Docker-related starter files for your project (Docker Inc., v0.1.0-beta.2)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.25.0)
  scout: Command line tool for Docker Scout (Docker Inc., v0.9.0)

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 94
 Server Version: 20.10.24
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.102.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.763GiB
 Name: docker-desktop
 ID: ZOWA:TFVA:ET24:T2FX:6V5H:65BG:IMK5:STLD:PTXH:LKKL:HYLH:PSSI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
berney *      docker-container                             
  berney0     unix:///var/run/docker.sock running v0.11.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
default       docker                                       
  default     default                     running 20.10.24 linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
desktop-linux                             error

Configuration

# syntax=docker/dockerfile:1
FROM scratch AS tarball
ARG stage3_file
COPY ${stage3_file} /

Build logs

docker buildx build --build-arg stage3_file=stage3-amd64-hardened-nomultilib-openrc-20230423T164653Z.tar.xz --build-context tarball=$HOME/.kubler/downloads -t berne/stage3 .
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
[+] Building 148.7s (6/6) FINISHED
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load build definition from Dockerfile                                                               0.1s
 => => transferring dockerfile: 128B                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                         2.1s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4  0.0s
 => => resolve docker.io/docker/dockerfile:1@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409  0.0s
 => [context tarball] load .dockerignore                                                                           0.0s
 => => transferring tarball: 2B                                                                                    0.0s
 => [context tarball] load from client                                                                           145.7s
 => => transferring tarball: 10.19GB                                                                             145.4s

Additional info

I read https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/ where Example #3: Override a Remote Dependency with a Local One shows overriding a context, which is what I want to do to allow copying the file from elsewhere, but I don't want to copy a huge directory when I just want a single file.

I want to avoiding having to create a directory, link the file, run docker buildx, and then clean up the tmp directory.

If the Dockerfile is in the 9.5GiB ~/.kubler/downloads directory, then docker buildx build --build-arg stage3_file=stage3-amd64-hardened-nomultilib-openrc-20230423T164653Z.tar.xz -t berne/stage3 . will copy just the 247MiB file.

[berney]

Workaround is to cd into the directory, using the minimal dockerfile I posted above: docker buildx build -f $dockerfile -t foo/bar ., to load the single file, so it's now foo/bar image, and then use docker buildx build --build-context tarball=docker-image://foo/bar

[jsternberg]

I'll be taking a look at this to improve this functionality. It appears as though named contexts don't filter the content of the context directory it sends and, as you've discovered in the workaround, the default context does do that correctly.

I'll be working to improve that so you'll see this optimization in both contexts.

At the same time, I wanted to check my understanding of something. In your example Dockerfile:

# syntax=docker/dockerfile:1
FROM scratch AS tarball
ARG stage3_file
COPY ${stage3_file} /

I'm not sure this is a correct Dockerfile to do this anyway. If I'm reading the blog post correctly, the build context replaces this stage tarball with its own content so I believe this entire stage (including the copy) are completely skipped. I don't see any mention of the copy being executed in your build logs. So I think you're just copying the entire directory into the image anyway.

I suspect your Dockerfile needs to be something like this:

# syntax=docker/dockerfile:1
FROM scratch
ARG stage3_file
COPY --from=tarball ${stage3_file} /

In this situation, you are using the named build context as a source and I think that will produce the correct image. I'll double check it to verify, but I believe that's correct.

[jsternberg]

As an example, I created a temporary project with this directory structure.

jsternberg:[~/P/test-context] $ find . -type f | xargs '-I{}' sh -c 'echo \# {} && cat {} && echo'
# ./alt-context/a.txt
Goodbye!

# ./Dockerfile
FROM scratch AS input
COPY a.txt /

FROM alpine AS tarball
COPY --from=input /a.txt /
RUN cat /a.txt

# ./a.txt
Hello, World!

Without any custom build context:

jsternberg:[~/P/test-context] $ docker buildx build -f Dockerfile --progress=plain --no-cache .
#0 building with "local" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 175B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/alpine:latest
#2 DONE 0.4s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [internal] load build context
#4 transferring context: 63B done
#4 DONE 0.0s

#5 [tarball 1/3] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#5 resolve docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a done
#5 CACHED

#6 [input 1/1] COPY a.txt /
#6 DONE 0.0s

#7 [tarball 2/3] COPY --from=input /a.txt /
#7 DONE 0.0s

#8 [tarball 3/3] RUN cat /a.txt
#8 0.032 Hello, World!
#8 DONE 0.0s

Notice how it prints Hello, World! and you can see the copy as [input 1/1].

When I use the custom build context:

jsternberg:[~/P/test-context] $ docker buildx build -f Dockerfile --build-context input=./alt-context --progress=plain --no-cache .
#0 building with "local" instance using docker-container driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 175B done
#1 DONE 0.0s

#2 [context input] load .dockerignore
#2 transferring input: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.4s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [tarball 1/3] FROM docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#5 resolve docker.io/library/alpine:latest@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a done
#5 CACHED

#6 [context input] load from client
#6 transferring input: 63B done
#6 DONE 0.0s

#7 [tarball 2/3] COPY --from=input /a.txt /
#7 DONE 0.0s

#8 [tarball 3/3] RUN cat /a.txt
#8 0.043 Goodbye!
#8 DONE 0.0s

Now we can see the stage input is loaded from the client. We can also see that the copy into the tarball is still present because it says COPY --from=input. The output is now different.

I can confirm that we'll copy the entire build context for named parameters instead of only the necessary files. Reproducer (some output filtered just for brevity):

$ dd if=/dev/urandom of=file1.txt bs=4096 count=1000
$ dd if=/dev/urandom of=file2.txt bs=4096 count=1000
$ ls -lh
-rw-r--r--@ 1 jsternberg  staff    72B Aug 14 16:39 Dockerfile
-rw-r--r--@ 1 jsternberg  staff   3.9M Aug 14 16:38 file1.txt
-rw-r--r--@ 1 jsternberg  staff   3.9M Aug 14 16:38 file2.txt
$ docker buildx build -f Dockerfile --build-context input=. --progress=plain .
#4 [context input] load from client
#4 transferring input: 8.19MB 0.1s done
#4 DONE 0.1s

When done with the normal context, it only copies 4.10MB.