build vs. buildx cache evaluation behaves differently on AWS FSx for Lustre mount

[rcwbr]

Behavior

In a directory mounted from an AWS FSx for Lustre volume, docker buildx build will invalidate the build cache when a file has been replaced, even by one with the same contents, while docker build will preserve the build cache.

Desired behavior

If the file content remains the same, the build cache should be leveraged as it is a) using docker build and b) using docker buildx build within a traditional filesystem. This would enable, for example, leveraging the build cache across multiple identical clones of a repo in a CI environment which happen to use this kind of mount.

Environment

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 9
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1025-aws
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 30.36GiB
 Name: ip-10-255-0-85
 ID: XYYA:2K4L:PS6H:N62Y:T24V:WCR3:6FC2:7AUH:WMQ7:EIRJ:ZAYP:OOJU
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

mount | grep lustre:

10.255.0.107@tcp:/5mgfhbmv on /mnt/fsx/fs1 type lustre (rw,noatime,flock,lazystatfs,_netdev)

Steps to reproduce

I omitted steps to set up an EC2 instance with a FSx for Lustre volume, because I suspect the issue is somewhat broader. I'm happy to test out another environment (perhaps a simpler network-mounted volume?) to nail down the scope of this issue, but I'm hoping for advice on what would be most sensible to try.

Prepare the build context:

$ docker buildx use default
$ mkdir -p /mnt/fsx/fs1/test/cache_test/src
$ cd /mnt/fsx/fs1/test/cache_test
$ echo 'FROM busybox:latest' > Dockerfile
$ echo 'COPY src /src' >> Dockerfile
$ echo 'RUN sleep 3' >> Dockerfile
$ touch src/cache_test_file

^ assuming an FSx mount at /mnt/fsx/fs1

Demonstrate the build cache invalidation:

$ docker buildx build . # will build without cache
[+] Building 3.6s (8/8) FINISHED                                                                                                        
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 358B                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                  0.0s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 647B                                                                                                  0.0s
 => CACHED [1/3] FROM docker.io/library/busybox:latest                                                                             0.0s
 => [2/3] COPY src /src                                                                                                            0.1s
 => [3/3] RUN sleep 3                                                                                                              3.3s
 => exporting to image                                                                                                             0.1s
 => => exporting layers                                                                                                            0.1s
 => => writing image sha256:2676c4bf10cce5d5efce5bbc1f5ba26541e5b30c89366911e81483420e6dd5de                                       0.0s

$ docker buildx build . # will build with cache
[+] Building 0.1s (8/8) FINISHED                                                                                                        
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 306B                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                  0.0s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 643B                                                                                                  0.0s
 => [1/3] FROM docker.io/library/busybox:latest                                                                                    0.0s
 => CACHED [2/3] COPY src /src                                                                                                     0.0s
 => CACHED [3/3] RUN sleep 3                                                                                                       0.0s
 => exporting to image                                                                                                             0.0s
 => => exporting layers                                                                                                            0.0s
 => => writing image sha256:2676c4bf10cce5d5efce5bbc1f5ba26541e5b30c89366911e81483420e6dd5de                                       0.0s

$ rm -rf src/ && mkdir src && touch src/cache_test_file && docker buildx build . # will build without cache
[+] Building 3.5s (8/8) FINISHED                                                                                                        
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 306B                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                  0.0s
 => [internal] load build context                                                                                                  0.0s
 => => transferring context: 647B                                                                                                  0.0s
 => CACHED [1/3] FROM docker.io/library/busybox:latest                                                                             0.0s
 => [2/3] COPY src /src                                                                                                            0.1s
 => [3/3] RUN sleep 3                                                                                                              3.3s
 => exporting to image                                                                                                             0.1s
 => => exporting layers                                                                                                            0.1s
 => => writing image sha256:cd55e083e5fe225f2c8dc318535104a2c9564529e7b7f0a6df549eadb42b1b28                                       0.0s

Since the file contents in the final build are the same as the previous builds, the cache should be used rather than rebuilt.

Discrepancy vs. docker build and vs. on a traditional filesystem

In the context prepared above, docker build will not invalidate the build cache in the same way:

$ cd /mnt/fsx/fs1/test/cache_test
$ docker build . # will build without cache
$ rm -rf src/ && mkdir src && touch src/cache_test_file && docker build . # will build with cache

Similarly, in a directory not mounted from the FSx volume, docker build and docker buildx build will preserve the build cache:

$ cp -r ../cache_test $HOME
$ cd $HOME/cache_test
$ rm -rf src/ && mkdir src && touch src/cache_test_file && docker buildx build . # will build without cache
$ docker buildx build . # will build with cache
$ rm -rf src/ && mkdir src && touch src/cache_test_file && docker buildx build . # will build with cache

[tonistiigi]

 # find .
.
./Dockerfile
./src
./src/cache_test_file
 # cat Dockerfile
FROM busybox:latest
COPY src /src
RUN sleep 3
 #
 # docker buildx build .
[+] Building 0.6s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 83B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                     0.5s
 => [internal] load build context                                                                                                                                                     0.0s
 => => transferring context: 61B                                                                                                                                                      0.0s
 => [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                               0.0s
 => => resolve docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                               0.0s
 => CACHED [2/3] COPY src /src                                                                                                                                                        0.0s
 => CACHED [3/3] RUN sleep 3                                                                                                                                                          0.0s
 #
 # rm -rf src/ && mkdir src && touch src/cache_test_file
 #
 # docker buildx build .
[+] Building 0.6s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 83B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                     0.5s
 => [internal] load build context                                                                                                                                                     0.0s
 => => transferring context: 65B                                                                                                                                                      0.0s
 => [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                               0.0s
 => => resolve docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                               0.0s
 => CACHED [2/3] COPY src /src                                                                                                                                                        0.0s
 => CACHED [3/3] RUN sleep 3                                                                                                                                                          0.0s
 #
 #
 # rm -rf src/ && mkdir src && touch src/cache_test_file
 #
 #
 # docker buildx build .
[+] Building 0.6s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 83B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                     0.5s
 => [internal] load build context                                                                                                                                                     0.0s
 => => transferring context: 65B                                                                                                                                                      0.0s
 => [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                               0.0s
 => => resolve docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                               0.0s
 => CACHED [2/3] COPY src /src                                                                                                                                                        0.0s
 => CACHED [3/3] RUN sleep 3                                                                                                                                                          0.0s
 #

[tonistiigi]

I don't know what is special about that AWS mount if that is only case reproducing for you. You have to figure out what it does differently.

[rcwbr]

I tried reproducing via both an SFTP and an NFS mount, neither displayed the same behavior, so it seems related to Lustre specifically rather than network mounts generally. Could you recommend what file attributes I should inspect which are evaluated for the cache? Or point me to a place to start in the source to find which attributes are relevant?

---

EDIT: the following timestamp behavior with timestamps is not consistent

---

It seems that the timestamps of the file are somewhat relevant. If I override the access/modify times of the file, the cache is not discarded:

$ docker buildx build . 
[+] Building 10.5s (8/8) FINISHED                                                                                                                                                                                                                                                         
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 358B                                                                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                                                                                                                    6.7s
 => [internal] load build context                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 647B                                                                                                                                                                                                                                                    0.0s
 => [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                                                                                                                              0.2s
 => => resolve docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                                                                                                                              0.0s
 => => sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353 2.29kB / 2.29kB                                                                                                                                                                                       0.0s
 => => sha256:34efe68cca33507682b1673c851700ec66839ecf94d19b928176e20d20e02413 527B / 527B                                                                                                                                                                                           0.0s
 => => sha256:7138284460ffa3bb6ee087344f5b051468b3f8697e2d1427bac1a20c8d168b14 1.46kB / 1.46kB                                                                                                                                                                                       0.0s
 => => sha256:e685c5c858e36338a47c627763b50dfe6035b547f1f75f0d39753db71e319016 772.79kB / 772.79kB                                                                                                                                                                                   0.1s
 => => extracting sha256:e685c5c858e36338a47c627763b50dfe6035b547f1f75f0d39753db71e319016                                                                                                                                                                                            0.0s
 => [2/3] COPY src /src                                                                                                                                                                                                                                                              0.1s
 => [3/3] RUN sleep 3                                                                                                                                                                                                                                                                3.4s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:47888894d6171d2e5d81c0c7f263e92880cbb025728373f314a1cb97e9cc55db                                                                                                                                                                                         0.0s
$ rm -rf src/ && mkdir src && touch src/cache_test_file
$ touch -d "3 am today" src/cache_test_file
$ docker buildx build . 
[+] Building 2.2s (8/8) FINISHED                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 306B                                                                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                                                                                                                    2.2s
 => [internal] load build context                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 643B                                                                                                                                                                                                                                                    0.0s
 => [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                                                                                                                              0.0s
 => CACHED [2/3] COPY src /src                                                                                                                                                                                                                                                       0.0s
 => CACHED [3/3] RUN sleep 3                                                                                                                                                                                                                                                         0.0s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:47888894d6171d2e5d81c0c7f263e92880cbb025728373f314a1cb97e9cc55db                                                                                                                                                                                         0.0s

However, this only works if the immediately previous build was built against the same timestamp. If a build happens in between without the same timestamp, the cache is still discarded:

$ rm -rf src/ && mkdir src && touch src/cache_test_file
$ docker buildx build . 
[+] Building 5.7s (8/8) FINISHED                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 306B                                                                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                                                                                                                    2.2s
 => [internal] load build context                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 647B                                                                                                                                                                                                                                                    0.0s
 => CACHED [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                                                                                                                       0.0s
 => [2/3] COPY src /src                                                                                                                                                                                                                                                              0.0s
 => [3/3] RUN sleep 3                                                                                                                                                                                                                                                                3.3s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:93ae8256216ac7728e396a9f67cfdbf17cd1ff203341637f9e6667668b8ade6c                                                                                                                                                                                         0.0s
$ rm -rf src/ && mkdir src && touch src/cache_test_file
$ touch -d "3 am today" src/cache_test_file
$ docker buildx build . 
[+] Building 5.7s (8/8) FINISHED                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 306B                                                                                                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/busybox:latest                                                                                                                                                                                                                    2.2s
 => [internal] load build context                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 647B                                                                                                                                                                                                                                                    0.0s
 => CACHED [1/3] FROM docker.io/library/busybox:latest@sha256:e7157b6d7ebbe2cce5eaa8cfe8aa4fa82d173999b9f90a9ec42e57323546c353                                                                                                                                                       0.0s
 => [2/3] COPY src /src                                                                                                                                                                                                                                                              0.1s
 => [3/3] RUN sleep 3                                                                                                                                                                                                                                                                3.3s
 => exporting to image                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:500046478acdd8a76d55378b99c8f573d8790d919947af008e3186eb852faeff                                                                                                                                                                                         0.0s

[rcwbr]

Actually, it appears that the timestamp behavior above is inconsistent -- my best guess would be due to very small time differences somewhere?

Could you recommend what file attributes I should inspect which are evaluated for the cache? Or point me to a place to start in the source to find which attributes are relevant?

@tonistiigi even just a pointer for where to start in source code would be a huge help here.