grpc: failed to unmarshal the received message: string field contains invalid UTF-8

[xmapst]

After upgrading to buildx version v0.19.2, the following error is reported when building the image

[+] Building 3.2s (5/20)                                                                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                0.0s
 => => transferring dockerfile: 829B                                                                                                                                                                                                0.0s
 => [internal] load metadata for nexus.internal.com/base-image:linux-v2.1.18                                                                                                                                               0.1s
 => [internal] load .dockerignore                                                                                                                                                                                                   0.0s
 => => transferring context: 2B                                                                                                                                                                                                     0.0s
 => CANCELED [ 1/16] FROM nexus.internal.com/base-image:linux-v2.1.18@sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b                                                                              2.8s
 => => resolve nexus.internal.com/base-image:linux-v2.1.18@sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b                                                                                         0.1s
 => => sha256:084539c767645814983a002ca411bfd2a0d80db871d19459f171481fdacf5e3b 2.85kB / 2.85kB                                                                                                                                      0.0s
 => => sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc 76.10MB / 76.10MB                                                                                                                                    1.9s
 => => sha256:777c67696248a1e3d74fb3a1be418d3a393beb8bb362370faa659a282a336006 5.35MB / 5.35MB                                                                                                                                      0.4s
 => => sha256:d6bfbb7dc215c3a340544c4c87042210e647b140d51a6f210541801fce8274fd 7.78kB / 7.78kB                                                                                                                                      0.0s
 => => sha256:1ff5115dcffa2e271b232c9dde16c0f7f399773da441c440b497381167582a5f 59.59MB / 142.52MB                                                                                                                                   2.9s
 => => sha256:11006e65265e13aa0d3f6d56ce11dee5dca992cdcb8a9760431a109aaa79e375 15.73MB / 38.09MB                                                                                                                                    2.9s
 => => sha256:374999033e8d93a2d24cb2b13d62c8240959ab05c848b8fef1de101a600e7adb 20.97MB / 98.93MB                                                                                                                                    2.9s
 => => sha256:977e16a77d624fa14338325b88ad9f2aa3059564fa2daac0dce60f8dee15f0dc 2.64MB / 2.64MB                                                                                                                                      1.1s
 => => sha256:b257af7807da588a476e41081426372189b78526e7c371de2680606c3c02822f 315.64kB / 315.64kB                                                                                                                                  0.5s
 => => sha256:7e226f4e7f678e9028d1b76c8c0d56d6900058be170ce69baa63ba5567f8f53a 14.68MB / 901.58MB                                                                                                                                   2.9s
 => => sha256:08be1ee72a529c98177917e0c10bbbc2713519220fad9b3f810b691249a2faa9 193B / 193B                                                                                                                                          0.5s
 => => sha256:91e577162b472fd8eacc65cf3641be7d736a6d285185d878119e7ae53e4079cd 124B / 124B                                                                                                                                          0.5s
 => => sha256:8cc69cd3c0537c8a40164bff8c41e624703873f6e31d529d6316ada963d6fa2b 60.82MB / 172.92MB                                                                                                                                   2.9s
 => => sha256:a941cf7ebf1a67f1c76848a42b49d08eaec0fc9b3d62e4133f2e9d547f1d44f9 15.37MB / 15.37MB                                                                                                                                    2.4s
 => => extracting sha256:2d473b07cdd5f0912cd6f1a703352c82b512407db6b05b43f2553732b55df3bc                                                                                                                                           0.9s
 => ERROR [internal] load build context                                                                                                                                                                                             2.5s
 => => transferring context: 121.49MB                                                                                                                                                                                               2.4s
------
 > [internal] load build context:
------
ERROR: failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
3057857  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/session/filesync.recvDiffCopy
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/filesync/diffcopy.go:104
github.com/moby/buildkit/session/filesync.FSSync
        /root/build-deb/engine/vendor/github.com/moby/buildkit/session/filesync/filesync.go:236
github.com/moby/buildkit/source/local.(*localSourceHandler).snapshot
        /root/build-deb/engine/vendor/github.com/moby/buildkit/source/local/source.go:264
github.com/moby/buildkit/source/local.(*localSourceHandler).Snapshot
        /root/build-deb/engine/vendor/github.com/moby/buildkit/source/local/source.go:153
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).Exec
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/llbsolver/ops/source.go:108
github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/jobs.go:1100
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /root/build-deb/engine/vendor/github.com/moby/buildkit/util/flightcontrol/flightcontrol.go:122
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

3066149 v0.19.2 /usr/libexec/docker/cli-plugins/docker-buildx buildx build --debug --network host -t test:test -f ../Dockerfile .
google.golang.org/grpc.(*ClientConn).Invoke
        google.golang.org/grpc@v1.66.3/call.go:35
github.com/moby/buildkit/api/services/control.(*controlClient).Solve
        github.com/moby/buildkit@v0.18.0/api/services/control/control_grpc.pb.go:88
github.com/moby/buildkit/client.(*Client).solve.func2
        github.com/moby/buildkit@v0.18.0/client/solve.go:269
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78
runtime.goexit
        runtime/asm_amd64.s:1695

3057857  /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
github.com/moby/buildkit/solver.(*edge).execOp
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/edge.go:966
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction[...].func2
        /root/build-deb/engine/vendor/github.com/moby/buildkit/solver/internal/pipe/pipe.go:78
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1695

3066149 v0.19.2 /usr/libexec/docker/cli-plugins/docker-buildx buildx build --debug --network host -t test:test -f ../Dockerfile .
github.com/moby/buildkit/client.(*Client).solve.func2
        github.com/moby/buildkit@v0.18.0/client/solve.go:285
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78

version info:

# docker buildx version
github.com/docker/buildx v0.19.2 1fc5647
# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker                                 
 \_ default    \_ default       running   v0.17.3    linux/amd64 (+3), linux/386
# docker info
Client: Docker Engine - Community
 Version:    27.4.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 6
  Running: 6
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 27.4.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.2-0-g7cb3632
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.10.0-26-amd64
 Operating System: Debian GNU/Linux 11 (bullseye)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.88GiB
........

[tonistiigi]

After upgrading to buildx version v0.19.2,

Upgrading from what version?

Please post reproduction steps.

[nul0op]

Hello, we get the same problem since an upgrade this WE. 27.3.1 -> 27.4.1
end result for a build that worked fine before:

#0 building with "default" instance using docker driver
[...]
#6 [application  3/15] RUN [...redacted...]
#6 CACHED
#7 [application internal] load build context
#7 transferring context: 848.24kB done
#7 ERROR: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8
#8 [application  4/15] RUN apt-get update
#8 0.334 runc run failed: container process is already dead
#8 CANCELED
------
 > [application internal] load build context:
------
failed to solve: Internal: rpc error: code = Internal desc = grpc: failed to unmarshal the received message: string field contains invalid UTF-8

version that works with the same build:

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:11 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:11 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

version that FAILS with the same build:

Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:56 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:56 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

if we can help pinpoint the problem, please ask

[nul0op]

some tests i did:

• changed to base image in the pipeline => same error
• changed the layer just before the problem (by saving a file somewhere), just to try to avoid a layer cache issue
• on the same OS release, same dockerfile, i didn't have the issue. only difference there: i didn't build the same Dockerfile before the os upgrade (i mean with a lower docker version)
• split "docker compose --build" in a separate docker build call, with the --no-cache flag => same issue

so for me, it's only happening while docker compose build is running under the gitlab-ce runner hood, on a debian bookworm host with latest docker-ce packages (as of today)

again, if i can test something else, just ask

[xmapst]

@tonistiigi Any progress?

[tonistiigi]

My best guess is that you have something in the build context that makes the paths or symlink fields in https://github.com/tonistiigi/fsutil/blob/master/types/stat.proto#L10-L17 contain values that are not strings. Not sure what case would do that but likely that our validation that these are actual strings and not non-printable bytes is more strict now.

[xmapst]

My best guess is that you have something in the build context that makes the paths or symlink fields in https://github.com/tonistiigi/fsutil/blob/master/types/stat.proto#L10-L17 contain values that are not strings. Not sure what case would do that but likely that our validation that these are actual strings and not non-printable bytes is more strict now.

After checking, it was found that some file names were indeed not encoded in UTF-8, but in GB2312.

[nul0op]

i confirm it's related to a filename encoding.
As a workaround we have excluded some folders from the image (.dockerignore) and it builds fine now.

[tonistiigi]

Do you have a practical testcase for this, so we can look if there is a way to mitigate with workaround or improve error.