Concurrent build failed because of error: traces export: Canceled: grpc: the client connection is closing

[iapershin]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

After upgrading buildx version from v0.13.0-rc2 to v0.19.2 there is an error when trying to build images in parallel:

traces export: Canceled: grpc: the client connection is closing

Builds runs programmatically from go code in a concurrent manner, code snippet attached to configuration

build1
#0 building with "default" instance using docker driver

#1 [internal] load build definition from r.Dockerfile
#1 transferring dockerfile: 75B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#0 building with "default" instance using docker driver

#1 [internal] load build definition from o.Dockerfile
#1 transferring dockerfile: 76B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#2 ...

#3 [auth] library/ubuntu:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#2 DONE 0.9s
#2 DONE 0.9s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM docker.io/library/ubuntu:22.04@sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
#4 DONE 0.0s

#5 [2/2] RUN echo 'build1'
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:57fac05fd75d78be369d191471f8f597760c0c69967f318bb7f7015afa5baf6b done
#6 DONE 0.0s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [1/2] FROM docker.io/library/ubuntu:22.04@sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
#5 DONE 0.0s

#6 [2/2] RUN echo 'build2'
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:04b150f29dddae6d4b8d18aca4409c98c30bab32fee641b8c73a04dec92bd4a7 done
#7 DONE 0.0s
2025/01/14 19:30:14 traces export: Canceled: grpc: the client connection is closing

Expected behaviour

Both builds are successful

Actual behaviour

Build fails

Buildx version

github.com/docker/buildx v0.19.3

Docker info

Builders list

n/a

Configuration

FROM ubuntu:22.04
RUN echo 'build1'
RUN sleep 3600

FROM ubuntu:22.04
RUN echo 'build2'

func main() {
	ctx := context.Background()
	cli, err := docker.NewDockerCli(docker.DefaultCliOptions(ctx))
	if err != nil {
		return
	}

	cmd1, cmd2 := cmd1(cli), cmd2(cli)

	var wg sync.WaitGroup

	wg.Add(2)

	go func() {
		fmt.Println("build1")
		defer wg.Done()
		cmd1.Execute()
	}()

	go func() {
		defer wg.Done()
		cmd2.Execute()
	}()

	wg.Wait()
}

func cmd1(dockerCli command.Cli) *cobra.Command {
	cmd := docker.NewBuildxCommand(dockerCli)
	args := []string{
		"build",
		"--file", "o.Dockerfile",
		"--progress", "plain",
		".",
	}
	cmd.SetArgs(args)
	return cmd
}

func cmd2(dockerCli command.Cli) *cobra.Command {
	cmd := docker.NewBuildxCommand(dockerCli)
	args := []string{
		"build",
		"--file", "r.Dockerfile",
		"--progress", "plain",
		".",
	}
	cmd.SetArgs(args)
	return cmd
}

Build logs

$ go run main.go
build1
#0 building with "default" instance using docker driver

#1 [internal] load build definition from r.Dockerfile
#1 transferring dockerfile: 75B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#0 building with "default" instance using docker driver

#1 [internal] load build definition from o.Dockerfile
#1 transferring dockerfile: 76B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#2 ...

#3 [auth] library/ubuntu:pull token for registry-1.docker.io
#3 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#2 DONE 0.9s
#2 DONE 0.9s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM docker.io/library/ubuntu:22.04@sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
#4 DONE 0.0s

#5 [2/2] RUN echo 'build1'
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:57fac05fd75d78be369d191471f8f597760c0c69967f318bb7f7015afa5baf6b done
#6 DONE 0.0s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [1/2] FROM docker.io/library/ubuntu:22.04@sha256:0e5e4a57c2499249aafc3b40fcd541e9a456aab7296681a3994d631587203f97
#5 DONE 0.0s

#6 [2/2] RUN echo 'build2'
#6 CACHED

#7 exporting to image
#7 exporting layers done
#7 writing image sha256:04b150f29dddae6d4b8d18aca4409c98c30bab32fee641b8c73a04dec92bd4a7 done
#7 DONE 0.0s
2025/01/14 19:30:14 traces export: Canceled: grpc: the client connection is closing

Additional info

No response

[tonistiigi]

What's this library

cli, err := docker.NewDockerCli(docker.DefaultCliOptions(ctx))
	if err != nil {
		return
	}

cmd := docker.NewBuildxCommand(dockerCli)

Some similarities with https://pkg.go.dev/github.com/werf/werf/v2 but that library does not call buildx binary internally afaics. Post runnable reproducer, preferably by invoking buildx binary, as we can't debug 3rd party libraries.

Iiuc what this werf/werf/v2 library is doing then there is no such support of invoking two cobra.Command definitions in parallel. These definitions are for defining the behavior of buildx cli only.

[iapershin]

Apologize for the confusion. Here is the code that can be run without unnecessary dependencies. I am the maintainer of werf project and we allow our users to run parallel builds. Before the update, this worked without any issues.

package main

import (
	"fmt"
	"os"
	"path/filepath"
	"sync"

	"github.com/docker/buildx/commands"
	_ "github.com/docker/buildx/driver/docker"
	"github.com/docker/cli/cli/command"
	cliconfig "github.com/docker/cli/cli/config"
	"github.com/docker/cli/cli/flags"
	"github.com/docker/go-connections/tlsconfig"
	"github.com/spf13/cobra"
)

func main() {
	cli, err := newDockerCli(defaultCliOptions())
	if err != nil {
		return
	}

	cmd1, cmd2 := cmd1(cli), cmd2(cli)

	var wg sync.WaitGroup

	wg.Add(2)

	go func() {
		fmt.Println("build1")
		defer wg.Done()
		cmd1.Execute()
	}()

	go func() {
		defer wg.Done()
		cmd2.Execute()
	}()

	wg.Wait()
}

func cmd1(dockerCli command.Cli) *cobra.Command {
	cmd := newBuildxCommand(dockerCli)
	args := []string{
		"build",
		"--file", "o.Dockerfile",
		"--progress", "plain",
		".",
	}
	cmd.SetArgs(args)
	return cmd
}

func cmd2(dockerCli command.Cli) *cobra.Command {
	cmd := newBuildxCommand(dockerCli)
	args := []string{
		"build",
		"--file", "r.Dockerfile",
		"--progress", "plain",
		".",
	}
	cmd.SetArgs(args)
	return cmd
}

func newBuildxCommand(dockerCli command.Cli) *cobra.Command {
	cmd := commands.NewRootCmd("", false, dockerCli)
	return cmd
}

func defaultCliOptions() []command.CLIOption {
	return []command.CLIOption{
		command.WithInputStream(os.Stdin),
		command.WithContentTrust(false),
	}
}

func newDockerCli(opts []command.CLIOption) (command.Cli, error) {
	newCli, err := command.NewDockerCli(opts...)
	if err != nil {
		return nil, err
	}

	clientOpts := flags.NewClientOptions()

	dockerCertPath := os.Getenv("DOCKER_CERT_PATH")
	if dockerCertPath == "" {
		dockerCertPath = cliconfig.Dir()
	}

	clientOpts.TLS = os.Getenv("DOCKER_TLS") != ""
	clientOpts.TLSVerify = os.Getenv("DOCKER_TLS_VERIFY") != ""

	if clientOpts.TLSVerify {
		clientOpts.TLSOptions = &tlsconfig.Options{
			CAFile:   filepath.Join(dockerCertPath, flags.DefaultCaFile),
			CertFile: filepath.Join(dockerCertPath, flags.DefaultCertFile),
			KeyFile:  filepath.Join(dockerCertPath, flags.DefaultKeyFile),
		}
	}

	clientOpts.LogLevel = "fatal"

	if err := newCli.Initialize(clientOpts); err != nil {
		return nil, err
	}
	return newCli, nil
}

[tonistiigi]

I haven't debugged this in code but having error in exporting traces here seems to make sense as tracing can only happen once for the cli invocation. This is why only one cli command can run at a time in one process. We trace the invocation of the cli.

If you do use buildx as a library, you should use the build and bake packages, not the cli commands.