Skip to content

long-running mutistage build gets canceled around 1min if stages are from mcr.microsoft.com #10106

Open
#10107
Open
long-running mutistage build gets canceled around 1min if stages are from mcr.microsoft.com#10106
#10107

Description

@foriequal0
foriequal0
opened on Jun 9, 2026

TL;DR:

  1. skaffold build (direct docker build succeed)
  2. docker backend (regardless of buildkit or not)
  3. multistage Dockerfile
  4. all stages are from mcr.microsoft.com (other registries have no problem)
  5. the first stage takes more than 60s
  6. then it gets canceled around 1 min or at the start of the second stage due to the internal EPIPE error that causes SIGPIPE.

detailed explaination, reproduction, eliminations:

We've been having this problem since a few days ago (first occurrence: 2026-06-05, widespread: 2026-06-08)

minimal reproduce

If we have the following Dockerfile:

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
RUN mkdir /src/ && touch /src/asdf && sleep 90

FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS aspnet
RUN mkdir /app/
COPY --from=sdk /src/asdf /app/asdf

and the following skaffold.yaml:

apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: test
build:
  artifacts:
    - image: test
  local:
    push: false

and run the following command:

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
skaffold build

The build gets canceled in the middle of the build (after around 1 min)

Generating tags...
 - test -> test:latest
Some taggers failed. Rerun with -vdebug for errors.
Checking cache...
 - test: Not found. Building
Starting build...
Building [test]...
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
 ---> c07906393326
Step 2/5 : RUN mkdir /src/ && touch /src/asdf && sleep 90
 ---> Running in 25dce5cfedc2
Build [test] was canceled

This is the last few lines of logs with -v trace

---8<--- snip

DEBU[0000] Could not import artifact from Docker, building instead (import of missing images disabled)  subtask=-1 task=Build
 - test: Not found. Building
INFO[0000] Cache check completed in 306.481713ms         subtask=-1 task=Build
Starting build...
Building [test]...
DEBU[0000] Running docker build: context: ., dockerfile: Dockerfile  subtask=test task=Build
DEBU[0000] Skipping credential configuration because docker-credential-gcloud is not on PATH.  subtask=-1 task=DevLoop
INFO[0000] Creating tar file from 1 file(s)              subtask=test task=Build
INFO[0000] Creating tar file completed in 112.382µs      subtask=test task=Build
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
 ---> c07906393326
Step 2/5 : RUN mkdir /src/ && touch /src/asdf && sleep 90
 ---> Running in 9d744bb3a429
Build [test] was canceled

DEBU[0066] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0066] failed to create firelog exporter due to error: open assets/firelog_generated/key.txt: file does not exist  subtask=-1 task=DevLoop
DEBU[0066] did not create any log exporter               subtask=-1 task=DevLoop

---8<--- snip

it happens regardless of BuildKit.

this is the last lines of logs with following skaffold.yaml to enable buildkit

apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: test
build:
  artifacts:
    - image: test
  local:
    push: false
    useBuildkit: true
---8<--- snip

DEBU[0000] Found dependencies for dockerfile: []         subtask=-1 task=DevLoop
DEBU[0000] Could not import artifact from Docker, building instead (import of missing images disabled)  subtask=-1 task=Build
 - test: Not found. Building
INFO[0000] Cache check completed in 306.195482ms         subtask=-1 task=Build
Starting build...
Building [test]...
DEBU[0000] Running command: [docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest --load]  subtask=test task=Build
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 237B done
#1 DONE 0.0s

#2 [internal] load metadata for mcr.microsoft.com/dotnet/sdk:10.0
#2 DONE 0.0s

#3 [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:10.0
#3 DONE 0.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [aspnet 1/3] FROM mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711
#5 resolve mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711 0.0s done
#5 DONE 0.2s

#6 [sdk 1/2] FROM mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6
#6 resolve mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6 0.0s done
#6 DONE 0.3s

#7 [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90
#7 ...

#8 [aspnet 2/3] RUN mkdir /app/
#8 DONE 0.2s

#7 [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90
#7 CANCELED
ERROR: failed to build: failed to solve: Canceled: context canceled
Build [test] was canceled

DEBU[0066] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0066] failed to create firelog exporter due to error: open assets/firelog_generated/key.txt: file does not exist  subtask=-1 task=DevLoop
DEBU[0066] did not create any log exporter               subtask=-1 task=DevLoop

---8<--- snip

It happens only with skaffold + docker, not with just docker, nor skaffold + podman

If I run docker command in the log directly, it finishes the build successfully

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest --load

[+] Building 91.1s (10/10) FINISHED                                                                                                                                                                                     docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                              0.0s
 => => transferring dockerfile: 237B                                                                                                                                                                                              0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/aspnet:10.0                                                                                                                                                             0.1s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:10.0                                                                                                                                                                0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                                                                                   0.0s
 => [sdk 1/2] FROM mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6                                                                                                      0.2s
 => => resolve mcr.microsoft.com/dotnet/sdk:10.0@sha256:c0790639332692a0d56cdd81ed581cfd24d040d9839764c138994866df89a3b6                                                                                                          0.0s
 => [aspnet 1/3] FROM mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711                                                                                                0.1s
 => => resolve mcr.microsoft.com/dotnet/aspnet:10.0@sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711                                                                                                       0.0s
 => [aspnet 2/3] RUN mkdir /app/                                                                                                                                                                                                  0.2s
 => [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90                                                                                                                                                                     90.3s
 => [aspnet 3/3] COPY --from=sdk /src/asdf /app/asdf                                                                                                                                                                              0.0s
 => exporting to image                                                                                                                                                                                                            0.3s
 => => exporting layers                                                                                                                                                                                                           0.1s
 => => exporting manifest sha256:c7128685369cee3f5533b74b0faf3e10a0e26393c36ad1bb5efa0235c23d64d7                                                                                                                                 0.0s
 => => exporting config sha256:b7662798278842ddbfcea6bd09a635ef9fda770e3f9511e9bbe9d7a70cae1915                                                                                                                                   0.0s
 => => exporting attestation manifest sha256:859bebcb0743d1aa4693bacc22d2882627db4bdb5bb4281389e71e9f2b6867d7                                                                                                                     0.0s
 => => exporting manifest list sha256:3c93970a8c7e1b725725aaf7b5c272446154c681bf0c258d0fab3b3920ed3acf                                                                                                                            0.0s
 => => naming to docker.io/library/test:latest                                                                                                                                                                                    0.0s
 => => unpacking to docker.io/library/test:latest                                                                                                                                                                                 0.0s

it also build successfully without the buildkit:

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
DOCKER_BUILDKIT=0 docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  3.072kB
Step 1/5 : FROM mcr.microsoft.com/dotnet/sdk:10.0 AS sdk
 ---> c07906393326
Step 2/5 : RUN mkdir /src/ && touch /src/asdf && sleep 90
 ---> Running in 47919a2bd636
 ---> Removed intermediate container 47919a2bd636
 ---> ed6003243083
Step 3/5 : FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS aspnet
10.0: Pulling from dotnet/aspnet
Digest: sha256:8c0b6857eab7b2aa57884c839bf4678414606bd7d17370f18a842ac5cf414711
Status: Downloaded newer image for mcr.microsoft.com/dotnet/aspnet:10.0
 ---> 8c0b6857eab7
Step 4/5 : RUN mkdir /app/
 ---> Running in b3c53e899f05
 ---> Removed intermediate container b3c53e899f05
 ---> 0a2bc21fcfeb
Step 5/5 : COPY --from=sdk /src/asdf /app/asdf
 ---> 93dd0fe9c9d1
Successfully built 93dd0fe9c9d1
Successfully tagged test:latest

Also it builds successfully with podman backend.
podman also fails.

It happens only with mcr.microsoft.com

If we change the Dockerfile to images from another registry, Skaffold has no problem with it.

FROM redis AS sdk
RUN mkdir /src/ && touch /src/asdf && sleep 90
FROM busybox AS aspnet
RUN mkdir /app/
COPY --from=sdk /src/asdf /app/asdf

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
skaffold build
---8<--- snip

DEBU[0002] Running command: [docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest --load]  subtask=test task=Build
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 180B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/redis:latest
#2 DONE 1.1s

#3 [internal] load metadata for docker.io/library/busybox:latest
#3 DONE 1.1s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [aspnet 1/3] FROM docker.io/library/busybox:latest@sha256:fd8d9aa63ba2f0982b5304e1ee8d3b90a210bc1ffb5314d980eb6962f1a9715d
#5 resolve docker.io/library/busybox:latest@sha256:fd8d9aa63ba2f0982b5304e1ee8d3b90a210bc1ffb5314d980eb6962f1a9715d 0.0s done
#5 DONE 0.1s

#6 [sdk 1/2] FROM docker.io/library/redis:latest@sha256:aa049e689e141a4358ad1d4562dc49c88a89fbab711fd8fcc33f684c80b26301
#6 resolve docker.io/library/redis:latest@sha256:aa049e689e141a4358ad1d4562dc49c88a89fbab711fd8fcc33f684c80b26301 0.0s done
#6 DONE 0.2s

#7 [aspnet 2/3] RUN mkdir /app/
#7 DONE 0.2s

#8 [sdk 2/2] RUN mkdir /src/ && touch /src/asdf && sleep 90
#8 DONE 90.2s

#9 [aspnet 3/3] COPY --from=sdk /src/asdf /app/asdf
#9 DONE 0.0s

#10 exporting to image
#10 exporting layers 0.1s done
#10 exporting manifest sha256:32e100d0f1739ade7d4a5cae261fa1d95789af17ef5d7307c13fccfd7e2bd6ac 0.0s done
#10 exporting config sha256:d40e11d6da493d7b9f22626d04ce044f1db3d28d3368e3a7d83a172e8dc51eed 0.0s done
#10 exporting attestation manifest sha256:1eb9a12fd3d3224f2c1d3186e361adf6dd64f2d99804d83539128be317afe89d 0.0s done
#10 exporting manifest list sha256:eea4c7323409f42196b751585776c8984f1eab41b67003369f7156c9a494cd77 0.0s done
#10 naming to docker.io/library/test:latest
#10 naming to docker.io/library/test:latest done
#10 unpacking to docker.io/library/test:latest 0.0s done
#10 DONE 0.3s
Build [test] succeeded
INFO[0095] Build completed in 1 minute 32.166 seconds    subtask=-1 task=Build

DEBU[0095] exporting metrics                             subtask=-1 task=DevLoop
DEBU[0095] failed to create firelog exporter due to error: open assets/firelog_generated/key.txt: file does not exist  subtask=-1 task=DevLoop
DEBU[0095] did not create any log exporter               subtask=-1 task=DevLoop

---8<--- snip

findings

The docker command is killed by skaffold. skaffold kills the docker command when it got SIGPIPE signal.
The cancel is from this context, and it has the reason broken pipe signal received

skaffold/cmd/skaffold/app/skaffold.go

Line 36 in f9beeb7

ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM, syscall.SIGINT, syscall.SIGPIPE)

strace shows that it got SIGPIPE with microsoft's IP address at the start of second stage.

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
strace -f -yy --trace=read,write,rt_sigprocmask,rt_sigreturn skaffold build
---8<--- snip

[pid 223504] read(8<TCP:[192.168.0.29:35698->150.171.69.10:443]>, 0xc00001cf00, 9472) = -1 ECONNRESET (Connection reset by peer)
[pid 223504] write(8<TCP:[192.168.0.29:35698->150.171.69.10:443]>, "\27\3\3\0\23+,\334h|\224!]\277\16$4*\243:\2015v\351", 24) = -1 EPIPE (Broken pipe)
[pid 223504] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=223450, si_uid=1000} ---
[pid 223504] rt_sigreturn({mask=[]})    = -1 EPIPE (Broken pipe)

---8<--- snip

[pid 223547] write(2<pipe:[3130117]>, "#7 CANCELED\n", 12 <unfinished ...>
[pid 223544] read(10<UNIX-STREAM:[3150140->3132335]> <unfinished ...>
[pid 223547] <... write resumed>)       = 12
[pid 223544] <... read resumed>, 0xc000802000, 32768) = -1 EAGAIN (Resource temporarily unavailable)
[pid 223456] read(10<pipe:[3130117]>, "#7 CANCELED\n", 32768) = 12
[pid 223456] write(1</dev/pts/0<char 136:0>>, "#7 CANCELED\n", 12#7 CANCELED
) = 12

---8<--- snip

I couldn't find SIGPIPE when I invoked the Docker CLI directly

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
DOCKER_BUILDKIT=0 strace -f -yy --trace=read,write,rt_sigprocmask,rt_sigreturn docker build . --file /home/foriequal0/dockertest/Dockerfile -t test:latest

environment:

multiple version of skaffold:

on our CI environment ->
OS: ubuntu 24.04
skaffold: v2.16.1, v2.21.0, v2.22.0, v2.22.1
docker:

Client:
 Version:           29.3.0
 API version:       1.54
 Go version:        go1.25.7
 Git commit:        5927d80
 Built:             Thu Mar  5 14:24:57 2026
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          29.3.0-namespace
  API version:      1.54 (minimum version 1.40)
  Go version:       go1.25.7
  Git commit:       HEAD
  Built:            Wed Apr 22 16:26:32 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          2.2.1+unknown
  GitCommit:
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

one of my team member's workstation ->
OS: nix OS
skaffold: ?
docker:

Client:
 Version:           28.5.2
 API version:       1.51
 Go version:        go1.25.9
 Git commit:        v28.5.2
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          28.5.2
  API version:      1.51 (minimum version 1.24)
  Go version:       go1.25.9
  Git commit:       v28.5.2
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.1
  GitCommit:        refs/tags/v2.2.1
 runc:
  Version:          1.3.3
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

one of my other team member's workstation ->
OS: WSL2
skaffold: ?
docker:

Client:
 Version:           29.5.2
 API version:       1.54
 Go version:        go1.26.3
 Git commit:        79eb04c
 Built:             Wed May 20 14:37:09 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop 4.76.0 (228118)
 Engine:
  Version:          29.5.2
  API version:      1.54 (minimum version 1.40)
  Go version:       go1.26.3
  Git commit:       568f755
  Built:            Wed May 20 14:38:09 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.4
  GitCommit:        193637f7ee8ae5f5aa5248f49e7baa3e6164966e
 runc:
  Version:          1.3.5
  GitCommit:        v1.3.5-0-g488fc13e
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

my workstation ->
OS: ubuntu 26.04
skaffold: v2.14.0-14-ga65fe4bb5
docker:

Client:
 Version:           29.1.3
 API version:       1.52
 Go version:        go1.24.13
 Git commit:        29.1.3-0ubuntu4.1
 Built:             Wed Apr 29 16:40:20 2026
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          29.1.3
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.24.13
  Git commit:       29.1.3-0ubuntu4.1
  Built:            Wed Apr 29 16:40:20 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          2.2.2
  GitCommit:
 runc:
  Version:          1.4.0-0ubuntu1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

workaround:

pulling all images before the skaffold build prevents this problem

docker image prune -f --all && docker buildx prune -f # make sure no cached layers left
docker image pull mcr.microsoft.com/dotnet/sdk:10.0
docker image pull mcr.microsoft.com/dotnet/aspnet:10.0
skaffold build

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions