Skip to content

Commit d607479

Browse files
huydhnhuydhn
authored
Add docker-build-remote-buildkit composite action (#8487)
Adds a reusable composite action that builds and pushes a Docker image from an OSDC/ARC runner via the in-cluster remote BuildKit service, wrapping the `build_with_remote_buildkit.sh` pattern from pytorch/pytorch. OSDC runners have no host docker daemon, so every image build has to go through remote BuildKit, and each caller currently wires it up by hand. The obvious wiring is also the broken one: `docker/setup-buildx-action` (and `buildx create --bootstrap`) runs `buildx inspect --bootstrap`, whose ~20s connect timeout expires on a cold or bursting builder pool. `_docker-build.yml` in this repo does exactly that today and has no retry. The action uses a bare `buildx create` and retries connection-phase failures only, never a genuine build error, and blocks until the push completes so downstream jobs can order on it with `needs:`.
1 parent 95af54d commit d607479

7 files changed

Lines changed: 408 additions & 19 deletions

File tree

‎.github/actionlint.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,7 @@ self-hosted-runner:
1212
# Below are OSDC ARC runner
1313
- mt-l-x86iamx-8-16
1414
- mt-l-arm64g2-6-32
15+
- mt-l-x86iavx512-8-64
16+
- mt-l-arm64g4-16-62
1517
- mt-rel-l-x86iavx512-8-64
1618
- mt-rel-l-arm64g4-16-62

‎.github/actions/docker-build-remote-buildkit/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# docker-build-remote-buildkit
2+
3+
Build and push a Docker image from an OSDC/ARC runner.
4+
5+
OSDC runners are ephemeral Kubernetes pods with **no host docker daemon**, so `docker build` and
6+
`docker/build-push-action`'s default driver do not work. OSDC instead runs a per-arch `buildkitd`
7+
service in every cluster. This action registers it as a remote `buildx` builder and runs the build
8+
against it.
9+
10+
```yaml
11+
- uses: pytorch/test-infra/.github/actions/docker-build-remote-buildkit@main
12+
with:
13+
context: ./docker
14+
file: ./docker/Dockerfile
15+
tags: ghcr.io/pytorch/my-image:${{ github.sha }}
16+
build-args: |
17+
BASE_IMAGE=quay.io/pypa/manylinux_2_28_x86_64
18+
```
19+
20+
The step blocks until the image has been pushed, so a job that builds an image and a job that
21+
consumes it can be ordered with a plain `needs:` — no polling for the tag to appear.
22+
23+
## Wrapping your own build script
24+
25+
Builds driven by a script or make target — as pytorch/pytorch's image builds are — pass the whole
26+
command instead of the buildx inputs:
27+
28+
```yaml
29+
- uses: pytorch/test-infra/.github/actions/docker-build-remote-buildkit@main
30+
env:
31+
REMOTE_BUILDKIT: 1 # if your script keys off it, as .ci/docker/build.sh does
32+
with:
33+
command: .ci/docker/manywheel/build.sh manylinux2_28-builder:cpu -t my-tag
34+
```
35+
36+
The command owns its tags and its `--push`; the action only registers the builder and applies the
37+
retry. `--load` still cannot work, since there is no local daemon.
38+
39+
## Why not `docker/setup-buildx-action`
40+
41+
Both `docker/setup-buildx-action` and `docker buildx create --bootstrap` run
42+
`buildx inspect --bootstrap`, whose ~20s gRPC connect timeout expires on a cold or bursting builder
43+
pool before the autoscaler can add a builder. This action uses a bare `docker buildx create` and
44+
retries only *connection-phase* failures (`waiting for connection`, `failed to dial/list workers`,
45+
`context deadline exceeded`, `server preface`, …). Once BuildKit has started the build, a failure is
46+
a real build error and is never retried.
47+
48+
Each builder pod serves one build at a time (HAProxy `maxconn 1`), so during a burst your build
49+
queues behind the pool scaling up. The retry loop defaults to `connect-attempts: 480` at
50+
`connect-delay: 15`, i.e. it will wait roughly two hours for a builder. Bound how long a job is
51+
actually willing to wait with its own `timeout-minutes` rather than by lowering the attempts.
52+
53+
## Notes
54+
55+
- **Push, don't load.** There is no local daemon, so `--load` has nothing to load into. `push: false`
56+
builds and discards the image.
57+
- **Remote BuildKit is per-arch**, so the build always targets the runner's own architecture. To
58+
build for another platform, run the job on a runner of that architecture.
59+
- BuildKit allows roughly 120 minutes per build — keep the job's `timeout-minutes` above that for
60+
large images.
61+
62+
Adapted from `pytorch/pytorch:.github/scripts/build_with_remote_buildkit.sh`.

‎.github/actions/docker-build-remote-buildkit/action.yml‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
name: Build a Docker image on remote BuildKit
2+
3+
description: >
4+
Build (and push) a Docker image from an OSDC/ARC runner, which has no host docker daemon.
5+
Registers the in-cluster remote BuildKit builder for the runner's architecture and retries
6+
connection-phase failures while the autoscaled builder pool is cold, so callers get a build
7+
that either succeeds or fails for a real reason. The step blocks until the image is pushed,
8+
so downstream jobs can depend on it with `needs:`.
9+
10+
inputs:
11+
command:
12+
description: >
13+
A build command to run on the remote builder instead of assembling a `docker buildx build`
14+
invocation — a repo build script or make target, for example. When set, the buildx inputs
15+
below (tags, context, file, build-args, labels, target, push, cache-from, cache-to) are
16+
ignored and the `image` output is not set. The command is responsible for its own tags and
17+
for `--push`; note there is no local daemon, so `--load` cannot work.
18+
default: ''
19+
tags:
20+
description: >
21+
Newline- or comma-separated list of image tags, e.g. ghcr.io/pytorch/my-image:abc123.
22+
Required unless `command` is set.
23+
default: ''
24+
context:
25+
description: Build context path.
26+
default: .
27+
file:
28+
description: Path to the Dockerfile. Defaults to <context>/Dockerfile.
29+
default: ''
30+
build-args:
31+
description: Newline-separated KEY=VALUE build arguments.
32+
default: ''
33+
labels:
34+
description: Newline-separated KEY=VALUE image labels, e.g. the output of docker/metadata-action.
35+
default: ''
36+
target:
37+
description: Dockerfile stage to build.
38+
default: ''
39+
push:
40+
description: >
41+
Push the image to the registry. There is no local docker daemon on an OSDC runner, so
42+
`--load` has nothing to load into; a build with push=false is only useful as a syntax check.
43+
default: 'true'
44+
cache-from:
45+
description: Newline-separated buildx --cache-from entries.
46+
default: ''
47+
cache-to:
48+
description: Newline-separated buildx --cache-to entries.
49+
default: ''
50+
connect-attempts:
51+
description: >
52+
How many times to retry a connection-phase failure before giving up. The default of 480 at
53+
the default 15s delay waits roughly two hours for a builder, which covers a fully cold or
54+
saturated pool (one build per builder pod, so a burst queues). Bound the real ceiling with
55+
the job's `timeout-minutes`, not by lowering this.
56+
default: '480'
57+
connect-delay:
58+
description: Seconds to wait between connection-phase retries.
59+
default: '15'
60+
61+
outputs:
62+
image:
63+
description: >
64+
The first tag passed in, for convenience when wiring up downstream jobs. Empty in command
65+
mode.
66+
value: ${{ steps.build.outputs.image }}
67+
68+
runs:
69+
using: composite
70+
steps:
71+
- name: Build and push with remote BuildKit
72+
id: build
73+
shell: bash
74+
env:
75+
COMMAND: ${{ inputs.command }}
76+
TAGS: ${{ inputs.tags }}
77+
CONTEXT: ${{ inputs.context }}
78+
FILE: ${{ inputs.file }}
79+
BUILD_ARGS: ${{ inputs.build-args }}
80+
LABELS: ${{ inputs.labels }}
81+
TARGET: ${{ inputs.target }}
82+
PUSH: ${{ inputs.push }}
83+
CACHE_FROM: ${{ inputs.cache-from }}
84+
CACHE_TO: ${{ inputs.cache-to }}
85+
REMOTE_BUILDKIT_CONNECT_ATTEMPTS: ${{ inputs.connect-attempts }}
86+
REMOTE_BUILDKIT_CONNECT_DELAY: ${{ inputs.connect-delay }}
87+
run: |
88+
set -euo pipefail
89+
90+
# Command mode: hand the whole thing to the helper, which registers the
91+
# builder and retries only connection-phase failures.
92+
if [[ -n "${COMMAND}" ]]; then
93+
exec bash "${GITHUB_ACTION_PATH}/build_with_remote_buildkit.sh" bash -c "${COMMAND}"
94+
fi
95+
96+
args=()
97+
98+
# Tags may be newline- or comma-separated.
99+
first_tag=""
100+
while IFS= read -r tag; do
101+
tag="$(echo "${tag}" | tr -d '[:space:]')"
102+
[[ -z "${tag}" ]] && continue
103+
[[ -z "${first_tag}" ]] && first_tag="${tag}"
104+
args+=(--tag "${tag}")
105+
done < <(echo "${TAGS}" | tr ',' '\n')
106+
if [[ -z "${first_tag}" ]]; then
107+
echo "::error::no image tag was provided" >&2
108+
exit 1
109+
fi
110+
echo "image=${first_tag}" >> "${GITHUB_OUTPUT}"
111+
112+
# Remote BuildKit is per-arch, so the only buildable platform is the
113+
# runner's own.
114+
case "$(uname -m)" in
115+
aarch64|arm64) args+=(--platform linux/arm64) ;;
116+
*) args+=(--platform linux/amd64) ;;
117+
esac
118+
119+
[[ -n "${FILE}" ]] && args+=(--file "${FILE}")
120+
[[ -n "${TARGET}" ]] && args+=(--target "${TARGET}")
121+
122+
while IFS= read -r build_arg; do
123+
[[ -z "${build_arg}" ]] && continue
124+
args+=(--build-arg "${build_arg}")
125+
done <<< "${BUILD_ARGS}"
126+
127+
while IFS= read -r label; do
128+
[[ -z "${label}" ]] && continue
129+
args+=(--label "${label}")
130+
done <<< "${LABELS}"
131+
132+
while IFS= read -r cache; do
133+
[[ -z "${cache}" ]] && continue
134+
args+=(--cache-from "${cache}")
135+
done <<< "${CACHE_FROM}"
136+
137+
while IFS= read -r cache; do
138+
[[ -z "${cache}" ]] && continue
139+
args+=(--cache-to "${cache}")
140+
done <<< "${CACHE_TO}"
141+
142+
if [[ "${PUSH}" == "true" ]]; then
143+
args+=(--push)
144+
else
145+
echo "::warning::push=false — the image is built but discarded (no local docker daemon to --load into)"
146+
fi
147+
148+
bash "${GITHUB_ACTION_PATH}/build_with_remote_buildkit.sh" \
149+
docker buildx build "${args[@]}" "${CONTEXT}"

‎.github/actions/docker-build-remote-buildkit/build_with_remote_buildkit.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/usr/bin/env bash
2+
# Build a docker image on the remote BuildKit pool used by OSDC/ARC runners,
3+
# which have no host docker daemon. Registers a remote buildx builder, then runs
4+
# the passed command ("$@", e.g. a build.sh or `make ... -image`) and retries
5+
# only connection-phase failures while the autoscaled pool is cold/at capacity.
6+
# Genuine build errors (once BuildKit has started the build) are never retried.
7+
#
8+
# Adapted from pytorch/pytorch .github/scripts/build_with_remote_buildkit.sh.
9+
set -euo pipefail
10+
11+
case "$(uname -m)" in
12+
aarch64|arm64) buildkit_addr="tcp://buildkitd-arm64.buildkit:1234" ;;
13+
*) buildkit_addr="tcp://buildkitd-amd64.buildkit:1234" ;;
14+
esac
15+
16+
# Bare `create` on purpose: `docker buildx create --bootstrap` and
17+
# docker/setup-buildx-action both run `buildx inspect --bootstrap`, whose ~20s
18+
# gRPC connect timeout fails on a cold or bursting pool before the autoscaler
19+
# can add a builder.
20+
docker buildx create --name remote-buildkit --driver remote --use "${buildkit_addr}" >/dev/null 2>&1 \
21+
|| docker buildx use remote-buildkit
22+
23+
log="$(mktemp)"
24+
trap 'rm -f "${log}"' EXIT
25+
26+
# 480 x 15s ~= 2h of waiting for a builder, which covers a fully cold or
27+
# saturated pool. Bound the real ceiling with the job's timeout-minutes.
28+
attempts="${REMOTE_BUILDKIT_CONNECT_ATTEMPTS:-480}"
29+
delay="${REMOTE_BUILDKIT_CONNECT_DELAY:-15}"
30+
for attempt in $(seq 1 "${attempts}"); do
31+
set +e
32+
"$@" 2>&1 | tee "${log}"
33+
rc="${PIPESTATUS[0]}"
34+
set -e
35+
if [[ "${rc}" -eq 0 ]]; then
36+
exit 0
37+
fi
38+
# Retry only while buildx never reached a worker (cold pool). Once BuildKit has
39+
# started the build it emits progress ("load build definition", context
40+
# transfer, "[n/m]" steps); a failure after that is a real build error.
41+
if [[ "${attempt}" -lt "${attempts}" ]] \
42+
&& ! grep -qE "load build definition|transferring context|\[[0-9 ]+/[0-9 ]+\]" "${log}" \
43+
&& grep -qiE "waiting for connection|failed to (dial|list workers)|connection (refused|reset)|no such host|context deadline exceeded|server preface" "${log}"; then
44+
echo "Remote BuildKit not ready yet (attempt ${attempt}/${attempts}); retrying in ${delay}s..." >&2
45+
sleep "${delay}"
46+
continue
47+
fi
48+
exit "${rc}"
49+
done

‎.github/workflows/_docker-build.yml‎

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -21,33 +21,25 @@ jobs:
2121
include:
2222
# --- CPU ---
2323
- runner: mt-rel-l-x86iavx512-8-64
24-
platform: linux/amd64
2524
base_image: quay.io/pypa/manylinux_2_28_x86_64
2625
install_cuda: ""
2726
variant_tag: cpu-x86_64
28-
buildkit_addr: tcp://buildkitd-amd64.buildkit:1234
2927
dockerfile: Dockerfile
3028
- runner: mt-rel-l-arm64g4-16-62
31-
platform: linux/arm64
3229
base_image: quay.io/pypa/manylinux_2_28_aarch64
3330
install_cuda: ""
3431
variant_tag: cpu-aarch64
35-
buildkit_addr: tcp://buildkitd-arm64.buildkit:1234
3632
dockerfile: Dockerfile.aarch64
3733
# --- CUDA (all versions in one image, self-hosted DinD runners) ---
3834
- runner: mt-rel-l-x86iavx512-8-64
39-
platform: linux/amd64
4035
base_image: quay.io/pypa/manylinux_2_28_x86_64
4136
install_cuda: "1"
4237
variant_tag: cuda-x86_64
43-
buildkit_addr: tcp://buildkitd-amd64.buildkit:1234
4438
dockerfile: Dockerfile
4539
- runner: mt-rel-l-arm64g4-16-62
46-
platform: linux/arm64
4740
base_image: quay.io/pypa/manylinux_2_28_aarch64
4841
install_cuda: "1"
4942
variant_tag: cuda-aarch64
50-
buildkit_addr: tcp://buildkitd-arm64.buildkit:1234
5143
dockerfile: Dockerfile.aarch64
5244
container:
5345
image: "ghcr.io/actions/actions-runner:latest"
@@ -63,14 +55,7 @@ jobs:
6355
username: ${{ github.repository_owner }}
6456
password: ${{ secrets.GITHUB_TOKEN }}
6557

66-
- name: Set up Docker Buildx
67-
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
68-
with:
69-
driver: remote
70-
endpoint: ${{ matrix.buildkit_addr }}
71-
7258
- name: Docker meta
73-
if: inputs.push
7459
id: meta
7560
uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0
7661
with:
@@ -81,15 +66,18 @@ jobs:
8166
type=sha,prefix=${{ matrix.variant_tag }}-,format=short
8267
type=raw,value=${{ matrix.variant_tag }}-latest,enable=${{ github.ref == 'refs/heads/main' }}
8368
69+
# Builds on the in-cluster remote BuildKit service: OSDC runners have no
70+
# host docker daemon. The action registers the builder for this runner's
71+
# architecture without bootstrapping it, and retries while the builder
72+
# pool scales up.
8473
- name: Build and push
85-
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6.19.2
74+
uses: pytorch/test-infra/.github/actions/docker-build-remote-buildkit@osdc-remote-buildkit-action
8675
with:
8776
context: ./docker
8877
file: ./docker/${{ matrix.dockerfile }}
89-
platforms: ${{ matrix.platform }}
9078
push: ${{ inputs.push }}
91-
tags: ${{ steps.meta.outputs.tags || '' }}
92-
labels: ${{ steps.meta.outputs.labels || '' }}
79+
tags: ${{ steps.meta.outputs.tags }}
80+
labels: ${{ steps.meta.outputs.labels }}
9381
build-args: |
9482
BASE_IMAGE=${{ matrix.base_image }}
9583
INSTALL_CUDA=${{ matrix.install_cuda }}

‎.github/workflows/docker-build.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ on:
1010
pull_request:
1111
paths:
1212
- "docker/**"
13+
# Validate changes to the build workflows themselves (push: false on PRs)
14+
- ".github/workflows/docker-build.yml"
15+
- ".github/workflows/_docker-build.yml"
1316
workflow_dispatch: {}
1417

1518
concurrency:

0 commit comments

Comments
 (0)