Move docker-builds to OSDC runners

[huydhn]

Part 1 of moving ExecuTorch's Linux CI to OSDC. #22107 migrates linux_job_v2 -> v3, #22108 the native jobs.

OSDC pods have no docker daemon, so the builds move to the in-cluster BuildKit pool: build.sh switches to docker buildx build --push under REMOTE_BUILDKIT, driven by test-infra's docker-build-remote-buildkit action. The tag is computed in the workflow now, but it is the same string calculate-docker-image produced, so jobs still on v2 keep resolving. That action only retries failures from before BuildKit starts, so the three retries calculate-docker-image wrapped the build in are kept around the SDK downloads.

sccache resolved its S3 credentials from EC2 instance metadata, which a BuildKit pod has none of, so every compiler invocation died on a metadata timeout. They now reach the build as a BuildKit secret, and install_pytorch.sh caches locally when it is absent. Dropping the S3 backend would have been the smaller change, but the PyTorch build layer takes ~19 minutes only because sccache serves nearly all of it, and uncached it would not fit the job timeout.

pull_request gives way to ciflow/docker, since a fork PR gets no OIDC token and could never push. The tag is registered in pytorch-probot.yml and auto-applied on the paths that need a rebuild, so such a PR still rebuilds its images without anyone having to remember — the same rule pytorch has.

All 16 images build and push green on OSDC in run 32799247406: 16–48 min each against 37–56 on EC2, arm64 included.

Authored with Claude Code.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22106

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit d4c09e3 with merge base 9a2d135 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[huydhn]

Follow-up, deliberately not in this PR: build.sh keeps --no-cache, so there is no Docker layer cache. That was near-free on EC2 (ephemeral runners started with an empty cache anyway), but on OSDC a BuildKit pod is ephemeral with a pod-private cache, so a registry cache — pytorch/pytorch's --cache-from/--cache-to type=registry against a <image>-buildcache tag — is the only way to get reuse.

The plan is to cache on pull_request/push and keep --no-cache on schedule, so the weekly cron still refreshes apt/pip rather than replaying cached layers. Separate PR to keep this one small.

Note sccache already covers the expensive part: the PyTorch build layer is ~19 min of a ~40 min image build, almost entirely cache hits.

[shoumikhin]

Went through this and checked each piece against pytorch's already migrated workflow. The mechanics look right. A few things below.

1. Every PR touching these paths will build all 16 images twice

pull_request is still a trigger, and the new labeler rule adds ciflow/docker on exactly the same paths, which creates the tag and starts the workflow a second time. The two runs get different concurrency keys, so neither cancels the other.

Cost: the last full matrix on main (run 32407405402) used about 610 job minutes, so this roughly doubles that on every push.

On a fork it is also always red: the pull_request half cannot get an OIDC token, so all 16 jobs fail at "Configure AWS credentials". The comment at the top of the workflow already describes this.

Suggestion: drop the pull_request trigger in this PR, or add the labeler rule only once it is gone. pytorch's version has no pull_request trigger.

2. The build no longer retries

calculate-docker-image ran the build through nick-fields/retry with 3 attempts and a 90 second wait, with the comment "this step frequently fails with network error downloading various stuffs".

docker-build-remote-buildkit only retries failures that happen before the build starts. Once BuildKit prints progress, the helper stops retrying. With --no-cache and images that download the QNN, MediaTek, Arm and Zephyr SDKs, a single network hiccup now fails the whole job.

3. Docker Hub login is gone

The old path fetched a read-only Docker Hub token and ran docker login before building, to avoid anonymous pull rate limits on the ubuntu base image. Nothing does that now. If the BuildKit pool has no Docker Hub mirror, expect toomanyrequests when 16 no-cache builds start at once.

4. Small comment fix

.github/labeler.yml says the rule is there "without the pull_request trigger it used to rely on", but that trigger is still present in this PR.

5. Question on the labeler rule

Auto-applying ciflow/docker means the tag run starts on its own, including for pull requests from forks, instead of a maintainer applying the label. Is that intended here, or would you rather keep that label manual?

Things I checked that look correct

• The computed tag is byte for byte what calculate-docker-image produces, so jobs still on linux_job_v2 keep resolving the same image.
• build.sh splices "$@" in before the context, so -t works, and ${REMOTE_BUILDKIT:-} is safe under set -u. Nothing after the build needs a local daemon.
• A paths: filter does not block tag pushes, so the ciflow/docker/* trigger will fire.
• timeout-minutes: 240 looks fine. The slowest image in the last main run took 56 minutes, which leaves room even after a long wait for a cold builder.
• Moving executorch-ubuntu-22.04-cuda-windows off the GPU runner is right. That Dockerfile branch only downloads and installs, it never touches a device.

[huydhn]

Thanks — addressed in 9ebfb7a..d4c09e3.

1. Dropped the pull_request trigger, so ciflow/docker is the only PR path now.

2. Good catch, I'd missed it. The action stops retrying once BuildKit prints progress, so the SDK downloads were unprotected. Three attempts with a 90s wait are back around build.sh.

3. Should be a non-issue: buildkitd mirrors docker.io through the cluster's Harbor pull-through cache, so no login needed. If we do see toomanyrequests I'll add one.

4. Fixed.

5. Intentional — it keeps a PR that changes the images rebuilding them automatically, which is what the pull_request trigger did before. pytorch does the same: .github/labeler.yml auto-applies ciflow/docker on .ci/docker/**, .github/workflows/docker-builds.yml and .lintrunner.toml, alongside ciflow/trunk, ciflow/inductor and others. Our rule is the same shape.

[shoumikhin]

Re-checked at the new head. My earlier items look addressed, and run 32799247406 built 16 of 16 images green, so the runner move and the sccache secret work.

Four things left.

1. The retry has no per attempt limit. The old path killed each attempt at 90 minutes and tried again. The new for attempt in 1 2 3 loop has no bound, so a hung download holds attempt 1 until the 240 minute job timeout and attempts 2 and 3 never run. Fix: timeout 90m ./build.sh ....

2. The tag does not cover requirements-lintrunner.txt. build.sh copies that file into the build context, but the tag is git rev-parse HEAD:.ci/docker, so the file changes the image without changing the tag. The new labeler rule lists it, and buildx build --push always overwrites, where calculate-docker-image pushed only when the tag was missing. So a pull request touching only that file overwrites the tag main is using. Simplest fix is to drop it from the labeler rule.

3. The tag trigger cannot be tested from here. ciflow/docker is not in main's .github/pytorch-probot.yml, and the bot reads that file from the default branch, so labelling this pull request creates no tag. Maybe keep pull_request one more cycle and drop it after a tag run goes green.

4. doc-build.yml passes no ref. Every other linux_job_v2 call passes the head SHA, doc-build gets the merge commit, so it can ask for a tag nothing ever builds and then sit until its timeout: 90 ends it. The same ref line fixes it.