Skip to content

Move Linux CI jobs from linux_job_v2 to linux_job_v3 (OSDC) - #22107

Draft
huydhn wants to merge 3 commits into
mainfrom
osdc/linux-job-v3
Draft

Move Linux CI jobs from linux_job_v2 to linux_job_v3 (OSDC)#22107
huydhn wants to merge 3 commits into
mainfrom
osdc/linux-job-v3

Conversation

@huydhn

@huydhn huydhn commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Part 2 of moving ExecuTorch's Linux CI to OSDC. Independent of #22106; #22108 stacks on this.

87 call sites across 24 workflows move to linux_job_v3. ROCm stays on v2 — no OSDC hardware.

Most of the diff is mechanical. The part worth review is the CI image: v3 hands docker-image straight to the pod's container:, which the kubelet pulls before any step runs, so v2's in-job calculate-docker-image — which both resolved ci-image:<name> and waited for docker-builds to push it — has nowhere left to run. _docker-image.yml replaces both halves, and callers name the image in full. Reusable workflows keep taking ci-image:<name> unchanged, so their own callers did not move.

doc-build's upload-gh-pages no longer needs run-with-docker: false, since v3 checks out and runs the script in one container.

Review order: _docker-image.yml, then _unittest.yml as a worked caller example, then the rest.

Draft until a CI run shows what OSDC's PyPI reachability breaks — cuda.yml already needed PIP_EXTRA_INDEX_URL cleared.

Authored with Claude Code.

Every Linux job except ROCm moves to the OSDC reusable workflow. ROCm has no
OSDC hardware and maps to itself in arc.yaml, so rocm.yml stays on v2.

Most of the diff is mechanical: the v2 -> v3 rename, EC2 runner labels swapped
for their OSDC equivalents per pytorch/pytorch .github/arc.yaml, and the removal
of use-custom-docker-registry, which v3 ignores. Two things are not.

The first is the CI image. v2 resolved `ci-image:<name>` in-job through
calculate-docker-image, which also waited for docker-builds to push it. v3 hands
`docker-image` straight to the runner pod's container, which the kubelet pulls
before any step of the job runs, so neither the resolution nor the wait can
happen inside the job any more. The new _docker-image.yml does both: it emits
`git rev-parse HEAD:.ci/docker`, and on a commit that touches .ci/docker it
blocks until docker-builds has pushed. Callers depend on it and spell the image
out as 308535385114.dkr.ecr.us-east-1.amazonaws.com/executorch/ci-image:<name>-<hash>,
the same reference calculate-docker-image derived. Reusable workflows that take
a docker-image input keep taking `ci-image:<name>` unchanged, so their callers
did not have to move.

The second is the jobs that relied on the EC2 host. doc-build's upload-gh-pages
no longer needs run-with-docker: false, because v3 checks out and runs the
script in the same container, so the credentials git push needs are already in
the repo's git config. The two arm backend jobs that raise
fs.inotify.max_user_watches now tolerate failure, since that is a node-level
setting an unprivileged pod cannot change.

Review order: _docker-image.yml first, then _unittest.yml as a small worked
example of the caller change, then the rest.

Note that arc.yaml still maps linux.arm64.2xlarge to l-arm64g2-6-32, which
ci-infra renamed to l-arm64g2-6-25 in #625; this uses the deployed name.

Authored with Claude Code.
@pytorch-bot

pytorch-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22107

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 15 Pending, 5 Unclassified Failures

As of commit 61eba98 with merge base 9a2d135 (image):

NEW FAILURES - The following jobs have failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 24, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

docker-builds loses its pull_request trigger, so on a PR it only runs once the
ciflow/docker label tags the commit. Without that tag there is nothing to wait
for, and the poll would sit for four hours before timing out. Give the tag five
minutes to appear, then fail with what to do about it.

Authored with Claude Code.
@huydhn

huydhn commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Two follow-ups pushed:

  • _docker-image.yml now fails fast (after a 5 minute grace period) when a commit changes .ci/docker but no docker-builds run exists for its SHA, telling you to apply the ciflow/docker label. Needed because Move docker-builds to OSDC runners #22106 drops docker-builds' pull_request trigger — without this the poll would sit for four hours before timing out.
  • Blocked, like Move docker-builds to OSDC runners #22106, on meta-pytorch/pytorch-gha-infra#1443 for role/arc trust policy access.

@huydhn
huydhn deployed to upload-benchmark-results August 25, 2026 00:08 — with GitHub Actions Active
arc.yaml maps linux.arm64.2xlarge to l-arm64g2-6-32 and ci-infra renamed that
def to l-arm64g2-6-25, but neither name schedules: the three arm64 legs of
executorch's first OSDC docker-builds run sat queued for nearly four hours and
were cancelled without a runner ever picking them up. mt-l-arm64g4-16-62 is the
only arm64 label with live production evidence, and it is what pytorch/pytorch's
own OSDC docker-builds uses.

That also folds in the one job that was mapped to mt-l-arm64g3-16-62, which is
the same size and equally unattested, so all arm64 work now lands on one proven
label.

Authored with Claude Code.
@huydhn
huydhn deployed to upload-benchmark-results August 25, 2026 06:57 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant