Skip to content

Commit 42b7b7b

Browse files
huydhnhuydhn
authored
Add a guide to writing CI jobs on OSDC runners (#8518)
A user-facing guide for repo owners (and their agents) putting CI jobs on OSDC runners: `linux_job_v3` and the standalone container pattern, picking a runner label, fork-PR/OIDC behaviour, building images on remote BuildKit, and the gotchas that catch people out. This was only written down in scattered form across ci-infra, `arc.yaml` and people's heads, so everyone migrating a workflow rediscovers the same handful of surprises. This is already available at https://github.com/pytorch/pytorch/wiki/Using-OSDC-runners-%E2%80%94-a-guide-to-writing-CI-jobs, but I want to commit this here so that we can properly track future changes to the guide.
1 parent 50fb33d commit 42b7b7b

1 file changed

Lines changed: 298 additions & 0 deletions

File tree

‎docs/osdc_runners.md‎

Lines changed: 298 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,298 @@
1+
# Using OSDC runners — a guide to writing CI jobs
2+
3+
OSDC runners are PyTorch's Kubernetes-hosted self-hosted GitHub Actions runners, run with ARC (Actions
4+
Runner Controller) and operated from [`pytorch/ci-infra`](https://github.com/pytorch/ci-infra). This
5+
guide covers how to target them from a workflow: picking a runner, writing the job, building images, and
6+
the constraints to design around.
7+
8+
## Mental model
9+
10+
An OSDC runner is an **ephemeral Kubernetes pod** on a cluster node. Targeting one is two decisions:
11+
12+
1. **Hardware**, chosen by the runner *label* (`mt-l-x86…`). The label is the only thing that decides
13+
vCPU, RAM, GPU and disk.
14+
2. **Software**, which comes entirely from **a container image you provide**. The runner pod ships only
15+
the actions-runner agent: no preinstalled toolchain, no system python, cuda, compilers or git-lfs, and
16+
**no host docker daemon**. Whatever your job needs must be in the image you name, and to *build* an
17+
image you use the remote BuildKit service described in §5, not `docker build`.
18+
19+
Everything else below follows from those two facts.
20+
21+
---
22+
23+
## 1. Quick start with `linux_job_v3`
24+
25+
For most jobs, call the reusable
26+
[`linux_job_v3.yml`](https://github.com/pytorch/test-infra/blob/main/.github/workflows/linux_job_v3.yml)
27+
rather than writing the pod plumbing yourself. It handles checkout, the container, AWS credentials and
28+
artifact upload. You supply a **runner label**, a **container image** and a **script**:
29+
30+
```yaml
31+
jobs:
32+
my-build:
33+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
34+
with:
35+
runner: mt-l-x86iavx512-8-64 # OSDC label → 8 vCPU / 64 GiB
36+
docker-image: ghcr.io/pytorch/my-ci-image:latest # prebuilt, pullable image
37+
timeout: 60
38+
submodules: recursive
39+
script: |
40+
python -m pip install -e .
41+
pytest test/
42+
```
43+
44+
For a GPU job, set `gpu-arch-type: cuda` and pick a GPU label. The container then gets `--gpus all`
45+
automatically:
46+
47+
```yaml
48+
my-gpu-test:
49+
uses: pytorch/test-infra/.github/workflows/linux_job_v3.yml@main
50+
with:
51+
runner: mt-l-x86aavx2-29-113-a10g # 1× A10G
52+
gpu-arch-type: cuda
53+
gpu-arch-version: "12.4"
54+
docker-image: ghcr.io/pytorch/my-cuda-image:latest
55+
script: pytest test/ -m gpu
56+
```
57+
58+
Common inputs: `runner`, `docker-image`, `gpu-arch-type`/`gpu-arch-version`, `script`, `timeout`,
59+
`repository`, `ref`, `submodules`, `download-artifact`, `upload-artifact`, `upload-artifact-to-s3`,
60+
`secrets-env`. If you omit `docker-image` it defaults to `pytorch/almalinux-builder:<arch>`.
61+
62+
## 2. Standalone pattern
63+
64+
If your job doesn't fit the single-container shape of `linux_job_v3`, write it directly against a raw ARC
65+
label plus a `container:` block:
66+
67+
```yaml
68+
jobs:
69+
build:
70+
runs-on: mt-l-x86iavx512-16-128 # OSDC label
71+
container:
72+
image: ghcr.io/pytorch/my-ci-image:latest # your image = your toolchain
73+
options: --gpus all # only for GPU labels
74+
permissions:
75+
id-token: write # required to assume role/arc
76+
contents: read
77+
steps:
78+
- uses: actions/checkout@v4
79+
with: { submodules: recursive }
80+
- run: python -m pip install -e . && pytest test/
81+
```
82+
83+
Rules for the standalone pattern:
84+
85+
- **No host docker.** You cannot run `docker build` or `docker pull` on the node. Put everything in
86+
`container.image` and build new images with BuildKit, as described in §5. This is the most common
87+
breakage when porting a job from EC2.
88+
- **Avoid `sudo` and job-time package installs.** Bake everything the job needs into the image rather
89+
than running `yum install` or `apt-get install` from a step. `sudo` may not even be present in your
90+
image.
91+
- **No relative action paths.** `uses: ./.github/actions/foo` does not resolve; see §6 #6. Reference
92+
actions by their full `owner/repo/.github/actions/<name>@ref` path instead.
93+
- **AWS access goes through `role/arc`.** The pod carries no credentials of its own, so assume the role
94+
via GitHub OIDC as shown in §4 rather than reusing an EC2 host role.
95+
96+
## 3. Picking a runner label
97+
98+
### How to read a label
99+
100+
Labels you write in a workflow look like this:
101+
102+
`mt-l-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu}[-{count}]]`
103+
104+
- `mt-l-` is the prefix for production Linux runners. **Use it for every job.**
105+
- `arch` is `x86` or `arm64`. `vendor` is `i` for Intel-ISA, `a` for AMD-ISA, `g2`/`g3`/`g4` for Graviton
106+
generation. `features` is `avx2`, `avx512` or `amx`.
107+
- `{vcpu}-{memory}` is vCPU count and **GiB**. A GPU suffix is `t4`, `a10g`, `l4`, `a100` or `h100`, plus
108+
a count when there is more than one.
109+
- A `b` before the architecture means bare metal, so the job gets a whole node to itself.
110+
111+
For example, `mt-l-x86aavx2-29-113-a10g` is a Linux x86 AMD-ISA AVX2 runner with 29 vCPU, 113 GiB and one
112+
A10G. The full grammar, including prefixes used by other fleets, is in
113+
[`runner_naming_convention.md`](https://github.com/pytorch/ci-infra/blob/main/osdc/docs/runner_naming_convention.md).
114+
115+
### Where the labels come from
116+
117+
| Question | Where to look |
118+
|---|---|
119+
| Which labels exist, and what hardware does each one give me? | [`osdc/modules/arc-runners/defs/`](https://github.com/pytorch/ci-infra/tree/main/osdc/modules/arc-runners/defs), one YAML file per label listing its vCPU, memory, disk and GPU |
120+
| Which OSDC label replaces the EC2 label my job uses today? | [`pytorch/pytorch:.github/arc.yaml`](https://github.com/pytorch/pytorch/blob/main/.github/arc.yaml), under `runner_mapping` |
121+
122+
Common mappings, with the full list in `arc.yaml`:
123+
124+
| EC2 label | OSDC label | Hardware |
125+
|---|---|---|
126+
| `linux.2xlarge` | `mt-l-x86iavx512-8-64` | 8 vCPU / 64Gi / 200G |
127+
| `linux.4xlarge` | `mt-l-x86iavx512-16-128` | 15 vCPU / 116Gi / 300G |
128+
| `linux.12xlarge` | `mt-l-x86iavx512-48-384` | 46 vCPU / 350Gi / 600G |
129+
| `linux.arm64.m7g.4xlarge` | `mt-l-arm64g3-16-62` | 15 vCPU / 56Gi / 256G |
130+
| `linux.g5.4xlarge.nvidia.gpu` | `mt-l-x86aavx2-29-113-a10g` | 1× A10G, 29 vCPU / 113Gi |
131+
| `linux.aws.h100` | `mt-l-x86iamx-22-225-h100` | 1× H100, 22 vCPU / 225Gi |
132+
133+
### Special cases
134+
135+
- **`-fab` H100 variants.** Every H100 label has a `-fab` counterpart with IMEX channels configured for
136+
multi-node GPU fabric. Use the plain label unless your job specifically needs IMEX.
137+
- **`rel-` release pool.** `rel-l-x86iavx512-44-340` and `rel-l-arm64g3-44-340` are reserved for building
138+
release artifacts such as wheels, and live in a protected runner group rather than the general pool.
139+
Don't point an ordinary job at them; contact the Dev Infra team if you need access.
140+
- **Non-OSDC runners are unchanged.** Labels with no OSDC equivalent, such as ROCm, XPU and TPU, map to
141+
themselves in `arc.yaml`, so those jobs keep running exactly where they do today.
142+
143+
## 4. Fork PRs and OIDC
144+
145+
OSDC jobs get AWS access by assuming `arn:aws:iam::308535385114:role/arc` through GitHub OIDC, which
146+
requires a **writable OIDC token**. Give the job the permission and assume the role:
147+
148+
```yaml
149+
permissions:
150+
id-token: write
151+
contents: read
152+
steps:
153+
- uses: aws-actions/configure-aws-credentials@v4
154+
with:
155+
role-to-assume: arn:aws:iam::308535385114:role/arc
156+
aws-region: us-east-1
157+
role-duration-seconds: 18000 # the server-side maximum
158+
```
159+
160+
`linux_job_v3` does this for you. GitHub treats fork PRs differently, and that changes what a job can do:
161+
162+
| | Same-repo PR or push | Fork PR |
163+
|---|---|---|
164+
| `id-token: write` (OIDC token) | granted | **withheld**, no token can be minted |
165+
| Repo secrets | available | **not exposed** |
166+
| Can assume `role/arc` | yes | **no**, `configure-aws-credentials` fails |
167+
168+
For that reason
169+
[`linux_job_v3`](https://github.com/pytorch/test-infra/blob/main/.github/workflows/linux_job_v3.yml)
170+
marks its "Configure AWS credentials" step **`continue-on-error: true`**: on a fork PR the assume-role
171+
step fails and the job continues *without* AWS credentials. The practical consequences:
172+
173+
- **Uploading artifacts or docs to S3 does not work on fork PRs**, which covers `upload-artifact-to-s3`
174+
and doc-preview upload. Use GitHub-native `upload-artifact` if the artifact must survive on a fork PR.
175+
- **Any step that needs `role/arc`, a registry push or a secret will fail or be skipped on a fork PR.**
176+
Gate those steps on `github.event.pull_request.head.repo.fork == false`, or move image builds and
177+
pushes to a same-repo trigger such as `push` or a `workflow_run` after merge.
178+
- If you need to run privileged work against fork-PR *content*, use a `pull_request_target` workflow with
179+
care. It runs with the base repository's token and secrets against fork code, which is easy to turn
180+
into a security hole, so restrict it to trusted steps that never execute fork code, such as labeling.
181+
182+
## 5. Building a Docker image with BuildKit
183+
184+
This section applies only if your job builds an image rather than pulling one.
185+
186+
There is no docker daemon on an OSDC runner, so `docker build` fails. OSDC instead runs a per-architecture
187+
remote `buildkitd` in every cluster, reachable at `tcp://buildkitd-amd64.buildkit:1234` and
188+
`tcp://buildkitd-arm64.buildkit:1234`. Drive it through test-infra's composite action, which registers the
189+
builder for the runner's architecture and retries the connection failures a cold builder pool produces:
190+
191+
```yaml
192+
- name: Build & push
193+
uses: pytorch/test-infra/.github/actions/docker-build-remote-buildkit@main
194+
with:
195+
context: ./docker
196+
tags: ghcr.io/pytorch/my-ci-image:${{ github.sha }}
197+
push: true # --push, never --load: there is no local daemon to load into
198+
```
199+
200+
If your build is driven by a script or a make target rather than a direct `buildx` call, pass the whole
201+
command instead of the buildx inputs. The command owns its own tags and `--push`:
202+
203+
```yaml
204+
with:
205+
command: .ci/docker/manywheel/build.sh manylinux2_28-builder:cpu
206+
```
207+
208+
Do not use `docker/setup-buildx-action`, and do not pass `--bootstrap`: both run
209+
`buildx inspect --bootstrap`, whose short connect timeout expires on a cold builder pool before the
210+
autoscaler can add a builder. The action's
211+
[README](https://github.com/pytorch/test-infra/tree/main/.github/actions/docker-build-remote-buildkit)
212+
covers the remaining inputs.
213+
214+
### Make sure the image exists before you use it
215+
216+
**Never assume the image is there.** Check the registry first and build it only if it is missing, so the
217+
job is correct whether or not something else already built it:
218+
219+
```yaml
220+
- name: Resolve the image, building it if it is missing
221+
id: image
222+
run: |
223+
set -euo pipefail
224+
# A content-addressed tag: "exists" then means exactly the right image,
225+
# which a mutable tag like :latest can never guarantee.
226+
TAG="ghcr.io/pytorch/my-ci-image:$(git rev-parse HEAD:docker)"
227+
echo "tag=${TAG}" >> "${GITHUB_OUTPUT}"
228+
# Registry-side inspect; no local daemon needed.
229+
if docker buildx imagetools inspect "${TAG}" >/dev/null 2>&1; then
230+
echo "exists=true" >> "${GITHUB_OUTPUT}"
231+
else
232+
echo "exists=false" >> "${GITHUB_OUTPUT}"
233+
fi
234+
235+
- name: Build the image
236+
if: steps.image.outputs.exists == 'false'
237+
uses: pytorch/test-infra/.github/actions/docker-build-remote-buildkit@main
238+
with:
239+
context: ./docker
240+
tags: ${{ steps.image.outputs.tag }}
241+
push: true
242+
```
243+
244+
The build step only returns once the push has completed, so anything after it can use
245+
`steps.image.outputs.tag` safely. What not to do: start a job that pulls an image some other workflow is
246+
still building, and retry the pull until it appears. That either fails on a missing manifest or silently
247+
runs against a stale image, and it is why the check above resolves a content-addressed tag rather than a
248+
floating one.
249+
250+
## 6. Constraints and gotchas
251+
252+
1. **You must supply a prebuilt, pullable image.** Build new images with BuildKit, as described in §5.
253+
There is no host `docker build`, and job-time package installs such as `sudo yum install` belong in
254+
the image instead.
255+
2. **Size the label by what the job needs**, not by the name of the EC2 runner it used to run on. The
256+
EC2-to-ARC translation is not 1:1 on RAM. Check the runner defs linked in §3 and pick the RAM and GPU
257+
you need.
258+
3. **Machine-wide CPU counts are misleading.** `std::thread::hardware_concurrency()`, `os.cpu_count()`
259+
and `multiprocessing.cpu_count()` report the node's CPUs, not the pod's cpuset, for example 192
260+
rather than 16. Pin thread counts explicitly, or use an affinity-aware call such as `nproc` or
261+
`len(os.sched_getaffinity(0))`.
262+
4. **Linux only.** There is no Windows or macOS, and no ROCm, XPU or TPU on OSDC; those stay on their
263+
partner clouds.
264+
5. **Fork PRs have no OIDC token and no secrets.** See §4.
265+
6. **No relative action paths.** `uses: ./.github/actions/<name>` fails even with the repository checked
266+
out at the workspace root, because the ARC Kubernetes hook copies the workspace into the job
267+
container and the runner process itself never sees the checkout. The symptom is
268+
`Can't find 'action.yml'` *after* a successful checkout. Use
269+
`uses: <owner>/<repo>/.github/actions/<name>@<ref>` instead, which fetches the action from GitHub.
270+
The action then comes from `<ref>` rather than the PR's own checkout, so a PR that changes an action
271+
must point the ref at its own branch to test it.
272+
7. **Path expressions return host paths, not container paths.** `${{ runner.temp }}` and
273+
`${{ github.workspace }}` evaluate to the runner's own filesystem (`/home/runner/_work/...`),
274+
while your job sees the container mount (`/__w/...`). Passing one of those expressions to a
275+
command running in the container gives "path not found" for a file you just created. Use the
276+
`$RUNNER_TEMP` and `$GITHUB_WORKSPACE` environment variables instead, or write the path to a step
277+
output and reference that.
278+
8. **`container.image` cannot read `secrets` or `env`.** Only `github`, `needs`, `strategy`, `matrix`,
279+
`vars` and `inputs` are allowed there, so `image: ${{ inputs.docker-image }}` works but
280+
`image: ${{ secrets.X }}` fails at startup with "context not available". For a private image, put the
281+
credentials in `container.credentials`, which does allow `secrets`. Likewise `uses:` is always
282+
literal: you cannot template the action or its `@ref`.
283+
284+
---
285+
286+
## Appendix: sources
287+
288+
Where the information above comes from. These are primarily of interest to the Dev Infra team; link to
289+
them rather than re-deriving their contents.
290+
291+
- Runner naming convention — [`osdc/docs/runner_naming_convention.md`](https://github.com/pytorch/ci-infra/blob/main/osdc/docs/runner_naming_convention.md)
292+
- Deployed runner defs — [`osdc/modules/arc-runners/defs/`](https://github.com/pytorch/ci-infra/tree/main/osdc/modules/arc-runners/defs) and `arc-runners-h100`
293+
- Which labels are live per cluster — [`osdc/clusters.yaml`](https://github.com/pytorch/ci-infra/blob/main/osdc/clusters.yaml)
294+
- Runner image pinning — [`osdc/docs/runner-image-autoresolve.md`](https://github.com/pytorch/ci-infra/blob/main/osdc/docs/runner-image-autoresolve.md)
295+
- BuildKit service — [`osdc/modules/buildkit/README.md`](https://github.com/pytorch/ci-infra/blob/main/osdc/modules/buildkit/README.md)
296+
- Image build action — [`test-infra/.github/actions/docker-build-remote-buildkit`](https://github.com/pytorch/test-infra/tree/main/.github/actions/docker-build-remote-buildkit)
297+
- EC2 to OSDC label mapping — [`pytorch/pytorch:.github/arc.yaml`](https://github.com/pytorch/pytorch/blob/main/.github/arc.yaml)
298+
- Reusable workflow — [`test-infra/.github/workflows/linux_job_v3.yml`](https://github.com/pytorch/test-infra/blob/main/.github/workflows/linux_job_v3.yml)

0 commit comments

Comments
 (0)