|
| 1 | +--- |
| 2 | +name: migrate-workflow-ec2-to-osdc |
| 3 | +description: Step-by-step playbook for migrating a pytorch/pytorch .github/workflows/*.yml from EC2 to OSDC (ARC) runners — covers both dial-up and 100% opt-in patterns, with the inputs that must be plumbed through _linux-build.yml / _linux-test.yml. Use when migrating a workflow off EC2 onto on-site data center / EKS-hosted self-hosted runners (OSDC / ARC), enabling the ARC experiment, or wiring up `use-arc` / `runner_prefix` / `python-version` / `compiler` / `cuda-version` inputs. |
| 4 | +--- |
| 5 | + |
| 6 | +# Migrating a workflow from EC2 to OSDC |
| 7 | + |
| 8 | +OSDC (= ARC = on-site data center, EKS-hosted self-hosted runners) replaces EC2 runners. Migration touches the workflow file *and* requires a few inputs to flow into the reusable `_linux-build.yml` / `_linux-test.yml` so the OSDC code paths (`build-osdc`, `test-osdc`) activate. EC2 paths (`build`, `test`) and OSDC paths gate on `inputs.use-arc` (`!inputs.use-arc` vs `inputs.use-arc`), so flipping that input swaps execution lanes. |
| 9 | + |
| 10 | +This playbook is for workflows that **call the reusable `_linux-build.yml` / `_linux-test.yml`** (e.g. `test-b200.yml`, `pull.yml`, `trunk.yml`, `tsan.yml`, `operator_microbenchmark.yml`). For *standalone* OSDC workflows (raw ARC label + `container:` directive), see `osdc-workflow-pattern.md`. |
| 11 | + |
| 12 | +## Decide: dial-up vs. 100% opt-in |
| 13 | + |
| 14 | +| Pattern | When to use | Example | |
| 15 | +|---|---|---| |
| 16 | +| **Dial-up** (preferred) | Existing workflow with broad coverage; you want to ramp OSDC adoption via labels like `pull.yml`/`trunk.yml`. | `test-b200.yml` (PR #181544) | |
| 17 | +| **100% opt-in** | Workflow is meant to *always* run on OSDC (e.g. testing the single B200 we own on EKS). | `operator_microbenchmark.yml`, `attention_op_microbenchmark.yml` (B200 jobs) | |
| 18 | + |
| 19 | +The two pieces — `runner_prefix` and `use-arc` — **must move together**. Hardcoding one but driving the other off the determinator is a bug. See `feedback-osdc-migration-dial-up.md`. |
| 20 | + |
| 21 | +## Migration steps (dial-up pattern) |
| 22 | + |
| 23 | +Worked reference: PR #181544 / commit `f156b7ddfd1` ("Migrate smoke test on B200 to OSDC"). The diff was 10 lines. |
| 24 | + |
| 25 | +### 1. Make sure `get-label-type` opts into the ARC experiment |
| 26 | + |
| 27 | +In the `get-label-type` job that calls `_runner-determinator.yml`, add: |
| 28 | + |
| 29 | +```yaml |
| 30 | +check_experiments: arc,lf |
| 31 | +``` |
| 32 | +
|
| 33 | +Without this the determinator won't consider the ARC experiment and `use-arc` will always be `false`. |
| 34 | + |
| 35 | +### 2. Plumb four inputs into the build job |
| 36 | + |
| 37 | +On the call to `./.github/workflows/_linux-build.yml`: |
| 38 | + |
| 39 | +```yaml |
| 40 | +with: |
| 41 | + runner_prefix: "${{ needs.get-label-type.outputs.label-type }}" # likely already present |
| 42 | + ...existing inputs... |
| 43 | + use-arc: ${{ needs.get-label-type.outputs.use-arc == 'true' }} |
| 44 | + python-version: "3.10" # match the docker-image-name's python |
| 45 | + compiler: gcc11 # match the docker-image-name's compiler |
| 46 | + cuda-version: "13.0" # match the docker-image-name's cuda (or "" for CPU) |
| 47 | +``` |
| 48 | + |
| 49 | +The last three feed `setup-linux` so it can configure the OSDC container env. Read them off the existing `docker-image-name` (e.g. `pytorch-linux-jammy-cuda13.0-cudnn9-py3-gcc11` → py3.10 / gcc11 / cuda13.0). |
| 50 | + |
| 51 | +### 3. Add `get-label-type` to the test job's `needs` and plumb the same inputs |
| 52 | + |
| 53 | +On the call to `./.github/workflows/_linux-test.yml`: |
| 54 | + |
| 55 | +```yaml |
| 56 | +needs: |
| 57 | + - <existing-build-job> |
| 58 | + - get-label-type # add this |
| 59 | +with: |
| 60 | + ...existing inputs... |
| 61 | + use-arc: ${{ needs.get-label-type.outputs.use-arc == 'true' }} |
| 62 | + python-version: "3.10" |
| 63 | + compiler: gcc11 |
| 64 | + cuda-version: "13.0" |
| 65 | +``` |
| 66 | + |
| 67 | +The test job needs a direct `needs: get-label-type` dependency to read its outputs (it can't rely on transitive `needs` through the build job). |
| 68 | + |
| 69 | +### 4. Drop the OSDC-incompatible inputs |
| 70 | + |
| 71 | +If the EC2 path used inputs that don't apply on OSDC (e.g. `aws-role-to-assume:` for ECR pulls), remove them. OSDC has its own AWS role wired into `setup-linux` (`arn:aws:iam::308535385114:role/arc`) and pulls images from `ghcr.io/pytorch`, not ECR. |
| 72 | + |
| 73 | +In #181544 this meant deleting: |
| 74 | +```yaml |
| 75 | +aws-role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only |
| 76 | +``` |
| 77 | + |
| 78 | +### 5. Verify |
| 79 | + |
| 80 | +- `gh workflow run` or push a PR with the relevant `ciflow/*` tag. |
| 81 | +- Look for the `build-osdc` / `test-osdc` jobs running (they're separate jobs in `_linux-build.yml` / `_linux-test.yml`, gated on `inputs.use-arc`). When `use-arc` is `false`, the original `build` / `test` jobs run instead. |
| 82 | + |
| 83 | +## Migration steps (100% opt-in pattern) |
| 84 | + |
| 85 | +For a job that should always run on OSDC, skip the determinator-driven plumbing and hardcode: |
| 86 | + |
| 87 | +```yaml |
| 88 | +with: |
| 89 | + runner_prefix: "mt-" |
| 90 | + use-arc: true |
| 91 | + python-version: "3.10" |
| 92 | + compiler: gcc11 |
| 93 | + cuda-version: "13.0" |
| 94 | +``` |
| 95 | + |
| 96 | +Both jobs (build *and* test) need the four inputs. The build job's `runner:` stays as the EC2 label (e.g. `linux.r7i.4xlarge`); `_linux-build.yml`'s `build-osdc` job translates that to the right ARC runner via `map_ec2_to_arc.py`. |
| 97 | + |
| 98 | +Reference: `operator_microbenchmark.yml` jobs `opmicrobenchmark-build-b200` / `opmicrobenchmark-test-b200`. |
| 99 | + |
| 100 | +## What the inputs do (mental model) |
| 101 | + |
| 102 | +`_linux-build.yml` and `_linux-test.yml` each define **two jobs**: |
| 103 | + |
| 104 | +- `build` / `test` — EC2 path. Runs directly on the runner host VM. Gated on `!inputs.use-arc`. |
| 105 | +- `build-osdc` / `test-osdc` — OSDC path. Runs inside a `container:` from `ghcr.io/pytorch/${docker-image-name}`. Gated on `inputs.use-arc`. |
| 106 | + |
| 107 | +The OSDC jobs call `setup-linux` with `use-arc: true` and pass `python-version` / `compiler` / `cuda-version` so the container env matches the build environment. That's why the migration must pass these three inputs — `setup-linux` errors out without them on the OSDC path. |
| 108 | + |
| 109 | +## Common gotchas |
| 110 | + |
| 111 | +- **Forgetting `check_experiments: arc,lf`** → `use-arc` is always `false`, so OSDC never activates and you'll think the migration silently failed. |
| 112 | +- **Forgetting `needs: get-label-type` on the test job** → `${{ needs.get-label-type.outputs.use-arc }}` evaluates to empty, dial-up doesn't engage. |
| 113 | +- **Mismatched `python-version` / `compiler` / `cuda-version` vs. `docker-image-name`** → container has one toolchain, `setup-linux` configures another, build fails confusingly. |
| 114 | +- **Leaving `aws-role-to-assume:` for ECR** → harmless on OSDC (it's only used by EC2 path) but stale and misleading; remove it. |
| 115 | +- **Mixing patterns** (dynamic `runner_prefix` + hardcoded `use-arc: true`, or vice versa) — see `feedback-osdc-migration-dial-up.md`. Pick one pattern and apply it consistently. |
| 116 | + |
| 117 | +## Cross-references |
| 118 | + |
| 119 | +- `osdc-workflow-pattern.md` — for *standalone* OSDC workflows (no reusable build/test). |
| 120 | +- `feedback-osdc-migration-dial-up.md` — why both inputs must move together. |
| 121 | +- `ci-consolidation.md` — what's shared / different between EC2 (`build`/`test`) and OSDC (`build-osdc`/`test-osdc`) jobs in the reusable workflows. |
| 122 | +- `arc-cpu-build-test.md` — how the OSDC path actually runs once activated. |
0 commit comments