Repository navigation
ci(rocm): add ROCm CI workflow for MI300X self-hosted runners - #1606
Merged
Merged
Conversation
indianspeedster
requested review from
guapisolo,
yueming-yuan and
yushengsu-thu
as code owners
July 9, 2026 19:47
Contributor
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
indianspeedster
force-pushed
the
main
branch
2 times, most recently
from
July 10, 2026 00:27
7f5b504 to
6045ed4
Compare
Collaborator
|
I am worried that this per PR CI will run too long |
Contributor
Author
Happy to adjust as per your feedback. @guapisolo |
Collaborator
|
I think these test files are must added, about 9300s in total. Other parts can be throwed. (updated) |
Collaborator
|
btw. iirc, AMD GPU has very large hbm. spliting into two 4xgpu runner might make more sense. But need to confirm cpu memory is enough. |
Collaborator
|
As we've decided to split amd gpu runners to two 4gpu runners. |
Collaborator
|
Also, add a tag |
Add two new workflow files to support ROCm CI on AMD MI300X hardware: - _run-ci-rocm.yml: Reusable workflow (counterpart to _run-ci.yml). Uses the ROCm container with HIP/CUDA compatibility to run existing CUDA-registered test suites. Container options provide ROCm device access (kfd, dri, video/render groups), GPU readiness via rocm-smi, and streamlined dep install (container ships pre-built SGLang/Megatron, skip_dependency_install defaults to true). - pr-test-rocm.yml: Dispatch workflow matching pr-test.yml structure. resolve-ci-image -> stage-c-8-gpu-mi300x calling _run-ci-rocm.yml. Same input contract (ci_megatron_pr, ci_sglang_pr, ci_image_tag). workflow_dispatch-only while the MI300X runner stabilizes. No changes to existing files — purely additive.
The MI300X dispatch workflow passed ci_megatron_pr / ci_sglang_pr to _run-ci-rocm.yml, which does not declare them, so every dispatch failed at parse time with "Invalid input". Drop the two `with:` entries and the matching dispatch inputs: skip_dependency_install defaults to true, so those refs were never consumed anyway — SGLang/Megatron versions come baked into the ROCm image. Also remove the unreachable run-cpu job and cpu_runner input (CPU tests are hardware-agnostic and already covered by _run-ci.yml), restore the image-tag format guard from pr-test.yml, make container_image required so the dated default tag cannot rot here, and document the ROCm mirror as the deliberate exception to the 1:1 suite-stage mapping in docs/ci/00-stage.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per review on PR radixark#1606, the MI300X host is split into two 4-GPU runners rather than serving one 8-GPU job, so the stage becomes 4-GPU: * stage renamed stage-c-8-gpu-mi300x -> stage-c-4-gpu-mi300x, and the suite added to CI_SUITES[HWBackend.ROCM]. * runs_on becomes ["self-hosted", "amd", "mi300x", "4gpu"]. The live runners carry `amd`, not `rocm`, so the previous label set matched no runner and the job would have queued forever. * the dispatch runs --hw rocm --suite stage-c-4-gpu-mi300x rather than --hw cuda --suite stage-c-8-gpu-h100, which never read register_rocm_ci at all, and filters on --labels amd --match-all-labels. * `amd` is added to the KNOWN_LABELS registry; ci_register rejects unregistered labels. Its PR-side trigger `run-ci-amd` still has to be created in repo Settings -> Labels by a maintainer. _run-ci-rocm.yml gains a bare `--env HIP_VISIBLE_DEVICES`, mirroring `--env CUDA_VISIBLE_DEVICES` in _run-ci.yml. The job container is a sibling started with --device=/dev/dri, so it sees all 8 GPUs regardless of the runner's device list; without this forwarding the two concurrent 4-GPU jobs would both claim every GPU. docs/ci/00-stage.md is the governing doc for pr-test-rocm.yml (# doc-dev sentinel) and moves in the same change.
Replaces the earlier blanket registration of every 8-GPU CUDA test with the 8 files selected in review on PR radixark#1606 (9300s total, down from ~10.2h). Five are already 4-GPU on CUDA and take register_rocm_ci() directly: test_qwen3_4B_fsdp_hybrid_shard_r2s2 600s test_qwen3_4B_offload_disk_stream 600s test_glm5_2_744b_a40b_5layer_ci 900s test_inkling_small_4layer_ci 1800s test_deepseek_v4_flash_4layer_ci 1900s Three are 8-GPU on CUDA and cannot run on a 4-GPU runner as written, so each gets a standalone test_amd_<name>.py with its own 4-GPU CaseConfig rather than an IS_HIP branch in the original: test_amd_r3_mtp 1100s cp 2 -> 1 test_amd_mtp1_spec_v2_r3 1600s world 8 -> 4 (dp 2 -> 1) test_amd_deepep_fp8_bridge 800s -> cp2/pp1/tp2/ep4 Each variant keeps the CUDA case's feature flags and metric gates and only narrows parallelism; the per-rank shard is unchanged in every case, so only data parallelism is given up. MI300X carries 192GB per GPU, so 4 of them hold more HBM than the 8x80GB H100 node these were tuned against. The 30B bridge case reuses the 4-GPU shape already proven for that model in test_r3_baseline.py / test_r3_deepep_fp8.py. All 8 carry the `amd` label so --labels amd --match-all-labels selects exactly this set.
guapisolo
reviewed
Aug 5, 2026
| @@ -0,0 +1,165 @@ | |||
| name: CI Job (ROCm) | |||
Collaborator
There was a problem hiding this comment.
As we have 2 amd ci runner. We'd better put them into 2 shards
Per review, the stage gains a partition_id: [0, 1] matrix and passes --auto-partition-id / --auto-partition-size 2, so each of the two 4-GPU runners takes one shard instead of the pair sitting idle behind a single job. run_suite.py balances by est_time: 4500s / 4800s, cutting wall clock from ~2.6h to ~1.3h. fail-fast: false so one shard's failure does not cancel the other mid-run, matching the CUDA stages in pr-test.yml. docs/ci/00-stage.md moves in the same change (# doc-dev sentinel).
Summarize the two-shard MI300X dispatch path and document the three MI350 suites owned by the external SGLang ROCm 7.2 nightly workflow.
Run MI300X CI from a base-branch pull_request_target gate, reusing the shared policy for PR and nightly selection while preserving manual full-suite behavior and two shards. Explicitly check out the PR merge ref, withhold WANDB_API_KEY from fork jobs, and document and test the workflow contract.
guapisolo
approved these changes
Aug 6, 2026
guapisolo
left a comment
guapisolo
left a comment
Collaborator
There was a problem hiding this comment.
approve to unblock. need amd folks check nightly CI
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add two new workflow files to support ROCm CI on AMD MI300X hardware:
_run-ci-rocm.yml: Reusable workflow (counterpart to _run-ci.yml). Uses the ROCm container with HIP/CUDA compatibility to run existing CUDA-registered test suites. Container options provide ROCm device access (kfd, dri, video/render groups), GPU readiness via rocm-smi, and pre-installed SGLang/Megatron dependencies.
pr-test-rocm.yml: Dispatch workflow matching pr-test.yml structure. resolve-ci-image → stage-c-8-gpu-mi300x calling _run-ci-rocm.yml. Same input contract (ci_megatron_pr, ci_sglang_pr, ci_image_tag). workflow_dispatch-only while the MI300X runner stabilizes.
No changes to existing files — purely additive.