[CI Monitor] Daily Report - 2026-06-28

[amd-bot]

Daily Cross-Workflow Summary — 2026-06-28

Snapshot: 2026-06-28 02:42 UTC · Only completed runs counted in trends · Auto-updated every 30 min

TL;DR

🔴 RED · ~18 active clusters · 6 🆕 today (R226–R231) · ~12 carrying over · 4 review-ready fixes still open (none merged since yesterday) · scout did not run (cron Mon/Thu)
👉 Today's ask: R225 is now 2 days unfixed and still breaks DeepSeek-V3.2 MTP on BOTH nightlies — its breaker #29413 (9214b933) is merged with no revert/guard in flight (#29499 is an adjacent optimization, not a fix). Revert or ROCm-guard it. R222 (ROCm Conv3D CUDA-only) remains the largest pr-test-amd breaker (~16 job-blocks across 5 runs, still no fix). Land the 4 review-ready fixes that did NOT move yesterday: #29376 (R214), #27141+#29391 (R195), #28889 (R192), #27757 (R2). New today: a never-passed diffusion update_weights_from_disk 500 (R226, both PR workflows) + 3 latest-run-only 8-GPU pr-test-amd regressions (R227/R228/R219) needing a rerun to separate node-flake from regression. Both release-docker workflows ✅ green. pr-test-amd-rocm720 again ≈0 clean signal (cron self-cancel + HF-429 cascade).

Workflow status

Workflow	Latest run	✅	❌	Trend (completed real failures)	Δ vs yesterday
nightly-test-amd	Jun-27 28297034445	0	~8 real (rest HF-infra)	10·10·10·~6·~8	+R229 NEW
nightly-test-amd-rocm720	Jun-27 28296988041	0	~10 real (rest HF-infra)	12·11·~5·~10	+R229,R230 NEW
release-docker-amd-nightly	Jun-27 (latest)	✅	0	0·0·0	0
release-docker-amd-rocm720-nightly	Jun-27 (latest)	✅	0	0·0·0	0
amd-aiter-scout	none (last Jun-25 28199192232)	—	—	—	no run (not Mon/Thu)
pr-test-amd	rolling Jun-27→28, latest 28306914695	0	R222+R192+R214+R226/run	worsening (+R227/R228/R219 latest)	+R226,R227,R228 NEW
pr-test-amd-rocm720	Jun-27 28297015673	0	≈0 clean (cron-cancel + HF-429)	≈0 real	+R231 confirmed

Notes: (1) amd-aiter-scout did not run (cron Mon/Thu); R221/R223 carry over dormant, no fresh data. (2) Both nightlies' Jun-27 runs are still dominated by HF weight-download hangs / 429 (infra), not code. (3) pr-test-amd-rocm720 run 28297015673 is again ≈0 clean signal: two crons share one cancel-in-progress group + an HF-429 fast-fail cascade cancels most downstream jobs.

🆕 NEW clusters today

R226 · 🆕 · Diffusion update_weights_from_disk → HTTP 500 "Inplace update to inference tensor outside InferenceMode" — pr-test-amd + pr-test-amd-rocm720

• Status: NEW 2026-06-28; never-passed (new test file test_update_weights_from_disk.py). Appears in multimodal-gen 1-GPU shard 3 across 3 pr-test-amd runs and pr-test-amd-rocm720. A secondary fixture bug (perturbed-VAE clone missing transformer dir → setup ERRORs on FLUX.2) rides in the same job.
• Top hypothesis: [LOW] server-side weight-apply path performs an in-place write on an inference-mode tensor (and a shape mismatch on FLUX.2/Qwen-Image), so every update_weights_from_disk request returns 500. Disconfirming: never-passed ⇒ could be a brand-new test exercising an unimplemented diffusion path rather than a regression. In-flight fix: ❌ none found.
• Suggested triage: confirm whether this test was newly registered (never green anywhere) vs. regressed; if new, treat as feature-gap on the diffusion weight-update endpoint and route to the diffusion owners; fix the perturbed-VAE fixture to materialize a transformer dir.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd	multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	test_update_weights_specific_modules[Qwen-Image] (+4)	500 "Inplace update to inference tensor"	Log
pr-test-amd	multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	test_update_weights_from_disk_default[Qwen-Image] (+ FLUX.2 setup ERRORs)	500 / fixture No weights dir for transformer	Log
pr-test-amd-rocm720	multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	TestUpdateWeightsFromDisk.test_update_weights_specific_modules[Qwen-Image] (+ offload variants)	assert 500 == 200	Log

R229 · 🆕 · Kimi-K2.6 8-GPU eval TIMEOUT from slow weight load (3300s+ load exhausts 3600s budget) — both nightlies

• Status: NEW 2026-06-28 (startup-bound TIMEOUT, not accuracy); regression since ~Jun 24-25. Disconfirming vs infra: this is a deterministic load-time-exceeds-budget, not a transient 429 — borderline between "model too big for budget" and a load-path slowdown.
• Top hypothesis: [LOW] weight-load wall-clock for Kimi-K2.6 (8-way) now exceeds the per-file 3600s budget (load ~3303-3359s observed). In-flight fix: ❌ none (rocm720 per-job cited #24076/#29178/#28905 as candidates, none confirmed).
• Suggested triage: bump the per-file timeout for Kimi-K2.6 OR pre-cache weights on the runner; profile load_model to see if a recent loader change slowed it.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-8-gpu-kimi-k26	test_kimi_k26_eval_amd.py	N/A (eval runner)	TIMEOUT 3600s → exit 255 (load 3303s)	Log
nightly-test-amd-rocm720	nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	test_kimi_k26_gsm8k_accuracy	TIMEOUT 3600s (load 3359s)	Log

R230 · 🆕 · DeepSeek-V4-Pro server SIGKILL (-9) during 8-way fp8 weight load (MI35x ROCm 7.2) — nightly-rocm720

• Status: NEW 2026-06-28 (first failure for these two files today; flaky). Both offline + online retry exit -9. In-flight fix: ❌ none.
• Top hypothesis: [LOW] OOM / SIGKILL during 8-way fp8 weight load (host or device memory pressure on MI35x). Disconfirming: first-seen today ⇒ may be a one-off node memory issue; needs a rerun.
• Suggested triage: rerun once; if it recurs, capture dmesg/OOM-killer logs and peak host RAM during load.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	test_deepseek_v4_pro_fp4.py	TestDeepseekV4ProFp4.setUpClass	Server exit -9 (SIGKILL on load)	Log
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	test_deepseek_v4_pro_fp4_cp.py	TestDeepseekV4ProFp4CPInterleave.setUpClass	Server exit -9 (offline+online -9)	Log

R227 / R228 / R231 · 🆕 · latest-run-only pr-test regressions (need rerun to separate flake from regression)

• R227 [LOW] — DeepSeek-R1-MXFP4 8-GPU MTP prefill GPU memory access fault → watchdog → server killed, pr-test-amd latest run only. May share a root with R225 (spec/MTP path). Suggested triage: rerun; if persistent, bisect the MTP/spec-decode window.
• R228 [LOW] — Qwen3-Coder-Next 8-GPU decode hang → scheduler watchdog 300s → SIGQUIT → connection refused, pr-test-amd latest run only. Rerun to rule out node flake.
• R231 [LOW] — torch.compile InductorError (AssertionError in post-grad decompose_triton_kernel_wrapper_functional / layernorm forward_hip) on ROCm diffusion T2V/denoising, pr-test-amd-rocm720 (was "minor-new" yesterday, now confirmed in a 2nd run). Suggested triage: a clean non-colliding rerun; if it persists, it's a real ROCm torch.compile gap, not infra.

ID	Workflow	Job (shard)	Test File	Test Function	Error	Log
R227	pr-test-amd	stage-c-8gpu-mi35x (0)	test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4MTP.test_a_gsm8k	GPU mem fault → watchdog → killed	Log
R228	pr-test-amd	stage-c-8gpu-mi35x (1)	test_qwen3_coder_next_8gpu.py	TestQwen3CoderNext.test_bs_1_speed	decode hang → SIGQUIT → conn refused	Log
R231	pr-test-amd-rocm720	multimodal-gen-2gpu (1)	test_server_2_gpu.py	test_diffusion_generation[wan2_2_t2v_a14b_2gpu …]	InductorError AssertionError	Log

Carry-over active clusters (still red)

R225 · AssertionError "All of them must not be None" in DSA eager draft-extend (dsa_backend.py:721) — both nightlies, DeepSeek-V3.2 MTP

• Status: 2 days persistent (since Jun-27); 4 jobs across both nightlies (MI35x perf + accuracy MTP). AMD/ROCm only (NVIDIA shielded by the CUDA-graph draft-extend path enabled in the same commit).
• Top hypothesis: [HIGH] breaker #29413 (9214b933, merged Jun-27 06:53) CUDA-gates the new draft-extend graph consumer (_is_cuda or _is_musa) while leaving the AMD eager init_forward_metadata assert (lines 717-722) requiring the now-nulled CPU seq-len mirror. Disconfirming: end-to-end nulling of extend_*_cpu inferred, not traced. In-flight fix: ❌ none — #29413 is merged; #29499 (open) is a DSA replay optimization, NOT a revert/guard.
• Suggested triage: revert 9214b933 on a branch + rerun test_deepseek_v32_mtp_perf_mi35x.py; if confirmed, derive extend_*_cpu from GPU tensors in the eager is_draft_extend_v2 branch or force needs_cpu_seq_lens=True on ROCm. Ping the #29413 author.

Workflow	Job (shard)	Test File	Test Function	Error	Log
nightly-test-amd	nightly-perf-8gpu-mi35x-dsv32-mtp	test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	dsa_backend.py:721 → exit -9	Log
nightly-test-amd	nightly-acc-8gpu-mi35x-dsv32-mtp	test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	same	Log
nightly-test-amd-rocm720	nightly-perf-8gpu-mi35x-dsv32-mtp-rocm720	test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	same	Log
nightly-test-amd-rocm720	nightly-acc-8gpu-mi35x-dsv32-mtp-rocm720	test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	same	Log

R222 · ROCm RuntimeError "causal Conv3D cat/pad fusion is only available on CUDA" (Wan/diffusion VAE) — pr-test-amd (largest) + rocm720

• Status: every pr-test-amd run since #29281 (merged Jun-26); ~16 job-blocks across 5 runs (28297966351, 28289294173, 28282108241, 28273478888, 28306914695) + rocm720 (83838610250 I2V/mova). Hits all Wan2.1/2.2 T2V/I2V + mova variants on 1-GPU and 2-GPU shards.
• Top hypothesis: [MEDIUM] #29281 added a CUDA-only fused causal-Conv3D fast path in WanVAE decode with no ROCm/Triton fallback. In-flight fix: ❌ none (no Conv3D-guard PR open; the Conv3D search returned only unrelated diffusion PRs).
• Suggested triage: guard the fused path behind is_cuda with an eager fallback, or revert #29281; rerun test_server_2_gpu.py::test_diffusion_generation[wan2_2_t2v_a14b_2gpu].

Representative rows (all shards share the same top frame): pr-test-amd 2gpu (1) test_server_2_gpu.py::test_diffusion_generation[wan2_2_i2v_a14b_2gpu …]; 1gpu (0) test_server_1_gpu.py::[wan2_1_t2v_1.3b_teacache_enabled …]; rocm720 2gpu (1) [mova_360p_tp2 / wan2_1_i2v_14b_480P/720P_2gpu].

ID	Cluster	Where (latest)	Status	In-flight fix
R192	FLUX.2 modelopt-FP8 torch._scaled_mm HIPBLAS_STATUS_NOT_SUPPORTED	pr-test-amd 2gpu (1) (test_server_2_gpu.py::[flux2_modelopt_fp8_tp2_t2i]); ~4 runs	never-passed	✅ #28889 open — land
R214	TokenizedGenerateReqInput missing input_embeds (TypeError)	pr-test-amd stage-b-1gpu (6) + rocm720 stage-b (6) (test_type_based_dispatcher.py)	recurring since #29214	✅ #29376 open — unblock & land
R195	Mamba extra_buffer needs CUDA/MUSA/NPU (FLA) on ROCm	nightly qwen35 83838643968, mi35x-qwen35 83838643970; rocm720 83838515323, 83838515348	persistent ≥Jun-19	✅ #27141+#29391 open — land
R19	Qwen3-235B-MXFP4 HIP hipErrorCapturedEvent capture abort	nightly 83838643963; rocm720 83838515337	never-passed ≥May-27	❌ none (per-job: #27650/#23581 candidates)
R2	Mistral/Mixtral GSM8K below threshold (chat-eval)	rocm720 83838515256 (Mistral-7B 0.361)	never-passed ≥Jun-13	✅ #27757 open — land
R211	DeepSeek-R1 HiCache MI35x — GPU mem fault during gsm8k prefill	nightly 83838643952	never-passed ≥Jun-20	❌ none
R196	VLM DP-encoder mem fault (write to read-only page)	nightly 4-gpu 83838643955 (test_encoder_dp.py::test_vlm_mmmu_benchmark)	flaky/model-dependent	⚠️ #18721 stale
R6	Qwen3-30B-A3B MoE — GPU mem fault (MI35x)	rocm720 83838515311	recurring (4/5; last pass Jun-23)	❌ none
R210	Qwen3.5 triton-DCP GSM8K 0.556<0.90	nightly mi35x-qwen35 83838643970 (test_qwen3p5_triton_dcp.py)	never-passed	⚠️ #29230 DNM
R219	DeepSeek-V3.2 (basic) 8-GPU HSA out-of-resources decode abort	pr-test-amd stage-c-8gpu (1) (test_deepseek_v32_basic.py::test_a_gsm8k)	latest-run flake	❌ none

Known stable / dormant clusters (no action today) · click to expand

ID	Cluster	Where	Status	Fix
R1	VLM MMMU accuracy below threshold	nightly (today masked by MMMU dataset/429 timeouts)	never-passed ≥Jun-13	❌ none
R155	DeepSeek-V3.2 (basic) MI35x GSM8K below threshold	rocm720 (today masked by xet download timeout)	never-passed on rocm720	⚠️ partial #25559/#29050
R213	MiniMax-M2.7 GSM8K borderline	nightly	borderline/flaky	❌ none
R220	Embeddings-API latency threshold	pr-test-amd stage-b-1gpu-large	flake (not seen today)	❌ none
R221	aiter-caused GPU Hang (exit 134) ROCm 7.2 LoRA	scout only — no run today	dormant	❌ none
R223	aiter-caused DSV4-Pro-MTP connection-refused	scout only — no run today	dormant	❌ none
R212/R224	DSV3.2-MTP perf hang / eval borderline	superseded by R225 on MTP jobs	dormant	❌ none

Infrastructure / orchestration noise (not test failures) · click to expand

• HF weight-download hangs / 429 Too Many Requests / Xet xet_get stalls: dominate both nightlies Jun-27 — nightly-amd (DeepSeek-R1-MXFP4-tp2, GLM-5.1-mxfp4, gpt-oss-120b tokenizer filelock, perf-vlm Qwen3-VL-30B 429, MMMU dataset timeout) and nightly-rocm720 (DSR1-mxfp4-tp4, DSV3-0324, gpt-oss-120b, Grok-2, Qwen3-235B, DSV3.2 xet timeout, VLM 429). Partial fix #23400 open.
• pr-test-amd-rocm720 cron self-cancel + HF-429 cascade: run 28297015673 — two crons share one cancel-in-progress group; HF-429 on stage-a/multimodal warmup → fast-fail cancels most downstream jobs. ≈0 clean pytest signal.
• pr-test-amd diffusion port-5555 --strict-ports cascade: HF-download timeout on first diffusion test leaks scheduler port 5555 → cascades the rest of the 1-GPU shard (cosmos3/wan/lingbot/qwen-image). Many multimodal-gen-1gpu rows.
• ROCm VRAM-not-clear / zombie KFD pre-flight gate: nightly glm5-mxfp4 83838644036, rocm720 hicache 83838515342. Node reboot required.
• mori build / git-clone network fail: pr-test-amd 83799696814 (corrupt libabsl_base.so invalid ELF).
• Kimi-K2-MXFP4 BCG watchdog -9 (pr-test-amd 83799697008): flaky (1/6) MoE weight-load watchdog timeout.

Workflow drill-down (per-workflow view)

nightly-test-amd · Jun-27 [28297034445](https://github.com/sgl-project/sglang/actions/runs/28297034445) · ~8 real (rest HF-infra)

Job (shard)	Test File	Test Function	Cluster	Error
nightly-perf-8gpu-mi35x-dsv32-mtp	test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	R225	DSA assert → exit -9
nightly-acc-8gpu-mi35x-dsv32-mtp	test_deepseek_v32_mtp_eval_mi35x.py	setUpClass	R225	DSA assert
nightly-8-gpu-qwen35	test_qwen35_eval_amd.py	setUpClass	R195	extra_buffer assert
nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	R195	extra_buffer assert
nightly-8-gpu-mi35x-qwen35	test_qwen3p5_triton_dcp.py	test_a_gsm8k	R210	gsm8k 0.556<0.90
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	R19	HIP capture -6
nightly-8-gpu-mi35x-deepseek-r1-hicache	test_deepseek_r1_hicache_mi35x.py	test_gsm8k	R211	GPU mem fault
nightly-4-gpu	test_encoder_dp.py	test_vlm_mmmu_benchmark	R196	write to read-only page -9
nightly-8-gpu-kimi-k26	test_kimi_k26_eval_amd.py	N/A	R229🆕	TIMEOUT 3600s (load 3303s)
(dsr1-mxfp4-tp2, glm51-mxfp4, gpt-oss-120b, perf-vlm 429, mmmu, glm5-mxfp4 VRAM gate)	various	—	infra	HF download / 429 / VRAM gate

nightly-test-amd-rocm720 · Jun-27 [28296988041](https://github.com/sgl-project/sglang/actions/runs/28296988041) · ~10 real (rest HF-infra)

Job (shard)	Test File	Test Function	Cluster	Error
nightly-perf-8gpu-mi35x-dsv32-mtp-rocm720	test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	R225	DSA assert
nightly-acc-8gpu-mi35x-dsv32-mtp-rocm720	test_deepseek_v32_mtp_eval_mi35x.py	setUpClass	R225	DSA assert
nightly-8-gpu-qwen35-rocm720	test_qwen35_eval_amd.py	setUpClass	R195	extra_buffer assert
nightly-8-gpu-mi35x-qwen35-rocm720	test_qwen35_eval_mi35x.py	test_lm_eval	R195	extra_buffer assert
nightly-acc-8gpu-mi35x-rocm720	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	R6	GPU mem fault -6
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	setUpClass	R19	hipErrorCapturedEvent
nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral)	R2	gsm8k 0.361
nightly-8-gpu-kimi-k26-rocm720	test_kimi_k26_eval_amd.py	test_kimi_k26_gsm8k_accuracy	R229🆕	TIMEOUT (load 3359s)
nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	test_deepseek_v4_pro_fp4.py / _cp.py	setUpClass	R230🆕	server exit -9 (load)
(dsr1-mxfp4-tp4, dsv3.1, dsv3.2 xet, grok2, qwen3-235b, gpt-oss, vlm-429, hicache VRAM gate)	various	—	infra	HF download / 429 / VRAM gate

pr-test-amd · rolling Jun-27→28 · latest [28306914695](https://github.com/sgl-project/sglang/actions/runs/28306914695)

Job (shard)	Test File	Test Function	Cluster	Error
multimodal-gen-2gpu (1) ×~16 blocks/5 runs	test_server_{1,2}_gpu.py	test_diffusion_generation[wan2_*]	R222	causal Conv3D CUDA-only
multimodal-gen-2gpu (1)	test_server_2_gpu.py	[flux2_modelopt_fp8_tp2_t2i]	R192	HIPBLAS_STATUS_NOT_SUPPORTED
multimodal-gen-1gpu (3)	test_update_weights_from_disk.py	test_update_weights_specific_modules	R226🆕	500 inplace-on-inference tensor
stage-b-1gpu-small (6)	test_type_based_dispatcher.py	test_type_dispatcher_e2e_performance	R214	TypeError (input_embeds)
stage-c-8gpu-mi35x (0)	test_deepseek_r1_mxfp4_8gpu.py	test_a_gsm8k	R227🆕	GPU mem fault (MTP)
stage-c-8gpu-mi35x (1)	test_qwen3_coder_next_8gpu.py	test_bs_1_speed	R228🆕	decode hang → SIGQUIT
stage-c-8gpu (1)	test_deepseek_v32_basic.py	test_a_gsm8k	R219	HSA out-of-resources
(diffusion port-5555 cascades, mori build, kimi-mxfp4 watchdog)	various	—	infra	downloads / cascades

pr-test-amd-rocm720 · Jun-27 [28297015673](https://github.com/sgl-project/sglang/actions/runs/28297015673) · ≈0 clean signal (cron-cancel + HF-429)

Real signal buried under cron cancel-in-progress self-cancellation + HF-429 fast-fail cascade: R226 (test_update_weights_from_disk.py 500, 1gpu (3) 83838610262), R214 (test_type_based_dispatcher.py, stage-b (6) 83838610271), R231 (InductorError, 2gpu (1) 83838610250 + 1gpu (0) 83838610266), R222 (causal Conv3D I2V/mova, same 2gpu job). Plus a perf-threshold miss (stage-b-1gpu-large (0) 83838610248) and a test_start_profile_2 watchdog/CUDA-graph-replay stall (stage-b (10) 83838610294) — both inconclusive without a clean rerun. Needs a non-colliding rerun for usable signal.

How this report is generated

• Only status == "completed" runs counted in trends. Both nightlies' Jun-27 runs treated as completed. Both release-docker workflows ✅ green; amd-aiter-scout did not run (cron Mon/Thu).
• 🆕 NEW today: R226 (diffusion update_weights 500), R227 (DSR1-MXFP4 8-GPU MTP mem fault), R228 (Qwen3-Coder-Next 8-GPU hang), R229 (Kimi-K2.6 weight-load timeout, both nightlies), R230 (DSV4-Pro fp8 -9, rocm720), R231 (ROCm torch.compile InductorError, confirmed from yesterday's minor-new).
• Carrying over: R225 now 2 days unfixed (breaker #29413 merged, no revert in flight); R222/R195/R214/R192/R2/R19/R211/R196/R6/R210/R219.
• In-flight fixes unchanged since yesterday (none merged): #29376 (R214), #27141+#29391 (R195), #28889 (R192), #27757 (R2).
• Confidence: FACT/HIGH/MEDIUM/LOW/SPECULATION. Bot does NOT assign Priority — engineers decide from cluster size + persistence + fix availability.

---

Generated by amd-bot · last updated 2026-06-28 02:42 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:42 UTC)

---

CI Monitor — 2026-06-28

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the Daily Cross-Workflow Summary is rendered above this section.

[amd-bot]

nightly-test-amd.yml — 15 failure(s)

Run started (UTC): 2026-06-27 17:49 UTC
Last scanned (UTC): 2026-06-28 00:43 UTC
Commits: sglang 592f6c8

Summary

Counts: 16 failures · 12 clusters · 7 carrying over (≥3 days) · 7 new today · 2 other (1 flaky, 1 two-night)

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HF weight-download stall → launch timeout	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	new today (passed Jun 24–26)	[MEDIUM] HF Xet download stall (infra)
2	HF weight-download stall → launch timeout	nightly-2-gpu-mi35x-glm51-mxfp4	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	new today (passed Jun 24–26)	[MEDIUM] HF Xet download stall (infra)
3	HF weight-download stall → launch timeout	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	setUpClass	new today	[MEDIUM] HF Xet download stall (infra)
4	Mamba extra_buffer unsupported on ROCm	nightly-8-gpu-qwen35	test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	setUpClass	8 days, Never-passed	[FACT] auto→extra_buffer asserts CUDA/MUSA/NPU; in-flight #27141
5	Mamba extra_buffer unsupported on ROCm	nightly-8-gpu-mi35x-qwen35	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	4 days, Never-passed	[HIGH] same extra_buffer assert; in-flight #27141
6	DSA draft-extend metadata assertion	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	2 nights (since Jun 26)	[HIGH] extend_prefix_lens_cpu not populated in prepare_for_draft_extend
7	DSA draft-extend metadata assertion	nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	new today (regression)	[HIGH] commit 9214b933 (#29413) DSA draft-extend graph
8	GPU mem fault during HiCache gsm8k prefill	nightly-8-gpu-mi35x-deepseek-r1-hicache	test/registered/amd/test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x.test_gsm8k	8 days, Never-passed	[LOW] HiCache kernel/aiter prefill OOB (no stack trace)
9	HIP graph-capture abort in aiter custom all-reduce	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	8 days, Never-passed	[MEDIUM] aiter AR _gather_ipc_meta illegal in capturing stream; in-flight #27650/#23581
10	Kimi-K2.6 eval timeout from slow weight load	nightly-8-gpu-kimi-k26	test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	N/A (whole-file eval)	4 days (since Jun 24)	[MEDIUM] weight-load time 2× (1556→3303 s), infra I/O
11	Qwen3.5 triton+DCP GSM8K accuracy < 0.90	nightly-8-gpu-mi35x-qwen35	test/registered/amd/test_qwen3p5_triton_dcp.py	TestQwen35TritonDCPGsm8k.test_a_gsm8k	4 days, Never-passed	[LOW] ~0.56–0.65 score, DCP/rope or threshold
12	MMMU dataset stall → VLM accuracy suite timeout	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	N/A (13-model suite)	recurring; new TIMEOUT mode today	[HIGH] ~1h52m dataset-download stall (infra)
13	GPU mem fault (RO-page write) DP-encoder mm forward	nightly-4-gpu	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	flaky (Jun 25,27 fail; 24,26 pass)	[LOW] DP-encoder embed scatter race / MI30x node
14	HF tokenizer-download filelock hang	nightly-accuracy-8-gpu	test/registered/amd/accuracy/mi30x/test_gpt_oss_eval_amd.py	test_gpt_oss_accuracy	new today (passed Jun 24–26)	[MEDIUM] 8 TP ranks contend on WeakFileLock (gpt-oss-120b)
15	HF Hub 429 rate limit during weight download	nightly-perf-2-gpu-vlm	test/registered/amd/perf/mi30x/test_vlms_perf_amd.py	test_bench_one_batch	new today (passed Jun 24–26)	[FACT] HF 429 → server exit -9 (infra)
16	Pre-test VRAM cleanup failure (zombie KFD)	nightly-8-gpu-mi35x-glm5-mxfp4	N/A (step 3, no test ran)	N/A	new today (infra)	[HIGH] zombie KFD context holds VRAM, needs reboot

Cluster: HF weight-download stall → launch timeout (rows 1–3)

• Affected: rows 1–3 — DeepSeek-R1-MXFP4 + GLM-5.1-MXFP4 (×2) all setUpClass timeouts on MI35x.
• Shared facts: every py-spy dump stuck in download_weights_from_hf → xet_get; fresh cold download; same-run docker-registry network timeout; all passed the prior 3 nights.
• Top hypothesis: [MEDIUM] HF Xet download/network stall consumed the full launch budget; disconfirming: no explicit HTTP/connection error printed — slow-vs-hung inferred from single snapshots.
• In-flight fix: ❌ none.
• Suggested triage: re-run; confirm warm cache vs cold; if recurring try HF_HUB_DISABLE_XET=1 / pre-warm MI35x cache.

Cluster: Mamba extra_buffer unsupported on ROCm (rows 4–5)

• Shared facts: AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA) at server_args.py:4500; auto strategy selects extra_buffer for Qwen3.5 hybrid; ROCm fails the guard. Never-passed on AMD.
• Top hypothesis: [FACT]/[HIGH] platform-support gap; disconfirming: (none) — deterministic platform guard, not a commit regression.
• In-flight fix: ✅ #27141 (ROCm extra_buffer support) + #29391 (auto→no_buffer fallback).
• Suggested triage: validate #27141/#29391; interim --mamba-radix-cache-strategy no_buffer.

Cluster: DSA draft-extend metadata assertion (rows 6–7)

• Shared facts: all 8 ranks crash AssertionError: All of them must not be None at dsa_backend.py:721 via eagle_worker_v2._draft_extend_for_decode → base_spec_worker.prepare_for_draft_extend; extend_prefix_lens_cpu never populated. Both MI35x DeepSeek-V3.2 MTP jobs.
• Top hypothesis: [HIGH] DSA draft-extend graph commit 9214b933 (#29413) — in perf regression window c98d31143d..592f6c849b, modifies the exact failing files; disconfirming: accuracy job failed Jun 26 with a different (hang) symptom, no revert A/B run.
• In-flight fix: ⚠️ candidate #29202 (AMD MTP draft-extend) — verify before duplicating.
• Suggested triage: git revert 9214b933 + re-run perf test; or populate extend_prefix_lens_cpu in prepare_for_draft_extend gpu_only branch.

Singleton clusters (rows 8–16)

• Row 8 HiCache mem fault: [LOW] latent HiCache/aiter prefill OOB; no stack trace, no green baseline (8 days). In-flight ⚠️ #20611 (FA3-specific, low relevance). Triage: A/B hicache on/off + capture HIP coredump.
• Row 9 aiter AR HIP capture: [MEDIUM] custom_all_reduce.py:338 _gather_ipc_meta illegal in capturing stream. In-flight ✅ #27650/#23581. Triage: re-run SGLANG_USE_AITER_AR=0.
• Row 10 Kimi timeout: [MEDIUM] weight-load doubled, no relevant loader commit in window; likely infra I/O. Triage: check Load weight end elapsed + runner storage.
• Row 11 Qwen3.5 triton+DCP accuracy: [LOW] stable ~0.6 GSM8K vs 0.90; ❌ no fix. Triage: A/B --dcp-size 1 and yarn rope; compare CUDA baseline / threshold.
• Row 12 MMMU timeout: [HIGH] ~1h52m dataset-generation stall, only 1/13 models ran; chronic sub-threshold accuracy is a separate [FACT] issue. ❌ no fix. Triage: warm dataset cache + per-model timeout.
• Row 13 DP-encoder mem fault: [LOW] flaky write-to-RO-page in mm_utils.embed_mm_inputs, model-independent. Candidate #26759 to evaluate. Triage: re-run N×, A/B --mm-enable-dp-encoder.
• Row 14 gpt-oss-120b tokenizer hang: [MEDIUM] 8 ranks contend on HF WeakFileLock; 20b passed same run. ❌ no fix. Triage: pre-warm tokenizer / HF_HUB_OFFLINE=1.
• Row 15 HF 429: [FACT] rate limit during weight enum; transient. Triage: re-run; pre-stage weights or HF_TOKEN.
• Row 16 VRAM/KFD: [HIGH] zombie KFD context on linux-mi35x-gpu-8-zf775-runner-bj4nh; no test ran. Triage: reboot runner ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Revert 9214b933 (#29413) on a branch, re-run both V3.2-MTP tests	rows 6–7	Only [HIGH]-confidence fresh sglang regression with a named commit; 2 jobs
2	Validate/merge #27141 (+#29391)	rows 4–5	In-flight fix exists for an 8-day Never-passed platform gap; 2 jobs
3	Re-run the 4 HF-download jobs; warm caches	rows 1–3, 14, 15	5 new-today rows are likely one transient HF/network event; cheap to confirm
4	Reboot runner ...zf775...bj4nh + re-run	row 16	Pure infra; node reboot unblocks the job immediately
5	Validate #27650/#23581; A/B SGLANG_USE_AITER_AR=0	row 9	8-day Never-passed with in-flight candidate fixes

---

Job	Failed Steps	Started
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2)	2026-06-27T17:49
nightly-2-gpu-mi35x-glm51-mxfp4	Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)	2026-06-27T17:49
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-06-27T18:00
nightly-8-gpu-mi35x-deepseek-r1-hicache	Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)	2026-06-27T18:56
nightly-8-gpu-kimi-k26	Accuracy Test (8-GPU Kimi-K2.6)	2026-06-27T18:30
nightly-4-gpu	Nightly Test (4-GPU)	2026-06-27T17:49
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-06-27T19:41
nightly-8-gpu-qwen35	Accuracy Test (8-GPU Qwen 3.5)	2026-06-27T18:48
nightly-accuracy-8-gpu	Accuracy Test (8-GPU GPT-OSS)	2026-06-27T18:29
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-06-27T21:57
nightly-perf-2-gpu-vlm	Performance Test (2-GPU VLM Models)	2026-06-27T18:24
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)	2026-06-27T20:36
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-06-27T22:36
nightly-8-gpu-mi35x-glm5-mxfp4	Ensure VRAM is clear	2026-06-27T22:34
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-06-27T18:07

Per-Job Analysis

nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 — failed step(s): Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: Dev/patched version detected in CI image, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The single test in job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 failed in setUpClass because the SGLang server never became ready within the 3600 s launch timeout (SERVER_LAUNCH_TIMEOUT = 3600, test file line 37; actual elapsed = 3626 s). The server process hung during model-weight download for amd/DeepSeek-R1-MXFP4-Preview and was killed after the timeout. No accuracy/GSM8K assertion was ever reached.

Regression Status

New failure today for this test file — but symptoms point to an infrastructure (download) hang, not a code regression.

Recent history of test_deepseek_r1_mxfp4_tp2_mi35x.py (setUpClass) in job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (branch=main, event=schedule, completed runs only). This job runs only this one test file, so job conclusion == test-file status:

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	❌ Failed	setUpClass	TimeoutError (server launch)
Jun 26	28256335319	same	✅ Passed	—	—
Jun 25	28191022667	same	✅ Passed	—	—
Jun 24	28119497313	same	✅ Passed	—	—

First observed failure: 2026-06-27. Last known passing: 2026-06-26. (Other test files in the broader nightly-test-amd.yml workflow fail independently; this verdict is scoped to this test file only.)

Failure Cluster

HuggingFace weight-download hang during server warmup (HF Xet backend xet_get stall → server launch timeout)

Facts (verified observations, no interpretation)

• Verbatim error (log line ~6271-6275): File ".../test_deepseek_r1_mxfp4_tp2_mi35x.py", line 125, in setUpClass → TimeoutError: Server failed to start within the timeout period; FAILED (errors=1); returned exit code 1; elapsed=3626, estimated_time=1800.0; runner step Process completed with exit code 255.
• TP1 (PID 3636) acquired the download lock at 17:55:54: [CI Download] Process 3636 ACQUIRED lock for amd/DeepSeek-R1-MXFP4-Preview.
• TP1 active thread is stuck in the HF download (py-spy at 18:00:44, ThreadPoolExecutor-2_0, active):
  xet_get (huggingface_hub/file_download.py:566) → _download_to_tmp_and_move → hf_hub_download → snapshot_download → ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → download_weights_from_hf (weight_utils.py:593) → … → load_model (model_runner.py:1443).
• TP0 (PID 3635) is blocked waiting on the same lock: _poll_until_acquired (filelock/_api.py:552) → download_weights_from_hf (weight_utils.py:554). TP0 cannot proceed because TP1 holds the lock and never finishes.
• Scheduler/TokenizerManager watchdogs fired soft-timeout repeatedly (300 s) from 18:00 onward; the main sglang serve process sat in _wait_for_scheduler_ready (engine.py:1376) the entire time.
• Network flakiness earlier in the same run (log line 1334): Local docker registry pull failed; falling back to public registry … context deadline exceeded.
• aiter version unchanged vs the 3 prior passing runs (dev image v0.1.14rc1.dev240+g7d604afe5, no override, no rebuild).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure: the HF Xet weight download for amd/DeepSeek-R1-MXFP4-Preview stalled on the network and never completed, so the server never finished load_model within 3600 s.
  • Supporting: the only active thread in TP1 is xet_get (the actual byte-transfer), not compute; TP0 is merely blocked on TP1's lock; same runner already showed a network timeout pulling the docker image earlier in this run; the test passed on the 3 immediately preceding nightly runs with no code change to the download path implicated.
  • Disconfirming: cannot definitively rule out a slow/corrupt cache or a regression in ci_weight_validation.py retry logic without the full download-progress lines; py-spy only gives a single point-in-time snapshot, so "stalled" vs "very slow but progressing" is inferred, not proven.
  • How to verify: re-run the job; if it passes with the weights already cached, the hang was a transient download. If it recurs, check download_weights_from_hf / ci_weight_validation.py retry+timeout behavior and whether HF_HUB_DISABLE_XET=1 avoids the xet_get stall.
• [LOW] A code change in the weight-download/validation path (ci_weight_validation.py, weight_utils.py) regressed download timeout/retry handling between Jun 26 and Jun 27.
  • Supporting: failure is "new today"; the stack passes through ci_download_with_validation_and_retry.
  • Disconfirming: no evidence any commit in the window touched that path; symptom is a network-bound xet_get stall, which is environmental; a single hung snapshot is equally consistent with pure infra.
  • How to verify: git log --oneline c98d31143d..592f6c849b -- python/sglang/srt/model_loader/ci_weight_validation.py python/sglang/srt/model_loader/weight_utils.py and inspect any hits.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet download (3 results, all unrelated: #28584, #29178, #28905) and weight download hang (4 results, all unrelated/sandbox PRs). No PR addresses this download-hang symptom.

Suggested Triage Steps

• Re-run the job once: if it passes (weights now cached on the runner), classify as a transient HF-Xet download stall and close.
• If it recurs, try HF_HUB_DISABLE_XET=1 (or pin the non-Xet download path) for this runner to confirm the Xet backend is the culprit.
• Inspect git log --oneline c98d31143d..592f6c849b -- python/sglang/srt/model_loader/ci_weight_validation.py python/sglang/srt/model_loader/weight_utils.py to rule out a download-path regression in the window.
• Consider a per-file download timeout/retry in ci_download_with_validation_and_retry so a stalled shard fails fast instead of consuming the full 3600 s server-launch budget.

Status

• Persistence: 1 run (first failure 2026-06-27 17:49 UTC); passed the 3 prior nightly runs (Jun 24-26).
• Scope: this job only (nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2), 1 test file. The hang is in the shared HF weight-download path, so other models on the same runner could be affected if it's a runner/network issue.
• Blocked work: DeepSeek-R1-MXFP4 TP2 accuracy signal on MI35x for this nightly.
• In-flight fix: ❌ none found in open PRs.

nightly-2-gpu-mi35x-glm51-mxfp4 — failed step(s): Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The server for amd/GLM-5.1-MXFP4 (TP=2, DSA + tilelang, fp8_e4m3 KV) never finished starting within the test's 5400 s (SERVER_LAUNCH_TIMEOUT) window, so setUpClass raised TimeoutError and 0 tests ran (Ran 0 tests in 5410.716s). Across the entire ~90-minute window the scheduler stayed blocked in HuggingFace weight download: TP1 held the download lock and was actively downloading, while TP0 waited for that lock — neither ever progressed to weight loading or warmup.

Regression Status

New failure today (Jun 27) — job passed the previous 3 nights.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (TestGLM51MXFP4TP2GSM8KMI35x) in job nightly-2-gpu-mi35x-glm51-mxfp4 (nightly-test-amd.yml, branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-2-gpu-mi35x-glm51-mxfp4	❌ Failed	setUpClass	TimeoutError: Server failed to start
Jun 26	28256335319	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—
Jun 25	28191022667	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—
Jun 24	28119497313	nightly-2-gpu-mi35x-glm51-mxfp4	✅ Passed	—	—

First observed failure: Jun 27 2026. Last known pass: Jun 26 2026.

Failure Cluster

Server launch timeout — stuck in HuggingFace weight download (filelock poll / fresh model download)

Facts (verified observations, no interpretation)

• 17:55:51 TP1 and TP0: Load weight begin. avail mem=279.75 GB.
• 17:55:51 TP1: [CI Download] Process 3633 ACQUIRED lock for amd/GLM-5.1-MXFP4 (a fresh download under /sgl-data/hf-cache/hub/, not a cache hit).
• Every pyspy dump from 18:00:38 through 19:20:45 (≈15 dumps, 5-min apart) shows TP0 (PID 3632) identically stuck:
  • _poll_until_acquired (filelock/_api.py:552) ← download_weights_from_hf (model_loader/weight_utils.py:554) ← do_load_weights (deepseek_common/deepseek_weight_loader.py:158) ← load_weights (deepseek_v2.py:2851) ← load_model (model_runner.py:1443).
• TP1 (PID 3633) in the same dumps is at download_weights_from_hf (model_loader/weight_utils.py:593) (the active download path); its native pyspy repeatedly returns Pyspy failed ... failed to get os threadid (consistent with being inside native download/xet code).
• Soft watchdog (watchdog_timeout=300, soft=True) fired repeatedly but did not kill the process; the hard cap was the test's SERVER_LAUNCH_TIMEOUT=5400.
• Final: Ran 0 tests in 5410.716s → TimeoutError: Server failed to start within the timeout period (test_utils.py:1028 popen_launch_server).
• Job step 6 ("Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K)") exited 255; "Exit code 255 indicates test failure (not network issue), not retrying".
• AITER unchanged (dev image preserved); not an aiter-scout run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] HuggingFace model-download stall / very slow network for amd/GLM-5.1-MXFP4 weights — the entire 90-min budget was spent in the download path, never reaching weight load or warmup.
  • Supporting: pyspy shows the process is literally inside download_weights_from_hf for the whole window; TP1 ACQUIRED the lock and is in the active download frame (weight_utils.py:593) the entire time; this was a fresh download ([CI Download] ... ACQUIRED lock), unlike a normal cache-hit start; the same job passed the prior 3 nights (when weights were presumably already cached).
  • Disconfirming: no explicit ConnectionError/Read timed out/HTTP error was emitted in the log (the download neither errored nor completed — it simply did not finish in time), so a pure "network down" cannot be confirmed; possible the file is just large and bandwidth was degraded.
  • How to verify: re-run the job and watch whether amd/GLM-5.1-MXFP4 is served from /sgl-data/hf-cache (cache hit → fast) vs re-downloaded; check whether the runner's HF cache was evicted or the upstream repo was re-published (changed revision forcing re-download) between Jun 26 and Jun 27.
• [LOW] HF cache eviction on this MI35x runner forced a cold re-download that didn't fit in the window.
  • Supporting: prior passing nights almost certainly hit a warm cache (no ACQUIRED lock/fresh-download pattern needed); today shows a cold download.
  • Disconfirming: cannot confirm cache state of prior runs without pulling their logs.
  • How to verify: diff the "Load weight begin" → "Load weight end" elapsed in the Jun 26 passing job's log vs today.
• [SPECULATION] filelock deadlock (TP0 waiting on a lock TP1 will never release).
  • Supporting: TP0 is in _poll_until_acquired.
  • Disconfirming: TP1 genuinely holds the lock and is in the active download frame (not idle/dead), so this is expected single-downloader behavior, not a deadlock — TP0 would proceed once TP1's download completes. No hypothesis here above LOW for a code bug.

No hypothesis reaches HIGH; this is most likely an infrastructure/download event rather than an sglang code regression (no relevant code change isolated; modified code path is the standard HF download, unchanged in the failing window).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for GLM-5.1-MXFP4 (5 results, all feature/quant work, none about download/launch timeout) and download weights timeout (14 results, none addressing this model-download stall). API calls: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+GLM-5.1-MXFP4 and ...+download+weights+timeout.

Suggested Triage Steps

• Re-run nightly-2-gpu-mi35x-glm51-mxfp4 and confirm whether amd/GLM-5.1-MXFP4 resolves from the runner's warm HF cache (/sgl-data/hf-cache/hub); a green re-run would confirm a transient download/network event.
• Compare "Load weight begin → end" elapsed in the Jun 26 passing job 83720226592 vs today to confirm the prior run hit a warm cache while today did a cold download.
• Check whether the amd/GLM-5.1-MXFP4 HF repo was re-published / revision changed around Jun 27 (would force a cold re-download on all runners).
• If cold downloads are expected to recur, consider pre-warming the model into the MI35x runner cache, or raising SERVER_LAUNCH_TIMEOUT / separating download from launch — owner's call.

Status

• Persistence: 1 run (new today, Jun 27 2026 17:49 UTC); passed the prior 3 consecutive nights.
• Scope: 1 job / 1 test (nightly-2-gpu-mi35x-glm51-mxfp4) in nightly-test-amd.yml; not observed in sibling jobs.
• Blocked work: GLM-5.1-MXFP4 TP=2 GSM8K accuracy signal on MI35x for this night only.
• In-flight fix: ❌ none found in open PRs.

nightly-accuracy-2-gpu — failed step(s): Nightly Test (2-GPU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: dev image — v0.1.14rc1.dev240+g7d604afe5 detected in CI image, rebuild skipped; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_mxfp4_tp2_gsm8k_mi35x.py	TestGLM51MXFP4TP2GSM8KMI35x.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The nightly-accuracy-2-gpu job ran a single test, test_glm51_mxfp4_tp2_gsm8k_mi35x.py, on an mi35x (gfx950) runner. setUpClass called popen_launch_server for amd/GLM-5.1-MXFP4 (TP=2) with a 5400 s launch timeout. The server never became ready within 5400 s and raised TimeoutError. Ran 0 tests in 5410.716s — the test body never executed.

Regression Status

Flaky / infrastructure (model-download-bound). This test file did not run in the previous three nightly-accuracy-2-gpu jobs — those landed on mi30x runners and ran test_gsm8k_eval_amd.py instead (the accuracy-2-gpu runner pool is arch-dynamic). The correct comparison is the dedicated sister job nightly-2-gpu-mi35x-glm51-mxfp4, which runs the exact same test file and passed the prior 3 nights. In this run BOTH the dedicated job and nightly-accuracy-2-gpu ran the test and BOTH failed with the identical TimeoutError.

Recent history of test_glm51_mxfp4_tp2_gsm8k_mi35x.py (setUpClass), tracked via the dedicated sister job nightly-2-gpu-mi35x-glm51-mxfp4 (same branch main, same event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-2-gpu-mi35x-glm51-mxfp4 (job)	❌ Failed	setUpClass	TimeoutError (server launch)
Jun 27	28297034445	nightly-accuracy-2-gpu (job)	❌ Failed	setUpClass	TimeoutError (server launch)
Jun 26	28256335319	nightly-2-gpu-mi35x-glm51-mxfp4 (job)	✅ Passed	—	—
Jun 25	28191022667	nightly-2-gpu-mi35x-glm51-mxfp4 (job)	✅ Passed	—	—
Jun 24	28119497313	nightly-2-gpu-mi35x-glm51-mxfp4 (job)	✅ Passed	—	—

First observed failure for this test file: 2026-06-27. Last known passing: 2026-06-26.

Failure Cluster

GLM-5.1-MXFP4 server launch timeout — stuck in HF weight download (Xet backend)

Facts (verified observations, no interpretation)

• Both schedulers (TP0 and TP1) were captured by py-spy idle inside the weight download path in every dump: download_weights_from_hf (weight_utils.py:593/554) → snapshot_download (_snapshot_download.py:467) → hf_hub_download (file_download.py:1016) → _download_to_tmp_and_move (file_download.py:1861) → xet_get (file_download.py:566). The model-load call chain is model_runner.load_model → loader._prepare_weights → deepseek_weight_loader.do_load_weights (GLM-5.1 uses the DeepSeek-V2 loader). (verbatim from log lines 7094–7140, 7211–7336, 7400–7518)
• The soft watchdog (soft_watchdog_timeout=300, soft=True) fired repeatedly during download but did not kill the process: 51 watchdog timeout lines in the dedicated job, vs 15 in the Jun-26 passing run.
• Total elapsed before setUpClass raised: Ran 0 tests in 5410.716s; harness recorded elapsed=5428, estimated_time=3600.0. Launch timeout is SERVER_LAUNCH_TIMEOUT = 5400 (test file line 33).
• The Jun-26 passing run on the same dedicated job also stalled in download_weights_from_hf with 15 soft-watchdog dumps — i.e. download was already slow but completed under the deadline. (verified in job 83720226592 log)
• Independent network signal on the same runner: Local docker registry pull failed; falling back to public registry (log line 1333–1334), and Pyspy failed ... failed to get os threadid during dumps.
• No Load weight end / server-ready line ever appears in the failing job; the model never finished loading.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Network/infrastructure-bound HuggingFace weight download (Xet backend) for amd/GLM-5.1-MXFP4 exceeded the 5400 s server-launch timeout on this mi35x runner.
  • Supporting: every py-spy dump shows both schedulers blocked in xet_get/hf_hub_download; no progress past weight download; the immediately-prior passing run on the same job exhibited the same stall but finished (15 vs 51 watchdog dumps); independent docker-registry network failure on the same runner; weights are downloaded fresh each run (download_dir=None, no local cache hit observed).
  • Disconfirming: I did not capture per-file MiB/s throughput numbers from the log, so a server-side hang (vs pure slowness) cannot be 100% excluded; both jobs in this run failed identically, which is also consistent with a transient HF/Xet outage at run time rather than a single-runner issue.
  • How to verify: re-run the dedicated nightly-2-gpu-mi35x-glm51-mxfp4 job; check whether weights are pre-cached on the runner; inspect HF Xet download throughput; if it passes on rerun with no code change, infra is confirmed.
• [LOW] sglang code regression in the GLM-5.1/DeepSeek-V2 weight loader between c98d31143d (Jun-26 pass) and 592f6c849b (Jun-27 fail).
  • Supporting: the failure newly appears at this SHA; HEAD commit 592f6c84 ("Split 2/4: mem-cache / HiCache / sparse KV pool", #28713) touches mem-cache/KV-pool code.
  • Disconfirming: the stall is entirely in huggingface_hub's xet_get (network download), which runs before any sglang model construction or KV-pool allocation — the changed code paths are not on the blocked stack. Strongly weakens a code-regression hypothesis.
  • How to verify: git log --oneline c98d31143d..592f6c849b -- python/sglang/srt/model_loader/ python/sglang/srt/models/deepseek_v2.py and check for download/loader changes; none expected.
• [LOW] aiter regression.
  • Supporting: AMD-only path.
  • Disconfirming: aiter version is unchanged from the Dockerfile default (7d604afe5, dev image preserved) and the failure occurs in HF download before any aiter kernel runs.

No hypothesis reaches HIGH confidence; the dominant evidence points to a model-download/infra flake, not code.

In-flight Fix Check

• In-flight fix: ❌ none found — search repo:sgl-project/sglang is:pr is:open GLM-5.1-MXFP4 download returned 0 results; broader model weight cache timeout search surfaced no PR addressing GLM-5.1-MXFP4 download caching or the 5400 s launch timeout.

Suggested Triage Steps

• Re-run the dedicated job nightly-2-gpu-mi35x-glm51-mxfp4 (job) — if it passes with no code change, this is confirmed as a transient download/infra flake.
• Check whether amd/GLM-5.1-MXFP4 weights are pre-staged/cached on the mi35x runner pool; if downloaded fresh each night, consider a persistent HF cache or local mirror to remove the Xet-download dependency from the launch path.
• Inspect HF Xet download throughput in the log window 18:14→19:25; compare against the Jun-26 passing run to quantify slowdown.
• If confirmed infra-bound, consider raising SERVER_LAUNCH_TIMEOUT (test line 33) or making the soft watchdog surface download progress — current behavior silently retries for 90 min then fails opaquely.
• Only if a rerun reproduces with cached weights: git log --oneline c98d31143d..592f6c849b -- python/sglang/srt/model_loader/ to inspect loader changes.

Status

• Persistence: First failure 2026-06-27; the dedicated sister job passed the 3 prior nights (Jun 24–26). 1 night failing so far.
• Scope: 2 jobs in this run (nightly-accuracy-2-gpu + nightly-2-gpu-mi35x-glm51-mxfp4), both on mi35x, both the same test_glm51_mxfp4_tp2_gsm8k_mi35x.py setUpClass timeout.
• Blocked work: GLM-5.1-MXFP4 TP=2 GSM8K accuracy gate produced no signal this night (0 tests ran).
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-mi35x-deepseek-r1-hicache — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache)

Commit Info

• sglang: 592f6c84 — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev/patched image preserved ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_r1_hicache_mi35x.py	TestDeepSeekR1HiCacheMI35x.test_gsm8k	GPU Memory access fault on all 8 nodes during gsm8k prefill → server dead → TIMEOUT after 3600s → exit 255	Log

Failure Summary

The DeepSeek-R1-0528 server launches and passes its health check with hierarchical cache enabled (--enable-hierarchical-cache --hicache-io-backend kernel --hicache-mem-layout page_first --hicache-write-policy write_through --hicache-storage-backend file). When test_gsm8k begins and the first batched prefill of concurrent requests runs (#new-token: 768, #queue-req: 20), all 8 GPUs simultaneously raise Memory access fault ... Reason: Unknown. The server processes die, the harness hangs, and the test is killed by the 3600s timeout (exit code 255). No accuracy assertion is ever reached.

Regression Status

Known recurring failure / Never-passed in queryable window — the same test file + function fails with the identical symptom in every completed scheduled run back to at least Jun 20. No green run found in the lookback window (Jun 20–27).

Recent history of test_deepseek_r1_hicache_mi35x.py (TestDeepSeekR1HiCacheMI35x.test_gsm8k) in job nightly-8-gpu-mi35x-deepseek-r1-hicache (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 27	28297034445	❌ Failed	test_gsm8k	Mem access fault (8 GPUs) → 3600s timeout
Jun 26	28256335319	❌ Failed	test_gsm8k	Mem access fault → 3600s timeout
Jun 25	28191022667	❌ Failed	test_gsm8k	Mem access fault → 3600s timeout
Jun 24	28119497313	❌ Failed	test_gsm8k	Mem access fault → 3600s timeout
Jun 23	28046906309	❌ Failed	test_gsm8k	Mem access fault → 3600s timeout
Jun 22	27975987522	❌ Failed	test_gsm8k	Watchdog timeout (300s soft) → 3600s timeout
Jun 21	27912890405	❌ Failed	test_gsm8k	Mem access fault → 3600s timeout
Jun 20	27879266774	❌ Failed	test_gsm8k	Mem access fault → 3600s timeout

First observed failure in window: Jun 20 (earliest queried); last known passing date: none found in lookback range Jun 20–27.

Failure Cluster

GPU memory access fault during HiCache gsm8k prefill (DeepSeek-R1 MI35x)

Facts (verified observations)

• [FACT] Server reaches healthy state: GET /health_generate HTTP/1.1" 200 OK at 19:40:06 (log line 21404), and Tree cache initialized: ... impl=HiRadixCache ... hierarchical=True at 19:39:57 (line 21311).
• [FACT] Fault fires during the second prefill batch of the gsm8k run — Prefill batch, #new-seq: 1, #new-token: 768, #cached-token: 0, ... #queue-req: 20 (line 21462, 19:40:12) immediately followed by Memory access fault by GPU node-2..9 ... Reason: Unknown (lines 21463–21470, 19:40:13).
• [FACT] Fault hits all 8 TP ranks within ~25ms, consistent with a collective/TP-wide kernel rather than a single-rank issue.
• [FACT] Same (test_file, test_function) fails with the same Memory access fault → 3600s timeout symptom in 7 of the last 8 completed scheduled runs; the Jun 22 run shows a soft watchdog-timeout variant of the same hang.
• [FACT] Failure predates HEAD commit 592f6c84 (merged Jun 27 17:19 UTC) — the identical fault occurred Jun 20–26, so this commit is the run's HEAD but cannot be the root cause.
• [FACT] An earlier soft watchdog dump at 19:35:47 (lines 20917–21178) occurs inside decode_cuda_graph_runner.py:690 capture (102s CUDA graph capture) and recovers (Capture target decode CUDA graph end. elapsed=102.02 s at 19:36:36) — this is not the fatal event.
• [FACT] aiter unchanged across runs (dev image v0.1.14rc1.dev240+g7d604afe5 preserved; no override).

Hypothesised Causes (with confidence)

• [LOW] Latent fault in the HiCache kernel I/O backend / page_first layout KV transfer path triggered by the first multi-request batched prefill under hierarchical cache.
  • Supporting: fault occurs only once batched gsm8k prefill begins (#queue-req: 20) on a hicache-enabled server; the job is the only one exercising this specific hicache flag combination on MI35x; symptom is a GPU memory access fault (out-of-bounds access), consistent with a KV-transfer/indexing bug.
  • Disconfirming: no stack trace pins a hicache kernel — the fault is Reason: Unknown with no Python traceback; the fault is TP-wide (could equally be the aiter attention/MoE prefill kernel path that runs at the same point); no green baseline exists to bound a regression window, so this could be a long-standing model/runner issue unrelated to hicache code.
  • How to verify: re-run the same command with --enable-hierarchical-cache removed (and/or --attention-backend swapped) on MI35x in container rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260627 to A/B isolate whether hicache is necessary to reproduce.
• [LOW] aiter fused-MoE / blockscale prefill kernel out-of-bounds at M:31065 prefill shape.
  • Supporting: the last kernel log before the fault is [aiter] [fused_moe] using 1stage ... fmoe_bf16_blockscaleFp8_g1u1_novs_silu_1tg_ps_32x256 at M:31065 (line ~21462 region).
  • Disconfirming: log-line proximity is not causation; aiter version is unchanged across all 8 failing runs, so an unchanged aiter cannot newly "cause" a persistent failure; other (non-hicache) MI35x DSR1 jobs using the same aiter pass.
  • How to verify: compare against the sister non-hicache DSR1 MI35x accuracy job (test_deepseek_r1_eval_mi35x.py referenced in the test header) on the same SHA — if it passes, aiter is exonerated and the hicache path is implicated.

No hypothesis reaches MEDIUM: there is no Python stack trace and no green run to bound a regression window. Needs an A/B repro (hicache on/off) to localize.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #20611 "[HiCache] Fix illegal memory access caused by stream race between FA3 forward and D2H transfer" — same symptom family (hicache illegal memory access) but targets the FA3 forward path, while this job runs --attention-backend aiter; treat as [LOW] relevance until confirmed. No open PR found specifically matching DeepSeek-R1 hicache MI35x memory-access-fault.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+hicache+memory+access+fault and ...+deepseek+hicache.

Suggested Triage Steps

• A/B isolate hicache: re-run the exact sglang serve command from the log with --enable-hierarchical-cache (and the dependent --hicache-* flags) removed, on MI35x in rocm/sgl-dev:v0.5.14-rocm700-mi35x-20260627. If the fault disappears, the hicache transfer/layout path is implicated.
• A/B isolate aiter: compare with the sister non-hicache DSR1 MI35x accuracy job on the same SHA; if it passes, aiter is exonerated.
• Capture a coredump/backtrace: re-run with AMD_LOG_LEVEL/HIP fault-reporting and --watchdog-timeout lowered so the process is dumped at fault time instead of hanging 3600s; the current logs have no stack trace to localize the faulting kernel.
• Since no green run exists in the window, extend history backward (pre Jun 20) or treat this as a never-passed AMD hicache port and disable/quarantine test_deepseek_r1_hicache_mi35x.py on MI35x until a backtrace is obtained.

Status

• Persistence: Failing in every completed scheduled run from Jun 20 through Jun 27 (8 consecutive runs); no passing run in the queryable window.
• Scope: 1 job (nightly-8-gpu-mi35x-deepseek-r1-hicache) in nightly-test-amd.yml. Blocks the DeepSeek-R1 HiCache accuracy signal on MI35x.
• Blocked work: HiCache gsm8k accuracy coverage for DeepSeek-R1 on MI35x.
• In-flight fix: ⚠️ only loosely-related #20611 (FA3-specific) found; no direct fix — needs new investigation with a backtrace.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:43 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/3)

nightly-8-gpu-kimi-k26 — failed step(s): Accuracy Test (8-GPU Kimi-K2.6)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev image — Dev/patched version detected → skipping rebuild, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	(whole-file eval runner; not a pytest function)	TIMEOUT after 3600 seconds → exit code 255	Log

Failure Summary

The 8-GPU Kimi-K2.6 accuracy eval (test_kimi_k26_eval_amd.py) hit the harness's 3600 s wall-clock timeout and the process was killed (exit 255). Weight loading alone consumed 3303 s (Load weight end. elapsed=3303.02 s), leaving only ~5 minutes of the budget for the eval, which had reached 72 % (944/1319) when it was terminated. The model and eval were otherwise functioning normally (no crash, no accuracy assertion).

Regression Status

New regression (started 2026-06-24). This job (nightly-8-gpu-kimi-k26) last passed on 2026-06-23 with accuracy 0.942 and a total runtime of 1793 s (weight load 1556 s). It has failed with the same timeout symptom every scheduled run since.

Recent history of test_kimi_k26_eval_amd.py in job nightly-8-gpu-kimi-k26 (main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Detail
Jun 27	28297034445	job	❌ Failed	TIMEOUT; weight load 3303 s, eval reached 72%
Jun 26	28256335319	job	❌ Failed	TIMEOUT; never reached Load weight end
Jun 25	28191022667	job	❌ Failed	TIMEOUT; never reached Load weight end
Jun 24	28119497313	job	❌ Failed	TIMEOUT; never reached Load weight end
Jun 23	28046906309	job	✅ Passed	acc 0.942, total 1793 s, weight load 1556 s

First observed failure: 2026-06-24. Last known passing: 2026-06-23.

Failure Cluster

Kimi-K2.6 8-GPU eval timeout from slow weight loading (3600s budget exhausted during model load)

Facts (verified observations)

• ✗ TIMEOUT: .../test_kimi_k26_eval_amd.py after 3600 seconds then First attempt failed with exit code 255 / ##[error]Process completed with exit code 255 (log lines 66330, 66345–66347).
• Today: Load weight begin at 18:44:11, Load weight end. elapsed=3303.02 s at 19:39:14 (line 52111). Eval then ran fast (~30–40 it/s) and reached 72%|███████▏ | 944/1319 (line 66321) before kill_process_tree at 19:42:44.
• Last passing run (Jun 23): Load weight end. elapsed=1555.72 s, Test Summary: 1/1 passed, "passed": true, "elapsed": 1793, Accuracy: 0.942. Weight-load time roughly doubled (1556 s → 3303 s).
• Jun 24/25/26 failing logs end at Load weight begin and never reach Load weight end — those runs timed out during loading (even slower than today).
• Infra signal today: Local docker registry pull failed; falling back to public registry … context deadline exceeded (lines 1357–1358) — network/registry trouble on runner linux-mi325-8gpu-...-st2ht.
• The Jun 23→24 window is 55 commits (ed26a109ee...d6aacd2801). The only loader files touched are under python/sglang/multimodal_gen/runtime/loader/** — unrelated to the Kimi text-LLM compressed-tensors weight path.
• aiter version is identical (dev image) across passing and failing runs ([CI-AITER-CHECK] markers) — not an aiter change.

Hypothesised Causes (with confidence)

• [MEDIUM] Weight-load I/O / runner throughput regression (infrastructure) — load time more than doubled (1556 s→3303 s) with no relevant loader code change in the 55-commit window.
  • Supporting: regression is entirely in Load weight elapsed; eval throughput itself is healthy; today's run shows a docker-registry pull failure / network deadline on the same runner; prior 3 runs were even slower (never finished loading), consistent with variable/degraded storage or checkpoint-fetch bandwidth.
  • Disconfirming: the failure has been consistent for 4 straight days, which is more persistent than typical transient infra flakiness; could instead be a stable change (e.g. checkpoint shards re-uploaded larger on HF, or a slow code path now on by default).
  • How to verify: re-run the job and compare Load weight end. elapsed; check disk/network read throughput on the st2ht runner; confirm the moonshotai/Kimi-K2.6 checkpoint size/shard count is unchanged vs Jun 23.
• [LOW] A commit in ed26a109ee...d6aacd2801 slowed the compressed-tensors weight load path (e.g. 76db6c9d9e shared CUDA graph mem pool, or a memory/cache change).
  • Supporting: the regression boundary aligns exactly with this 55-commit window.
  • Disconfirming: none of the 55 commits touch the Kimi LLM weight loader; only multimodal_gen loaders changed, which this text model does not use.
  • How to verify: bisect the window with a load-time-only probe (start server, measure Load weight end elapsed, no eval) on an MI325 8-GPU runner.
• [LOW] Eval budget genuinely too tight even at normal speed — at 1556 s load + ~240 s eval (Jun 23) the margin to 3600 s was already ~1800 s; any load slowdown eats it.
  • Supporting: today's eval would likely have passed (72% in ~3 min) given more time.
  • Disconfirming: the test passed comfortably on Jun 23, so the budget is adequate at baseline load speed.
  • How to verify: not a root cause on its own; only relevant if the load slowdown proves to be the permanent new normal.

In-flight Fix Check

• In-flight fix: ❌ none found targeting the eval timeout / load slowdown. Open Kimi-K2.6 PRs found are about correctness/quantization, not load time: #28905 (compressed-tensors MoE w2 scale loading), #29178 (load_full_w2 for grouped w2 scales), #23381 (MI355X tuning artifacts). (Searched is:pr is:open with keywords kimi k26 timeout and Kimi-K2.6 load.)

Suggested Triage Steps

• Re-run nightly-8-gpu-kimi-k26 and read Load weight end. elapsed — if it returns to ~1556 s the failure is transient infra; if it stays ~3300 s it's a stable regression.
• Inspect storage/network health on runner linux-mi325-8gpu-sglang-lmwqt-runner-st2ht (the docker-registry deadline at lines 1357–1358 suggests degraded I/O); confirm the moonshotai/Kimi-K2.6 checkpoint shard size/count is unchanged since Jun 23.
• If load time is stably ~2× higher, bisect ed26a109ee..d6aacd2801 using a load-time-only probe (no eval) to isolate any code contribution.
• As a stop-gap so the eval signal is not lost to load time, consider raising the per-test timeout above 3600 s for this 8-GPU Kimi-K2.6 job (maintainer decision — masks rather than fixes the slowdown).

Status

• Persistence: Failing 4 consecutive scheduled runs since 2026-06-24 (~4 days); last green 2026-06-23.
• Scope: nightly-8-gpu-kimi-k26 in nightly-test-amd.yml. The sibling nightly-8-gpu-mi35x-kimi-k26 passed Jun 26 but failed Jun 24/25, so the slow-load symptom may be intermittent across runners.
• Blocked work: No Kimi-K2.6 accuracy signal on AMD MI30x nightly since Jun 23.
• In-flight fix: ❌ none found for the timeout/load-slowdown.

nightly-4-gpu — failed step(s): Nightly Test (4-GPU)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: dev image, equals Dockerfile default; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	Server process exited with code -9 after Memory access fault by GPU node-2 ... Write access to a read-only page (models Qwen/Qwen2.5-VL-72B-Instruct and zai-org/GLM-4.1V-9B-Thinking)	Log

Failure Summary

The single test file test/registered/vlm/test_encoder_dp.py (DP-encoder VLM MMMU benchmark, launched with --mm-enable-dp-encoder --tp=4) failed. For both VLM models exercised this run, the server came up, completed CUDA-graph capture, then crashed on the first multimodal forward pass with a GPU Memory access fault ... Write access to a read-only page. The scheduler then hung (every /health_generate returned 503 for ~5 min) until the process was SIGKILLed (exit -9); the test retry budget was exhausted (retry() exceed maximum number of retries).

Regression Status

Flaky / intermittent — same test file alternates pass/fail day-to-day with an identical symptom across different VLM models.

Recent history of test/registered/vlm/test_encoder_dp.py (test_vlm_mmmu_benchmark) in job nightly-4-gpu (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	mem access fault → exit -9 (Qwen2.5-VL-72B, GLM-4.1V-9B)
Jun 26	28256335319	nightly-4-gpu	✅ Passed (377s)	—	—
Jun 25	28191022667	nightly-4-gpu	❌ Failed	test_vlm_mmmu_benchmark	same mem access fault → exit -9 (Qwen3-VL-32B)
Jun 24	28119497313	nightly-4-gpu	✅ Passed	—	—
Jun 23	28046906309	nightly-4-gpu	✅ Passed	—	—

First observed failure (this window): Jun 25. Last known pass: Jun 26. The fault reproduces under the default aiter image (no override), so it is not aiter-override-induced.

Failure Cluster

GPU memory access fault (write to read-only page) on first multimodal forward in the DP-encoder embedding path.

Facts (verified observations, no interpretation)

• Verbatim crash (3× per server, once per TP worker): Memory access fault by GPU node-2 (Agent handle: ...) on address 0x...: Reason: Write access to a read-only page. (job log lines ~5992, ~6998).
• Both models crash at the identical Python stack (py-spy dump, sync point at .item()):
  • _adjust_embedding_length (mm_utils.py:702) ← get_embedding_and_mask (mm_utils.py:778) ← embed_mm_inputs (mm_utils.py:855) ← general_mm_embed_routine (mm_utils.py:1085) ← forward (qwen2_5_vl.py:758) / forward (glm4v.py:674) ← model_runner.py:3098 ← run_batch (scheduler.py:3252).
• mm_utils.py:702 is num_mm_tokens_in_input_ids = mask.sum().item() — a device→host sync; it is the surfacing point of a deferred async fault from an earlier multimodal/vision-encoder kernel, not itself the faulting code.
• Config: --mm-enable-dp-encoder enabled (mm_enable_dp_encoder=True), tp_size=4, attention_backend='aiter', multimodal attention backend aiter_attn. IPC threads MmItemMemoryPoolRecycler / cuda_ipc_transport_utils.py:160 present.
• After the fault the scheduler thread stays active inside libhsa-runtime64/libamdhip64 (memcpy_and_sync from _local_scalar_dense_cuda) and never recovers; health checks 503 for ~5 min, then SIGKILL.
• Same fault, same stack, different model on Jun 25 (Qwen/Qwen3-VL-32B-Instruct) — confirms model-independence.
• Secondary (non-causal) observation this run: TP3 was blocked ~26 min on the HF download filelock (download_weights_from_hf → filelock/_api.py:552) for Qwen2.5-VL-72B, producing repeated soft watchdog timeouts (soft=True). This only slowed startup; the crash occurred later, after a successful load + graph capture.
• aiter version unchanged across the passing (Jun 26) and failing (Jun 25/27) runs per [CI-AITER-CHECK] (v0.1.14rc1.dev240+g7d604afe5).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Nondeterministic out-of-bounds / read-only write in the DP-encoder multimodal embedding scatter path (mm_utils.embed_mm_inputs / get_embedding_and_mask under --mm-enable-dp-encoder + cross-rank IPC buffers).
  • Supporting: identical top-of-stack across two unrelated VLM architectures; "write to a read-only page" is consistent with writing into a protected/IPC-shared buffer; intermittency fits a race in the DP encoder gather/scatter.
  • Disconfirming: no code-level diff identified; the same file passes on adjacent days with the same head image; could equally be runner/GPU-node flakiness on MI30x node-2.
  • How to verify: re-run test_encoder_dp.py N× on the same MI30x runner; then re-run with --mm-enable-dp-encoder removed (single-encoder path) to A/B whether the DP-encoder path is required to trigger the fault.
• [LOW] Hardware/runner instability on the specific MI30x node (GPU node-2).
  • Supporting: pass/fail alternates with no code change in the image; fault address differs each occurrence.
  • Disconfirming: the crash always lands in the same multimodal code path, which is more specific than generic hardware flake would predict.
  • How to verify: check whether other 4-gpu test files on the same runner/node show unrelated faults in the failing runs; pin the job to a different MI30x node and re-run.

No hypothesis reaches MEDIUM; needs an A/B repro (DP-encoder on/off) on the MI30x runner to localize.

In-flight Fix Check

• In-flight fix: ❌ none found clearly matching this fault. Searched open PRs for memory access fault encoder, dp encoder multimodal, test_encoder_dp.
• Possibly-adjacent (verify relevance, do not assume it fixes this): #26759 "[Fix] Keep DP-sharded mrope vision tensors on the compute device" — touches the DP vision-tensor device-placement path; worth checking against this stack before opening a new PR.

Suggested Triage Steps

• Re-run test/registered/vlm/test_encoder_dp.py ~5× on the same MI30x runner to quantify the flake rate and confirm intermittency.
• A/B the DP-encoder path: re-run with vs without --mm-enable-dp-encoder (same models, same node). If the fault only appears with DP encoder, focus on mm_utils.embed_mm_inputs/get_embedding_and_mask + the IPC scatter (utils/cuda_ipc_transport_utils.py).
• Confirm whether #26759 touches a tensor in this stack; if so, cherry-pick onto a branch and re-run the test to test the hypothesis.
• Capture a HIP coredump (set CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1 — the log notes it was disabled) to get the faulting kernel rather than only the .item() sync frame.

Status (factual, no priority assignment)

• Persistence: Intermittent — failed Jun 25 and Jun 27, passed Jun 24 and Jun 26 (and Jun 23) in the queryable window; not a continuous regression.
• Scope: 1 test file (test_encoder_dp.py), 1 job (nightly-4-gpu), 1 workflow (nightly-test-amd.yml). Reproduces under default aiter; model-independent (≥3 VLMs affected).
• Blocked work: DP-encoder VLM MMMU CI signal on AMD MI30x is unreliable.
• In-flight fix: ❌ none found; #26759 is a candidate to evaluate, not a confirmed fix.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev/patched image ([CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → server crash during HIP graph capture	Log

Failure Summary

The Qwen3-235B-A22B MXFP4 server (TP=4, EP=2, --attention-backend aiter, SGLANG_USE_AITER=1) crashes during setUpClass while initializing the decode CUDA/HIP graph. The scheduler aborts (SIGABRT, exit -6) so popen_launch_server raises Server process exited with code 1. No accuracy/throughput test runs — the failure is at server startup.

Regression Status

Known recurring / Never-passed (no green run found in queryable window). The identical signature (_gather_ipc_meta → HIP operation not permitted ... capturing stream, scheduler exit -6) appears in every completed scheduled run from 2026-06-20 through 2026-06-27.

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 27	run	❌ Failed	setUpClass	HIP capture error, exit -6
Jun 26	run	❌ Failed	setUpClass	same
Jun 25	run	❌ Failed	setUpClass	same
Jun 24	run	❌ Failed	setUpClass	same
Jun 23	run	❌ Failed	setUpClass	same
Jun 22	run	❌ Failed	setUpClass	same
Jun 21	run	❌ Failed	setUpClass	same
Jun 20	run	❌ Failed	setUpClass	same

Lookback window: 2026-06-20 → 2026-06-27 (8 completed runs); older runs (Jun 16–19) were cancelled, no comparable data. No passing run found for this test file in the window.

Failure Cluster

HIP graph-capture abort in aiter custom all-reduce (_gather_ipc_meta) during decode CUDA graph init.

Facts (verified observations)

• Verbatim error: HIP error: operation not permitted on an event last recorded in a capturing stream and ... operation not permitted when stream is capturing (hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported), raised from ProcessGroupNCCL.cpp:2055 watchdog and c10_hip_check_implementation (HIPException.cpp:45).
• Crash stack (from log, step 6):
  • aiter/dist/device_communicators/custom_all_reduce.py:338 _gather_ipc_meta ← :305 flush_graph_buffers ← :505 capture
  • sglang/srt/distributed/parallel_state.py:539 graph_capture ← :1779 graph_capture
  • sglang/srt/model_executor/runner/decode_cuda_graph_runner.py:687 capture ← :353 __init__
  • model_runner.py:2608 init_decode_cuda_graph ← scheduler.py:861 init_all_cuda_graphs ← :440 __init__
• Result: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) → Server process exited with code 1 → FAILED (errors=1).
• Reproducibility: 8/8 completed scheduled runs over 8 days, same test file + function, same stack.
• Config: TP=4, EP=2, SGLANG_USE_AITER=1, --attention-backend aiter, 8-GPU MI35x (gfx950).
• AITER version unchanged across passing-search window (dev image 7d604afe5 is the Dockerfile default for gfx950) — so this is not driven by an aiter version bump in CI.

Hypothesised Causes (with confidence)

• [MEDIUM] The fault originates in aiter custom all-reduce IPC-meta gather during HIP graph capture (_gather_ipc_meta/flush_graph_buffers), reached via sglang's graph_capture path when SGLANG_USE_AITER=1 enables the aiter AR. The capturing-stream HIP error means an event/IPC operation that is illegal mid-capture is being issued inside the capture region.
  • Supporting: top non-torch stack frame is aiter custom_all_reduce.py:338; error is a stream-capture-context violation; deterministic across all runs; several open PRs target exactly this aiter-AR-under-graph-capture interaction (see below).
  • Disconfirming: the failing frame is in aiter (/sgl-workspace/aiter), not modifiable from sglang; the test has never passed in the queryable window, so no sglang commit "introduced" a regression here — it may be an unsupported config for this model/GPU combo since the test landed.
  • How to verify: relaunch the model with SGLANG_USE_AITER_AR=0 (or --disable-cuda-graph) on MI35x in the CI container; if capture succeeds, the aiter AR path is confirmed as the trigger.
• [LOW] Newly-added test exercises an unsupported MXFP4 + aiter-AR + EP config on gfx950.
  • Supporting: test file first added 2026-06-25 (efbe67d2); never observed green; failure predates the file's own merge into this branch (job already failing Jun 20 in the suite), suggesting the path was broken before/at introduction.
  • Disconfirming: cannot confirm the suite/model config history beyond the 8-run window.
  • How to verify: check whether any historical run of this suite ever passed before the lookback window.

In-flight Fix Check

Open PRs directly targeting this aiter-AR / HIP-graph-capture failure class (keywords: custom all-reduce graph capture, aiter AR):

• In-flight fix: ✅ #27650 "[AMD] Fix PCG capture fail with aiter fused all-reduce+RMSNorm" (open since 2026-06-09) — most directly named at "PCG capture fail with aiter ... all-reduce".
• In-flight fix: ✅ #23581 "[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation" (open since 2026-04-23) — would disable the exact code path in the stack.
• In-flight fix: ✅ #28672 / #27948 "Skip custom all-reduce v2 CUDA graph capture with torch memory saver".
• In-flight fix: ✅ #27835 "[bugfix][AMD] Disable aiter allreduce+RMSNorm fusion under DP attention / EP".

No need to open a duplicate — validate one of the above against this test.

Suggested Triage Steps

• Reproduce on MI35x in the CI container with SGLANG_USE_AITER_AR=0 for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (TP=4, EP=2); if it passes, confirms the aiter-AR-under-capture trigger.
• Check whether #27650 or #23581 applies cleanly, then re-run test/registered/amd/test_qwen3_instruct_mxfp4.py on the 8-GPU MI35x suite.
• If neither fix covers this config, consider gating this nightly test (or its AR usage) on AMD until the aiter graph-capture path is fixed, since it has never produced a green signal.
• File/track the aiter-side issue against aiter/dist/device_communicators/custom_all_reduce.py:_gather_ipc_meta performing an illegal event op inside a capturing stream.

Status

• Persistence: Failing every completed scheduled run for 8 days (since at least 2026-06-20); never observed passing in the queryable window.
• Scope: This job only (nightly-8-gpu-mi35x-qwen3-235b-mxfp4); the failure is specific to the aiter AR + HIP graph capture path for this MXFP4 model on gfx950.
• Blocked work: No nightly accuracy/throughput signal for Qwen3-235B-A22B MXFP4 on MI35x.
• In-flight fix: ✅ candidate fixes exist — #27650, #23581 (plus #28672/#27835); none confirmed against this test yet.

nightly-8-gpu-qwen35 — failed step(s): Accuracy Test (8-GPU Qwen 3.5)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — v0.1.14rc1.dev240+g7d604afe5 detected, rebuild skipped; matches Dockerfile default AITER_COMMIT=7d604afe5...)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server exits code 1 during launch)	Log

Failure Summary

The server for Qwen/Qwen3.5-397B-A17B failed to launch during setUpClass, so 0 tests ran. The launch command (sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter ... --device cuda) aborted with AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). raised in server_args.py:4500. This is an AMD ROCm runner (mi30x), where the mamba extra-buffer code path is gated to CUDA/MUSA/NPU only.

Regression Status

Known recurring failure / likely Never-passed on AMD.

Recent history of test_qwen35_eval_amd.py (TestQwen35EvalAMD.setUpClass) in job nightly-8-gpu-qwen35 (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-8-gpu-qwen35	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU (FLA).
Jun 26	28256335319	nightly-8-gpu-qwen35	❌ Failed	setUpClass	same assertion
Jun 25	28191022667	nightly-8-gpu-qwen35	❌ Failed	setUpClass	same assertion (verified)

Lookback window: 8 scheduled runs Jun 20–27 (28297034445 back to 27879266774) — the nightly-8-gpu-qwen35 job failed in every one. No passing run for this test file found in the queryable window, and the failure is a platform-level guard (not a flake), so it is effectively Never-passed on AMD.

Failure Cluster

Mamba extra_buffer unsupported on ROCm — server launch assertion

Facts (verified observations, no interpretation)

• Verbatim error: AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). at python/sglang/srt/server_args.py:4500 (log line 4377).
• Call chain in stack trace: from_cli_args → __post_init__ → _handle_model_specific_adjustments → _handle_mamba_radix_cache (server_args.py:4529) → _validate_mamba_extra_buffer (server_args.py:4500).
• The failing assert is assert is_cuda() or is_musa() or is_npu(), "extra_buffer needs CUDA/MUSA/NPU (FLA).". On AMD ROCm all three are False (ROCm reports is_hip).
• Launch command: sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter --trust-remote-code --model-loader-extra-config '{"enable_multithread_load": true}' --watchdog-timeout 1200 --device cuda (log line 4379).
• Qwen/Qwen3.5-397B-A17B is a hybrid Mamba model (arch Qwen3_5MoeForConditionalGeneration), which is in the _support_mamba_cache_extra_buffer allow-list, so the auto mamba-radix strategy selects extra_buffer (server_args.py:_handle_mamba_radix_cache).
• aiter unchanged across runs (dev image v0.1.14rc1.dev240+g7d604afe5, per [CI-AITER-CHECK] markers, log lines 4164-4170) — not an aiter-version issue.
• Same assertion confirmed in the Jun 26 and Jun 25 runs' logs (identical message + same test file).

Hypothesised Causes (with confidence)

• [FACT] The auto mamba-radix-cache strategy selects extra_buffer for the Qwen3.5 hybrid model, but _validate_mamba_extra_buffer hard-requires CUDA/MUSA/NPU; on ROCm the assertion fails and the server aborts.
  • Supporting: assertion text and code at server_args.py:4500 directly gate on is_cuda() or is_musa() or is_npu(); arch is in the extra_buffer allow-list; runner is mi30x (ROCm). Reproduced in every scheduled run in the window.
  • Disconfirming: (none found) — the failure is deterministic and platform-level, not dependent on a specific commit in the window.
  • How to verify: launch the same command on an AMD MI30x container; confirm assertion. Confirm that forcing --mamba-radix-cache-strategy no_buffer (or --disable-radix-cache) avoids it.

This is a platform-support gap, not a code regression introduced by a commit in a pass→fail window (no green baseline exists for this test on AMD).

In-flight Fix Check

• In-flight fix: ✅ #27141 — "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" — directly targets this gap; chase reviewers, do NOT open a duplicate.
• Related: #29391 "Fix mamba radix cache auto strategy crashing on non-FLA platforms (MPS/MLX)" — same auto-strategy crash class on non-FLA platforms; may also cover ROCm.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+extra_buffer+no_buffer and ...+mamba+extra_buffer+ROCm.

Suggested Triage Steps

• Confirm whether #27141 (ROCm extra_buffer support) or #29391 (auto non-FLA fallback) resolves the AMD path; if so, prioritize merge and re-run nightly-8-gpu-qwen35.
• As an interim test-side workaround, consider passing --mamba-radix-cache-strategy no_buffer (the code's own suggested fallback: extra_buffer ... use no_buffer) or --disable-radix-cache in test_qwen35_eval_amd.py:setUpClass until ROCm extra_buffer lands — maintainer's call.
• Until a fix merges, this AMD nightly Qwen3.5 eval cannot produce accuracy signal.

Status (factual)

• Persistence: Failing in every completed scheduled run in the lookback window (Jun 20–27, 8/8 runs); no green run found for this test file.
• Scope: nightly-8-gpu-qwen35 (and sibling nightly-8-gpu-mi35x-qwen35 also fails) in nightly-test-amd.yml. Platform-specific to AMD ROCm; not reproducible on CUDA where the assert passes.
• Blocked work: Qwen3.5 accuracy/eval CI signal on AMD mi30x.
• In-flight fix: ✅ #27141 (ROCm extra_buffer support); also #29391 (auto non-FLA fallback).

nightly-accuracy-8-gpu — failed step(s): Accuracy Test (8-GPU GPT-OSS)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved; AITER_COMMIT_OVERRIDE empty, Dockerfile default also 7d604afe5, so no rebuild — verified from [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gpt_oss_eval_amd.py	test_gpt_oss_accuracy	AssertionError: Failed models: ['lmsys/gpt-oss-120b-bf16']	Log

Failure Summary

The test_gpt_oss_accuracy test runs a GSM8K accuracy benchmark against two models. lmsys/gpt-oss-20b-bf16 passed (accuracy=0.515, threshold 0.45). lmsys/gpt-oss-120b-bf16 (threshold 0.75, tp=8, launch timeout=900) never produced an accuracy number — its server failed to become ready within the 900s launch window, so the per-model try raised and the test aggregated it into the Failed models assertion.

Regression Status

New failure today (for this test file in this job). The nightly-accuracy-8-gpu job ran and passed the previous three scheduled nights; today's workflow-level red is the first time this job itself failed in the queryable window.

Recent history of test_gpt_oss_eval_amd.py (test_gpt_oss_accuracy) in job nightly-accuracy-8-gpu (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-accuracy-8-gpu	❌ Failed	test_gpt_oss_accuracy	Failed models: ['gpt-oss-120b-bf16']
Jun 26	28256335319	nightly-accuracy-8-gpu	✅ Passed	—	—
Jun 25	28191022667	nightly-accuracy-8-gpu	✅ Passed	—	—
Jun 24	28119497313	nightly-accuracy-8-gpu	✅ Passed	—	—

First observed failure for this test file: 2026-06-27. Last known passing: 2026-06-26 (run 28256335319).

Failure Cluster

HuggingFace tokenizer-download filelock hang during server launch (gpt-oss-120b, tp=8)

Facts (verified observations, no interpretation)

• accuracy=0.515 threshold=0.45 ✅ PASS for gpt-oss-20b-bf16; only one accuracy= line in the whole log — gpt-oss-120b-bf16 produced no accuracy value (it errored at launch, not on a low score).
• The 120b server launched at 18:40:20. From 18:40:30 onward, all 8 TP scheduler processes (TP0–TP7) were idle in the identical native stack, repeated at the 18:45 / 18:50 / 18:55 py-spy dumps:

  acquire (filelock/_api.py:506)
  WeakFileLock (huggingface_hub/utils/_fixes.py:99)
  _hf_hub_download_to_cache_dir (huggingface_hub/file_download.py:1237)
  hf_hub_download (huggingface_hub/file_download.py:1016)
  cached_file (transformers/utils/hub.py:278)
  from_pretrained (transformers/tokenization_utils_base.py:1712)
  get_tokenizer (utils/hf_transformers/tokenizer.py:552)
  init_tokenizer (scheduler.py:705)
  

• Repeated Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) for every TP rank at 18:45, 18:50, 18:55 — the schedulers never progressed past tokenizer init.
• Server launch used HF_HUB_OFFLINE=0 (CI_OFFLINE: Launching server HF_HUB_OFFLINE=0 model=lmsys/gpt-oss-120b-bf16) — i.e. the tokenizer was being fetched from the HF Hub, not served purely from local cache.
• Earlier infra signal: at 18:29:36 the local docker registry mirror was unreachable — [local-pull] Error response from daemon: Get "https://10.245.143.50:5000/v2/": ... Client.Timeout exceeded while awaiting headers (log line 1358).
• Per-model launch timeout=900 (test config line 77); the 900s elapsed (18:40:20 → 18:55:10) → Ran 1 test in 1059.325s, FAILED (failures=1), exit 255. The runner explicitly did not retry: "Exit code 255 indicates test failure (not network issue), not retrying".
• The same run's 20b model passed, so GPUs, aiter kernels, and the MoE/attention path are functional this run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure / HF Hub download contention — the 8 TP processes serialize on a single HuggingFace WeakFileLock while downloading the 120b tokenizer (HF_HUB_OFFLINE=0), and the download/lock never completes within 900s.
  • Supporting: identical filelock stack across all 8 ranks for the full window; no GPU/kernel error; 20b passed in the same run; local registry mirror timed out at 18:29 (network/mirror degradation that run); job passed the prior 3 nights with no code change to the test.
  • Disconfirming: the 20b tokenizer downloaded fine minutes earlier (18:37) over the same network — so a blanket "network was down" claim is too strong; the contention is specific to the 120b launch window.
  • How to verify: re-run nightly-accuracy-8-gpu on the same runner; check whether lmsys/gpt-oss-120b-bf16 tokenizer is pre-cached on the runner / whether the HF mirror was degraded at 18:40–18:55 UTC; consider pre-warming the tokenizer or setting HF_HUB_OFFLINE=1 after a single seed download.
• [LOW] sglang code regression in the commit window (c98d31143d..592f6c849b, Jun 26→27).
  • Supporting: only temporal correlation — job was green on c98d31143d, red on 592f6c849b.
  • Disconfirming: the failure is a tokenizer-download filelock hang before any model-specific sglang code runs (init_tokenizer at scheduler __init__); the same code path succeeded for 20b in this very run; no commit in the window is implicated by the stack trace. A code regression is unlikely to explain a download-stage hang.
  • How to verify: only pursue if a re-run reproduces with a warm tokenizer cache (ruling out infra); then bisect the window.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open gpt-oss accuracy (45 results, all feature/quant work; none addresses a 120b tokenizer-download / filelock launch hang). No PR matches this symptom.

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu on the same runner pool — if it passes, this confirms a transient HF Hub / mirror flake and no code action is needed.
• Check whether lmsys/gpt-oss-120b-bf16 tokenizer files are present in the runner's HF cache; if not, pre-warm the cache (or seed once then run with HF_HUB_OFFLINE=1) so 8 TP ranks don't contend on WeakFileLock for a cold download.
• Investigate the local registry mirror 10.245.143.50:5000 health at the run time (18:29 timeout) — degraded mirror/network around 18:40–18:55 UTC is the leading suspect.
• Only if a warm-cache re-run still hangs: bisect c98d31143d..592f6c849b for changes to scheduler.py:init_tokenizer / utils/hf_transformers/tokenizer.py.

Status (factual, no priority assignment)

• Persistence: First failure 2026-06-27; this job passed the prior 3 consecutive scheduled nights (Jun 24–26). Single occurrence.
• Scope: 1 test function (test_gpt_oss_accuracy), 1 model (gpt-oss-120b-bf16), 1 job (nightly-accuracy-8-gpu). The 20b model in the same test passed.
• Blocked work: Nightly 8-GPU GPT-OSS-120b accuracy signal on AMD MI325.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-mi35x-qwen35 — failed step(s): Accuracy Test MI35x (8-GPU Qwen 3.5)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). (server exits code 1 at launch)	Log
test/registered/amd/test_qwen3p5_triton_dcp.py	TestQwen35TritonDCPGsm8k.test_a_gsm8k	AssertionError: 0.556 not greater than 0.9 (GSM8K accuracy)	Log

Failure Summary

The nightly-8-gpu-mi35x-qwen35 job ran 2 test files; both failed (Test Summary: 0/2 passed). test_qwen35_eval_mi35x.py cannot start its server — sglang serve aborts during arg validation with extra_buffer needs CUDA/MUSA/NPU (FLA). test_qwen3p5_triton_dcp.py does start (it passes --disable-radix-cache) but the GSM8K score is ~0.56–0.65, far below the 0.90 threshold.

Regression Status

Known recurring failure — both test files have failed in every completed run in the queryable window (Jun 24–27). No green run found in the lookback range; effectively Never-passed within the window.

Recent history of test_qwen35_eval_mi35x.py (test_lm_eval) and test_qwen3p5_triton_dcp.py (test_a_gsm8k) in job nightly-8-gpu-mi35x-qwen35 (branch main, event schedule, completed runs only):

Date	Run	Job	eval_mi35x Status	triton_dcp Status	Error
Jun 27	run	job	❌ Failed	❌ Failed	extra_buffer assert / GSM8K 0.556
Jun 26	run	job	❌ Failed	❌ Failed	extra_buffer assert / GSM8K 0.610
Jun 25	run	job	❌ Failed	❌ Failed	extra_buffer assert / GSM8K 0.648
Jun 24	run	job	❌ Failed	❌ Failed (TIMEOUT 3600s)	extra_buffer assert / timeout

(Lookback limited to 4 days by the shallow clone / queryable run window. First observed failure in window: Jun 24; no passing run found.)

Failure Cluster

• Cluster A: "Server launch abort — extra_buffer needs CUDA/MUSA/NPU (FLA) on ROCm" (test_qwen35_eval_mi35x.py)
• Cluster B: "Qwen3.5 Triton+DCP GSM8K accuracy below 0.90" (test_qwen3p5_triton_dcp.py)

Facts (verified observations)

• Cluster A stack trace (verbatim): server_args.py:4500 _validate_mamba_extra_buffer → assert (is_cuda() or is_musa() or is_npu()), "extra_buffer needs CUDA/MUSA/NPU (FLA).". Called from _handle_mamba_radix_cache (server_args.py:4529) ← _handle_model_specific_adjustments ← __post_init__. The server process exits with code 1 before any model load; popen_launch_server then raises Server process exited with code 1.
• The eval_mi35x test launches with --attention-backend aiter and no --disable-radix-cache, so the mamba radix-cache path runs. For the hybrid Qwen3.5 arch, mamba_radix_cache_strategy="auto" selects extra_buffer (overlap schedule on + arch supported), which then asserts CUDA/MUSA/NPU. On the MI35x (ROCm/HIP) runner is_cuda()/is_musa()/is_npu() are all False, so the assert fires. (server_args.py:4496-4502, 4516-4529)
• Cluster B: GSM8K scores across runs: 0.556 / 0.602 (Jun 27, two retries), 0.617 / 0.610 (Jun 26), 0.590 / 0.648 (Jun 25), timeout (Jun 24). Threshold GSM8K_ACCURACY_THRESHOLD = 0.90 (test_qwen3p5_triton_dcp.py:24). The triton_dcp test uses --disable-radix-cache, so it avoids the Cluster A assert and the server starts (decode throughput ~475 tok/s logged). Config: --attention-backend triton, --dcp-size 2, yarn rope factor 4.0, context 1M.
• Both failures are AMD-only by construction (MI35x nightly accuracy suite). aiter version is unchanged across all 4 runs (dev image preserved each time).

Hypothesised Causes (with confidence)

• [HIGH] Cluster A: the mamba radix-cache extra_buffer strategy is gated to CUDA/MUSA/NPU only; on ROCm the auto resolver still picks extra_buffer for Qwen3.5 and the platform assert aborts startup.
  • Supporting: stack trace lands exactly on the is_cuda() or is_musa() or is_npu() assert; auto→extra_buffer selection logic at server_args.py:4516-4524 does not exclude ROCm; an open PR (#27141) exists specifically to add ROCm support for this path.
  • Disconfirming: cannot confirm a prior green run on ROCm in the window, so this may be a never-supported feature (not a fresh regression). Shallow clone prevents bisecting the exact introduction commit.
  • How to verify: launch sglang serve --model-path Qwen/Qwen3.5-397B-A17B --attention-backend aiter on MI35x with --mamba-radix-cache-strategy no_buffer (or --disable-radix-cache); expect the assert to disappear.
• [LOW] Cluster B: a Qwen3.5 Triton+DCP correctness issue on AMD (DCP=2 / yarn rope / 1M context path) producing ~0.6 GSM8K, OR the 0.90 threshold is mis-set for this config.
  • Supporting: scores are stable ~0.6 across 3 runs (not noise-level flapping), suggesting a systematic accuracy deficit rather than a flake; server runs to completion so it is a correctness, not crash, issue.
  • Disconfirming: no green baseline in the window to prove 0.90 was ever achievable here; cannot tell from logs alone whether the regression is in DCP, triton attention, rope override, or the threshold itself. No code-level evidence gathered.
  • How to verify: re-run the same server command with --dcp-size 1 and then without the yarn --json-model-override-args to isolate whether DCP or rope-scaling drives the drop; compare against a CUDA baseline of the same config.

In-flight Fix Check

• Cluster A: In-flight fix: ✅ [#27141](https://github.com/sgl-project/sglang/pull/27141) "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" — directly targets this assert; chase reviewers, do NOT open a duplicate. Also related: #29391 "Fix mamba radix cache auto strategy crashing on non-FLA platforms" (makes auto fall back to no_buffer instead of crashing). (search: repo:sgl-project/sglang is:pr is:open extra_buffer)
• Cluster B: In-flight fix: ❌ none found matching Qwen3.5 triton/DCP GSM8K accuracy on AMD. Possibly-related accuracy PR #24613 targets extra_buffer+mixed-chunk, which this test does not use. (search: qwen3.5 mamba, extra_buffer)

Suggested Triage Steps

• Cluster A: verify #27141 (and/or #29391) resolves the assert, then re-run test_qwen35_eval_mi35x.py on MI35x. As an interim signal check, confirm whether adding --mamba-radix-cache-strategy no_buffer to the test args lets the server start.
• Cluster B: bisect the accuracy contributors by re-running test_qwen3p5_triton_dcp.py with --dcp-size 1 vs 2, and with/without the yarn rope override, to localize the ~0.6 score. If a CUDA run of the identical config also lands ~0.6, the 0.90 threshold may be miscalibrated for this DCP config (file a threshold review); if CUDA passes, focus on the triton+DCP AMD path.

Status

• Persistence: Both test files failing in every completed run Jun 24–27 (4/4 runs); no passing run in the queryable window.
• Scope: 1 job (nightly-8-gpu-mi35x-qwen35), AMD MI35x nightly accuracy suite. Cluster A blocks the Qwen3.5 aiter/radix-cache eval; Cluster B blocks the Qwen3.5 triton+DCP accuracy signal.
• In-flight fix: Cluster A ✅ #27141 (open, awaiting merge); Cluster B ❌ none found.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:43 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 3/3)

nightly-perf-2-gpu-vlm — failed step(s): Performance Test (2-GPU VLM Models)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: Dockerfile default v0.1.14rc1.dev240+g7d604afe5, dev image preserved — AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi30x/test_vlms_perf_amd.py	test_bench_one_batch (model=Qwen/Qwen3-VL-30B-A3B-Instruct)	Exception: Server process exited with code -9 (server killed after HF 429 Too Many Requests during weight download)	Log

Failure Summary

The 2-GPU VLM perf job ran two models. Qwen/Qwen2.5-VL-7B-Instruct launched and benchmarked successfully (full perf table emitted). Qwen/Qwen3-VL-30B-A3B-Instruct (TP=2) failed: the server was SIGKILLed (exit code -9) because weight loading aborted when the HuggingFace Hub API returned 429 Too Many Requests ("you have reached your 'api' rate limit … 3000 api requests per 5 minutes period"). One test errored, suite reported FAILED (errors=1), job exited 255.

Regression Status

New failure (infrastructure / transient) — this job passed in the 3 immediately preceding scheduled runs.

Recent history of test_vlms_perf_amd.py (test_bench_one_batch) in job nightly-perf-2-gpu-vlm (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	nightly-perf-2-gpu-vlm	❌ Failed	test_bench_one_batch (Qwen3-VL-30B)	HF 429 → server exit -9
Jun 26	28256335319	nightly-perf-2-gpu-vlm	✅ Passed	—	—
Jun 25	28191022667	nightly-perf-2-gpu-vlm	✅ Passed	—	—
Jun 24	28119497313	nightly-perf-2-gpu-vlm	✅ Passed	—	—

Failure Cluster

HuggingFace Hub 429 rate limit during model weight download

Facts (verified observations, no interpretation)

• httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/Qwen/Qwen3-VL-30B-A3B-Instruct' (log:5147).
• HF response body: 429 Too Many Requests: you have reached your 'api' rate limit. … you hit the quota of 3000 api requests per 5 minutes period (log:5203).
• The 429 was raised inside download_weights_from_hf → fs.ls(...) → hf_api.repo_info (stack trace in weight_utils.py:568), i.e. during weight enumeration, before model load.
• Server received SIGQUIT then was killed; popen_launch_server raised Exception: Server process exited with code -9 (log:5222).
• Sister model in the same job (Qwen/Qwen2.5-VL-7B-Instruct) loaded and benchmarked successfully — its weights were already cached / fetched before the rate limit was hit.
• The same job (nightly-perf-2-gpu-vlm) passed on Jun 24, 25, 26 (conclusions verified via job API), so this is not a standing code regression.
• This is nightly-test-amd.yml (not amd-aiter-scout.yml), so no Failure Origin / Baseline A/B section applies.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The benchmark failed because the HuggingFace Hub API rate-limited the runner (429) while resolving Qwen/Qwen3-VL-30B-A3B-Instruct weights, causing the server to abort and be killed (exit -9).
  • Supporting: verbatim 429 Too Many Requests in the stack trace at the weight-download call site; the exit-9 exception is the direct downstream effect; job exited immediately after.
  • Disconfirming: (none found) — the error is explicit and self-contained in the log.
• [FACT] Not a code regression. This is an environmental/quota issue, independent of commit 592f6c84 or the aiter build.
  • Supporting: job green on the 3 prior scheduled runs; the failing path is HF Hub network I/O, not sglang model/kernel code; the 7B model in the same process succeeded.
  • Disconfirming: (none found).
• [SPECULATION] The 30B (TP=2) download may make more concurrent HF API calls (more shards / 2 ranks resolving weights) and therefore be more likely to trip the 3000-req/5-min quota than the 7B — explaining why only the 30B failed.
  • Supporting: 30B is sharded and uses --tp 2; the 7B succeeded earlier in the run.
  • Disconfirming: the 7B succeeded partly because it ran first and may have been cache-warm; ordering, not model size, could equally explain it. No evidence isolating concurrency as the trigger.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for huggingface rate limit (7 results, all keyword-only false positives — none address CI HF 429 handling). A transient rate-limit does not require a code fix; mitigation is environmental.

Suggested Triage Steps

• Re-run the job — HF 429s are transient (the quota resets in the 5-minute window); a green rerun confirms it was a one-off.
• If it recurs: pre-stage/cache the VLM weights on the runner (warm HF_HOME) or set HF_HUB_OFFLINE=1 for the perf benchmark so weight enumeration doesn't hit the live HF API (note the launch log already shows CI_OFFLINE: … HF_HUB_OFFLINE=0).
• If recurrence is frequent: consider an authenticated HF_TOKEN with a higher quota tier, or a retry/backoff wrapper around download_weights_from_hf for 429s.

Status (factual, no priority assignment)

• Persistence: First failure in the queryable window; passed Jun 24–26. Single occurrence.
• Scope: 1 job (nightly-perf-2-gpu-vlm), 1 model (Qwen3-VL-30B-A3B-Instruct). The 7B model in the same job passed.
• Blocked work: 2-GPU VLM nightly perf signal for the 30B model only.
• In-flight fix: ❌ none found — transient infrastructure (HF Hub quota), not a code defect.

nightly-perf-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev image detected, rebuild skipped; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch (TestNightlyDeepseekV32MTPPerformance)	Exception: Server process exited with code -9 (all 8 TP schedulers crashed with AssertionError: All of them must not be None in dsa_backend.py:721)	Log

Failure Summary

The DeepSeek-V3.2 MTP perf server crashed at warmup/first batch: all 8 TP scheduler processes raised AssertionError: All of them must not be None and the server exited with code -9 (SIGQUIT). The assertion is in the DeepSeek Sparse Attention (DSA) backend's init_forward_metadata, in the is_draft_extend_v2() branch, reached via the EAGLE/MTP draft-extend path (eagle_worker_v2._draft_extend_for_decode → base_spec_worker.prepare_for_draft_extend → dsa_backend.init_forward_metadata).

Regression Status

New regression — first failing Jun 27, 2026 with this exact symptom.

Recent history of test_deepseek_v32_mtp_perf_mi35x.py (test_bench_one_batch) in job nightly-perf-8-gpu-mi35x-deepseek-v32-mtp (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28297034445	job	❌ Failed	test_bench_one_batch	Server exit -9; AssertionError: All of them must not be None
Jun 26	28256335319	job	✅ Passed	—	—
Jun 25	28191022667	job	✅ Passed	—	—
Jun 24	28119497313	job	❌ Failed	test_bench_one_batch	Different symptom: AssertionError: Benchmark failed ... (MTP variant) (perf/streaming) — no All of them must not be None (verified: 0 occurrences in Jun 24 log)

Last known passing: Jun 26 (run 28256335319, head c98d31143d). First failing with this symptom: Jun 27 (head 592f6c84). Regression window: c98d31143d..592f6c849b (32 commits).

Failure Cluster

DSA draft-extend-v2 metadata assertion crash (All of them must not be None) — server exit -9 during MTP warmup.

Facts (verified observations)

• Verbatim error (8× across TP0–TP7), dsa_backend.py:721:

  File ".../layers/attention/dsa_backend.py", line 721, in init_forward_metadata
    and forward_batch.extend_prefix_lens_cpu is not None
  AssertionError: All of them must not be None
  

  Stack: scheduler.run_batch → model_worker.forward_batch_generation → eagle_worker_v2.forward_batch_generation:1171 → _draft_extend_for_decode:850 → base_spec_worker.prepare_for_draft_extend:163 → dsa_backend.init_forward_metadata:721.
• The assert (dsa_backend.py:717-722) requires extend_seq_lens_cpu, extend_seq_lens, and extend_prefix_lens_cpu all non-None in is_draft_extend_v2() mode.
• Server then received SIGQUIT and exited -9; the test reported Server process exited with code -9 (log line 21891).
• Two DSA commits land in the regression window and are NOT in the Jun 26 passing head (verified via git merge-base --is-ancestor):
  • 9214b933 "[DSA] Enable draft-extend CUDA graph for DeepSeek Sparse Attention" (#29413), Jun 27 06:53 UTC — modifies dsa_backend.py (+78/-37), eagle_worker_v2.py (the file in the failing stack), and dsa_backend_mtp_precompute.py.
  • ee77a7d3 "[Fix] DSA: size cudagraph page_table to req_to_token width" (#29379), Jun 26 19:17 UTC.
• 9214b933 adds DeepseekSparseAttnBackend to graph_supported_backend_types in eagle_worker_v2.py (gating supports_cuda_draft_extend_graph) and reworks the is_draft_extend_v2() branch in dsa_backend.py (replacing extend_seq_lens_cpu list construction with torch.full tensors).
• aiter version is identical (dev image, unchanged) across passing and failing runs — not a variable here.

Hypothesised Causes (with confidence)

• [HIGH] Commit 9214b933 (#29413) enabling the DSA draft-extend CUDA-graph path introduced a state where extend_seq_lens / extend_seq_lens_cpu / extend_prefix_lens_cpu are not populated on forward_batch before init_forward_metadata is called in is_draft_extend_v2() mode.
  • Supporting: lands in the regression window and is absent from the Jun 26 passing head; modifies the exact failing file (dsa_backend.py) and the two eagle_worker_v2.py functions in the stack trace; its stated purpose ("enable draft-extend CUDA graph for DSA") is precisely the code path that now fails; it adds DeepseekSparseAttnBackend to the graph-supported backends, changing how/when draft-extend metadata is prepared.
  • Disconfirming: no local repro / revert A/B was run; ee77a7d3 is another DSA change in the same window and cannot be fully excluded as a contributor.
  • How to verify: git revert 9214b933 on a branch and re-run test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py on an 8-GPU MI35x runner.
• [LOW] Commit ee77a7d3 (#29379) DSA cudagraph page_table sizing contributes.
  • Supporting: also a DSA cudagraph change in the window.
  • Disconfirming: touches page_table width, not the extend_*_lens fields named in the assertion.
  • How to verify: only if reverting 9214b933 does not resolve the crash.

In-flight Fix Check

• In-flight fix: ❌ none found for this specific assertion. Closest open PR is #29499 "[DSA] Optimize DSA CUDA graph replay metadata generation" (touches the same DSA cudagraph metadata area — worth checking whether it incidentally addresses the missing extend_*_lens population, but it is framed as an optimization, not a crash fix). Searched: repo:sgl-project/sglang is:pr is:open DSA draft extend (15 results, none target this assert).

Suggested Triage Steps

• Revert 9214b933 (#29413) on a branch and re-run the v3.2-MTP perf test on 8-GPU MI35x to confirm it is the culprit.
• If confirmed, inspect the is_draft_extend_v2() setup in 9214b933: ensure forward_batch.extend_seq_lens, extend_seq_lens_cpu, and extend_prefix_lens_cpu are populated in base_spec_worker.prepare_for_draft_extend / eagle_worker_v2._draft_extend_for_decode before init_forward_metadata runs on the DSA draft model runner.
• Ask the PR author (#29413) and #29499 author whether the new DSA draft-extend graph path was exercised on the MTP+DSA (V3.2) config; CUDA-graph capture for this backend appears untested on this nightly path.
• If a quick fix isn't available, consider gating the DSA draft-extend CUDA-graph path off for V3.2 MTP until the metadata population is fixed.

Status

• Persistence: 1 completed run (Jun 27); this exact symptom is new (Jun 25 & Jun 26 passed; Jun 24 failed with an unrelated benchmark/streaming error).
• Scope: nightly-perf-8-gpu-mi35x-deepseek-v32-mtp (8-GPU MI35x, DeepSeek-V3.2 MTP + DSA). Other v3.2 jobs (-basic, non-MTP) passed Jun 26; impact concentrated on the DSA + MTP draft-extend path.
• Blocked work: DeepSeek-V3.2 MTP perf signal on AMD MI35x nightly.
• In-flight fix: ❌ none found targeting this assertion (#29499 is adjacent but not a confirmed fix).

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image preserved, matches Dockerfile default — no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	Exception: Server process exited with code -9 (server never reached ready; root cause = AssertionError: All of them must not be None in all 8 schedulers)	Log

Ran 0 tests in 2011.316s — the GSM8K eval (test_a_gsm8k) never executed because setUpClass failed during server launch.

Failure Summary

The DeepSeek-V3.2 TP8 + EAGLE-MTP server failed to start. During warmup, all 8 scheduler ranks (TP0–TP7) crashed simultaneously at 23:17:50 with AssertionError: All of them must not be None in the DSA attention backend's draft-extend path. With no live schedulers, the launcher killed the main process (exit -9), so popen_launch_server raised and setUpClass errored.

Regression Status

Known recurring failure (2 consecutive nights, different symptoms).

Recent history of test_deepseek_v32_mtp_eval_mi35x.py (TestDeepseekV32TPMTP) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (nightly-test-amd.yml, event=schedule, branch=main, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 27	28297034445	❌ Failed	setUpClass	AssertionError in dsa_backend.py:721 draft-extend → server exit -9
Jun 26	28256335319	❌ Failed	setUpClass	Different: hang/TIMEOUT after 3600s; schedulers stuck in CUDA-graph capture (dsa_backend.py:622)
Jun 25	28191022667	✅ Passed	—	—
Jun 24	28119497313	✅ Passed	—	—

First observed failure: 2026-06-26. Last known passing: 2026-06-25 (head 9495737d82). First failing head: c98d31143d. Note: today's assertion-crash symptom is distinct from the 06-26 hang/timeout symptom — both are in the DSA + MTP path on MI35x, suggesting an unstable DSA+EAGLE-v2 path rather than one clean regression.

Failure Cluster

AssertionError in DSA backend draft-extend (extend_prefix_lens_cpu is None) during DeepSeek-V3.2 EAGLE-MTP warmup.

Facts (verified observations, no interpretation)

• All 8 ranks crashed at the same instant (23:17:50) with identical stack trace (verbatim):

  File ".../speculative/eagle_worker_v2.py", line 850, in _draft_extend_for_decode
    forward_batch = self.prepare_for_draft_extend(
  File ".../speculative/base_spec_worker.py", line 163, in prepare_for_draft_extend
    draft_model_runner.attn_backend.init_forward_metadata(forward_batch)
  File ".../layers/attention/dsa_backend.py", line 721, in init_forward_metadata
    and forward_batch.extend_prefix_lens_cpu is not None
  AssertionError: All of them must not be None
  

• dsa_backend.py:717-722 — the is_draft_extend_v2() branch asserts extend_seq_lens_cpu, extend_seq_lens, AND extend_prefix_lens_cpu are all non-None.
• base_spec_worker.py:152-156 — in prepare_for_draft_extend, the gpu_only path sets only extend_seq_lens_cpu (= [num_draft_tokens] * bs); it does not set extend_prefix_lens_cpu. The not gpu_only path sets neither. → extend_prefix_lens_cpu is None when the DSA backend assertion runs.
• After the crash: [23:17:55] No live scheduler processes found → kill_process_tree ... pid=3457 → popen_launch_server reports Server process exited with code -9.
• The early-startup filelock waits (TP0–7 blocked in download_weights_from_hf) and the soft-watchdog py-spy dumps (watchdog_timeout=300, soft=True) are slow CI download/build noise, not the failure — the server progressed past load and crashed later in warmup. (FACT)
• aiter version identical to Dockerfile default; no override → aiter is not a variable here. (FACT, from [CI-AITER-CHECK] markers)

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Contract mismatch between prepare_for_draft_extend and the DSA backend's is_draft_extend_v2 branch: the backend requires extend_prefix_lens_cpu, but the producer never populates it.
  • Supporting: both code sites read directly from the current HEAD source (dsa_backend.py:721 requires it; base_spec_worker.py:152-156 sets only extend_seq_lens_cpu); the failing stack trace connects them exactly; all 8 ranks fail deterministically at the same point.
  • Disconfirming: the same source lines exist at HEAD yet the test passed on 06-25 — so either this path was not reached on 06-25 (different runtime conditions, e.g. cuda-graph eligibility gating whether init_forward_metadata is called at base_spec_worker.py:162) or extend_prefix_lens_cpu was populated upstream then. The deterministic code-level mismatch is solid; what changed to start exercising it is not proven.
  • How to verify: on a branch, in prepare_for_draft_extend populate forward_batch.extend_prefix_lens_cpu in the gpu_only branch and re-run test_deepseek_v32_mtp_eval_mi35x.py on MI35x.
• [LOW] A commit in the 44-commit window 9495737d82..c98d31143d flipped this path into being exercised.
  • Supporting: regression window is bounded by the 06-25 pass / 06-26 fail. Candidates touching the spec/deepseek path by message: #28504 "Skip empty non-idle output batches", #29142 dual-stream MoE, #14194 dcp for deepseek_v2.
  • Disconfirming: 06-26 failed with a different symptom (hang in cuda-graph capture, not this assertion), so the window does not cleanly map to today's signature; the symptom may be timing/hardware-dependent. Local git blame is unreliable (shallow clone depth 98; the relevant lines attribute to the grafted boundary commit, whose real subject is unrelated). No single commit verified to touch dsa_backend.py/base_spec_worker.py draft-extend lines in-window.
  • How to verify: full-history git log 9495737d82..c98d31143d -- python/sglang/srt/layers/attention/dsa_backend.py python/sglang/srt/speculative/base_spec_worker.py python/sglang/srt/speculative/eagle_worker_v2.py on a non-shallow checkout, then bisect.

In-flight Fix Check

• In-flight fix: ⚠️ no exact match for the extend_prefix_lens_cpu assertion. Closest relevant open PR: #29202 "[AMD] Enable draft-extend CUDA graph and reduce bubble for MTP" — directly touches the AMD MTP draft-extend path and may interact with (or fix) this; worth checking before opening a duplicate. (searched extend_prefix_lens_cpu, draft_extend, dsa+draft, deepseek+v32+mtp)
• In-flight fix: ❌ none found that explicitly addresses this assertion.

Suggested Triage Steps

• Confirm the code-level mismatch by adding extend_prefix_lens_cpu population in base_spec_worker.py:prepare_for_draft_extend (gpu_only branch, alongside extend_seq_lens_cpu) and re-running the test on an MI35x container.
• Check whether #29202 already covers this draft-extend path; if so, chase reviewers rather than opening a new PR.
• Full-history bisect of 9495737d82..c98d31143d restricted to dsa_backend.py, base_spec_worker.py, eagle_worker_v2.py to find what began exercising the is_draft_extend_v2 branch.
• Separately investigate the 06-26 hang in CUDA-graph capture (dsa_backend.py:622 init_forward_metadata_out_graph) — it shares the DSA+MTP cluster but is a distinct symptom that may have the same underlying instability.

Status (factual, no priority assignment)

• Persistence: Failing 2 consecutive scheduled nights (since 2026-06-26); last green 2026-06-25. Today's assertion-crash signature is new vs. the 06-26 timeout/hang.
• Scope: 1 job — nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp. The sibling nightly-perf-8-gpu-mi35x-deepseek-v32-mtp and nightly-8-gpu-deepseek-v32-mtp jobs passed on 06-26, so the blast radius is the MTP accuracy config specifically.
• Blocked work: DeepSeek-V3.2 EAGLE-MTP accuracy signal on MI35x.
• In-flight fix: no exact fix; #29202 is a candidate to check.

nightly-8-gpu-mi35x-glm5-mxfp4 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 592f6c84
• aiter: N/A (dependency install step was skipped — no [CI-AITER-CHECK] markers in log; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
N/A (no test ran)	N/A	Pre-test infra step Ensure VRAM is clear failed: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test was executed. The job failed in step 3 "Ensure VRAM is clear" (scripts/ci/amd/ensure_vram_clear.sh rocm), before docker setup, dependency install, and the accuracy/perf tests — all of which were skipped (steps 4–7).

Failure Summary

The runner pre-flight VRAM clearance check failed. On runner linux-mi35x-gpu-8-zf775-runner-bj4nh, GPUs 0–3 each held 16% VRAM at job start and never dropped below the 5% threshold across all 3 cleanup attempts (SIGTERM/SIGKILL of GPU holders + 30 s wait each). No host process, docker container, or KFD PID held the GPUs (No KFD PIDs currently running), so the cleanup script had nothing to kill. This is a host/infrastructure condition, not a code or test failure.

Regression Status

Infrastructure issue (new today for this job — not a code regression).

Recent history of job nightly-8-gpu-mi35x-glm5-mxfp4 (same branch main, same event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	run	nightly-8-gpu-mi35x-glm5-mxfp4	❌ Failed (pre-test infra)	— (no test ran)	VRAM not clear → exit 1
Jun 26	run	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—
Jun 25	run	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—
Jun 24	run	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—

First observed failure for this job: 2026-06-27. Last known passing: 2026-06-26. (The parent workflow nightly-test-amd.yml shows failure conclusion every run, but that reflects other jobs in the matrix — this specific job passed on Jun 24–26.)

Failure Cluster

Pre-test VRAM cleanup failure on MI35x runner (zombie KFD context / node needs reboot)

Facts (verified observations, no interpretation)

• Runner: linux-mi35x-gpu-8-zf775-runner-bj4nh (line 2).
• At job start, GPUs 0–3 = GPU Memory Allocated (VRAM%): 16; GPUs 4–7 = 0 (lines 168–171).
• 3 cleanup attempts, each: kill GPU-file holders (SIGTERM, then SIGKILL) + 30 s wait. VRAM unchanged at 16% on GPUs 0–3 after every attempt (lines 219–223, 328–332, 437–441).
• Diagnostics each attempt: No KFD PIDs currently running, no docker containers, no zombie processes, fuser found no holders (lines 276, 385, 494).
• Final: === FAILED: VRAM cleanup unsuccessful after 3 attempts === (line 537); ##[error]Process completed with exit code 1. (line 545).
• Log self-diagnosis: "almost certainly a zombie KFD context (see [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061). The node will need to be rebooted before subsequent jobs can succeed." (lines 540–543).
• Steps 4–7 (docker setup, install deps, accuracy test, perf test) all skipped — no sglang code under test was exercised.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Stuck/zombie KFD GPU context on the runner host holds 16% VRAM on GPUs 0–3, which no userspace kill can reclaim — requires a node reboot.
  • Supporting: VRAM is non-zero yet rocm-smi --showpids reports no KFD PIDs and fuser/docker/zombie checks find no holder; the cleanup script's own hint names this exact condition ([Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061).
  • Disconfirming: (none found) — there is no evidence pointing to an sglang code path; tests never ran.
  • How to verify: reboot/recycle runner linux-mi35x-gpu-8-zf775-runner-bj4nh, then re-run the job; confirm rocm-smi shows 0% VRAM before next job.
• [SPECULATION] A prior job on the same physical node leaked GPU memory and exited uncleanly, leaving the KFD context behind.
  • Supporting: GPUs 0–3 (a 4-GPU prior workload footprint) are exactly the ones stuck.
  • Disconfirming: not investigable from this job's log alone — would need the previous job scheduled on this runner.
  • How to verify: inspect the runner's prior job history / host dmesg for KFD errors.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open VRAM clear (12 results, none address MI35x runner VRAM/KFD cleanup; matches were unrelated, e.g. #24182, #13920). This is a runner/infra remediation, not a code fix.

Suggested Triage Steps

• Reboot or recycle runner linux-mi35x-gpu-8-zf775-runner-bj4nh to clear the zombie KFD context, then re-run run 28297034445.
• After reboot, confirm rocm-smi --showmemuse reports 0% on all 8 GPUs before the next scheduled nightly.
• If the node repeatedly returns to a stuck-VRAM state across jobs, escalate to the runner-fleet owner referencing [Bug] Custom all-reduce IPC buffers use fixed VA, conflict with torch_memory_saver pause/resume → zombie processes on MI300X ROCm/aiter#2061 (zombie KFD context after unclean GPU job exit).
• No sglang code change is indicated; do not bisect commits for this failure.

Status (factual, no priority assignment)

• Persistence: First failure of this job in the queryable window (2026-06-27); passed on Jun 24, 25, 26.
• Scope: This job only (nightly-8-gpu-mi35x-glm5-mxfp4), pre-test infra step. No sglang test code was exercised.
• Blocked work: Blocks GLM-5-MXFP4 8-GPU MI35x accuracy + performance nightly signal until the runner is recovered.
• In-flight fix: ❌ none found — runner reboot required, not a code fix.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: 592f6c849b — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: 7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	(whole file — runs all 13 VLMs sequentially)	TIMEOUT after 7200 seconds (exit code 255)	Log

Failure Summary

The VLM MMMU accuracy suite timed out at its per-file budget of 7200 s. The MMMU dataset download/generation phase stalled for ~1 h 52 m (no log output 18:20:18 → 20:12:43 UTC), so only the first of 13 models (Qwen/Qwen2-VL-7B-Instruct) was evaluated before the timer tripped; the other 12 models never ran. The single completed model scored 0.2901 (its threshold is 0.30), but the test was killed by the timeout before it could assert.

Regression Status

Known recurring failure — different failure mode today. This job (nightly-accuracy-2-gpu-vlm, file test_vlms_mmmu_eval_amd.py) has failed every queryable scheduled run, but the mechanism differs: prior days failed fast on an accuracy AssertionError after completing all models; today is the first observed TIMEOUT (dataset-download stall consumed the whole budget).

Recent history of test_vlms_mmmu_eval_amd.py in job nightly-accuracy-2-gpu-vlm (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failure Mode	Elapsed
Jun 27	28297034445	job	❌ Failed	TIMEOUT (dataset stall, 1/13 models ran)	7200 s
Jun 26	28256335319	job	❌ Failed	AssertionError — 5 models below threshold	3265 s
Jun 25	28191022667	job	❌ Failed	(accuracy/exit-1 family)	—
Jun 24	28119497313	job	❌ Failed	(accuracy/exit-1 family)	—

No green run found in the queryable window (Jun 24–27); test file is effectively Never-passed here. Today's TIMEOUT mode is new today.

Failure Cluster

MMMU dataset download/generation stall → VLM accuracy suite per-file timeout (only 1/13 models evaluated)

Facts (verified observations)

• ✗ TIMEOUT: .../test_vlms_mmmu_eval_amd.py after 7200 seconds → Process completed with exit code 255 (log L5485, L5502). {"file": "...test_vlms_mmmu_eval_amd.py", "passed": false, "elapsed": 7200}.
• ~1 h 52 m silent stall: last log line before the gap is at 18:20:18.7 (MMMU "Generating split" phase), next output is 20:12:43.6 — nothing in between. This is the bulk of the 7200 s budget.
• Once inference actually started it was fast: progress bar 0/100 at 20:13:48, Score: 0.290 (mmmu_score=0.2901, model Qwen/Qwen2-VL-7B-Instruct) at 20:14:49 — i.e. all 100 MMMU samples ran in ~1 minute.
• Server health was fine: The server is fired up and ready to roll! at 18:17:55 (startup 197.5 s). A single early Health check failed ... detokenizer for last 20 seconds (18:18:18) did not prevent the eval.
• Test config: register_amd_ci(est_time=7200, suite="nightly-amd-accuracy-2-gpu-vlm", nightly=True); AMD_VLM_MODELS lists 13 models run sequentially (Qwen2-VL-7B, Qwen2.5-VL-7B, Qwen3-VL-30B, InternVL2_5-2B, MiniCPM-v/o-2_6, deepseek-vl2-small, Janus-Pro-7B, Kimi-VL-A3B, MiMo-VL-7B-RL, GLM-4.1V-9B, DeepSeek-OCR-2).
• Network friction earlier in the run: Local docker registry pull failed; falling back to public registry (L1333) and Local HF snapshot ... has no files matching ['*.safetensors']; will attempt download — model weights and dataset were pulled from public sources on this runner.
• Accuracy is chronically marginal: today Qwen2-VL-7B 0.2901 vs threshold 0.30; Jun 26 had 5 models below threshold (0.2799/0.2802/0.2601/0.27/0.2802).

Hypothesised Causes (with confidence)

• [HIGH] The timeout is caused by a MMMU dataset download/generation stall (~1 h 52 m) on this MI30x runner, not by an sglang code regression.
  • Supporting: 7200 s budget, ~6720 s of it is a single silent gap during the HF "Generating split" phase before any inference; once inference began, 100 samples completed in ~1 min; same test completed in 3265 s on Jun 26 with no stall; independent network-fallback warnings (docker registry + HF weight download) in the same run.
  • Disconfirming: the exact stall point (network fetch vs. HF datasets cache build vs. image decode) is not directly printed — the gap is blank lines (stripped tqdm), so "download" is inferred from context, not a verbatim "downloading" line.
  • How to verify: re-run the job; check whether the MMMU dataset/image cache is warm on the runner; add periodic logging or a dataset-prep step so the stall location is visible.
• [FACT] Even absent the timeout, the suite would still fail on accuracy: the only completed model (Qwen2-VL-7B 0.2901) is below its 0.30 threshold, and Jun 26 failed with 5 sub-threshold models. This is an independent, pre-existing problem (marginal AMD VLM thresholds), unrelated to today's stall.
• [LOW] HEAD commit 592f6c849b (minimax-m3 mem-cache/HiCache split) lands in the window but touches KV-pool/cache code, not VLM dataset loading or the MMMU eval path.
  • Disconfirming: the stall is in the test harness's dataset phase before model inference; no sglang stack trace; failure is reproducible-style infra timing, not a crash.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for vlm mmmu accuracy and MMMU timeout; matches (e.g. #20548, #26212) are VLM CUDA-graph features, none address the dataset-download stall or the AMD VLM accuracy thresholds.

Suggested Triage Steps

• Re-run nightly-accuracy-2-gpu-vlm once to confirm the stall is transient (network/HF) vs. persistent — if it recurs, the MMMU dataset is not cached on the MI30x runner and needs a warm cache or a pre-fetch step.
• Add coarse logging around the MMMU dataset load (start/end of download per subject) so the ~2 h gap becomes attributable instead of blank.
• Consider a per-model timeout (in addition to the 7200 s per-file budget) so one slow/stalled model can't starve the other 12; or split the 13-model suite into multiple files.
• Separately, treat the chronic accuracy shortfall as its own issue: Qwen2-VL-7B (0.2901<0.30) and the Jun 26 sub-threshold set suggest thresholds and/or AMD VLM output quality need review — not fixable by the timeout change.

Status

• Persistence: test_vlms_mmmu_eval_amd.py failing every scheduled run Jun 24–27 (no green in window). Today's TIMEOUT sub-mode is new (Jun 27); prior days failed on accuracy in ~3265 s.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm) in nightly-test-amd.yml, MI30x runner. Sister nightly-perf-2-gpu-vlm passed in each run.
• Blocked work: No AMD VLM MMMU accuracy signal — today 12 of 13 models were not evaluated at all.
• In-flight fix: ❌ none found — needs new investigation (infra/dataset cache + separate accuracy-threshold review).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:43 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 20 failure(s)

Run started (UTC): 2026-06-27 17:47 UTC
Last scanned (UTC): 2026-06-28 02:33 UTC
Commits: sglang 592f6c8

Summary

Counts: 24 failures · 12 clusters · 6 carrying over (≥3 days) · 18 new today

Most of today's red (≈14 of 24 failures) is HuggingFace download / runner infrastructure, not code: 8 server-launch timeouts from HF weight-download stalls (cluster C1), 3 HF 429 rate-limits (C2), plus Kimi slow-load, a perf-runner hang, a bench-test tokenizer hang and a zombie-KFD VRAM-clear failure. The genuine code regressions are C3 (DSA draft-extend v2 assert — fresh, HIGH, no fix) and the carry-overs C4 (ROCm mamba extra_buffer, 9 days, fix #27141 ready) and C5 (ROCm 7.2.0 hipEventQuery capture race, 7 days, fix #24151 ready). Run 28296988041, sglang 592f6c84, aiter 7d604afe5.

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	HF download stall → launch timeout	nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2	test_deepseek_r1_mxfp4_tp2_mi35x.py	setUpClass	new today	[HIGH] HF Xet weight-download stall (infra)
2	HF download stall → launch timeout	nightly-8-gpu-deepseek-v31	test_deepseek_v31_eval_amd.py	N/A (whole-file)	new today	[HIGH] HF xet download stall (infra)
3	HF download stall → launch timeout	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4	test_deepseek_r1_mxfp4_tp4_mi35x.py	setUpClass	new today	[MEDIUM] cold HF cache → download stall
4	HF download stall → launch timeout	nightly-8-gpu-mi35x-grok2	test_grok2_eval_mi35x.py	test_grok2_accuracy	new today	[MEDIUM] HF xet stall (infra)
5	HF download stall → launch timeout	nightly-8-gpu-qwen3-235b	test_qwen3_instruct.py	N/A (whole-file)	new today	[HIGH] HF download filelock stall
6	HF download stall → launch timeout	nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Llama-3.1-70B)	new today	[FACT] HF Xet download stall
7	HF download stall → launch timeout	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	new symptom (file Never-passed)	[MEDIUM] cold-cache stall + 3600s wrapper
8	HF download stall → launch timeout	nightly-accuracy-8-gpu	test_gpt_oss_eval_amd.py	test_gpt_oss_accuracy (120b)	new today	[HIGH] HF cache miss → filelock hang
9	HF Hub 429 rate-limit	nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (Qwen2.5-VL/Qwen2-VL)	new today	[FACT] HF 429 (fix #23400)
10	HF Hub 429 rate-limit	nightly-accuracy-8-gpu-mi35x	test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy (120b)	new today	[FACT] HF 429
11	HF Hub 429 rate-limit	nightly-accuracy-8-gpu-mi35x	test_qwen3_coder_next_eval_mi35x.py	test_qwen3_coder_next_accuracy	new today	[FACT] HF 429
12	ROCm mamba extra_buffer assert	nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	test_lm_eval	9 days persistent	[HIGH] #28151/66a7fd5c; fix #27141
13	ROCm mamba extra_buffer assert	nightly-8-gpu-qwen35	test_qwen35_eval_amd.py	setUpClass	9 days persistent	[HIGH] #28151; fix #27141
14	DSV4-Pro server SIGKILL on load	nightly-8-gpu-mi35x-deepseek-v4-pro	test_deepseek_v4_pro_fp4.py	setUpClass	flaky / new	[MEDIUM] host-OOM SIGKILL
15	DSV4-Pro server SIGKILL on load	nightly-8-gpu-mi35x-deepseek-v4-pro	test_deepseek_v4_pro_fp4_cp.py	setUpClass	flaky / new	[MEDIUM] host-OOM SIGKILL
16	DSA draft-extend v2 assert (None cpu mirror)	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test_deepseek_v32_mtp_eval_mi35x.py	setUpClass	new today	[HIGH] #29413/9214b933
17	DSA draft-extend v2 assert (None cpu mirror)	nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	new today	[HIGH] #29413/9214b933
18	hipErrorCapturedEvent in RCCL watchdog	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	setUpClass	7 days persistent	[MEDIUM] ROCm 7.2.0 event-query race; fix #24151
19	Mistral/Mixtral gsm8k accuracy	nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	test_gsm8k_all_models (Mistral-7B)	≥4 days persistent	[LOW] chat-template; fix #27757
20	Qwen3-30B-A3B GPU mem fault	nightly-accuracy-8-gpu-mi35x	test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy	4/5 runs since Jun 24 (intermittent)	[LOW] MoE forward GPU fault
21	Kimi-K2.6 weight load > timeout	nightly-8-gpu-kimi-k26	test_kimi_k26_eval_amd.py	test_kimi_k26_gsm8k_accuracy	3 days persistent	[FACT] slow load (3359s) > 3600s budget; #24076
22	Host VRAM not clearable (zombie KFD)	nightly-8-gpu-mi35x-deepseek-r1-hicache	N/A (infra pre-step)	N/A	new today	[HIGH] zombie KFD context — node reboot
23	Perf runner hang after warmup	nightly-perf-2-gpu-text	test_text_models_perf_amd.py	N/A (whole-file)	new today	[LOW] benchmark hang (flaky)
24	bench_serving gpt2 tokenizer hang	nightly-test-1-gpu-unit	test_bench_serving_functionality.py	test_custom_headers_sent_to_server	new today	[LOW] gpt2 tokenizer download hang (flaky)

Cluster: HF download stall → launch timeout (rows 1-8)

• Shared facts: every job's py-spy stacks park in download_weights_from_hf → snapshot_download/hf_hub_download → xet_get (or filelock._poll_until_acquired on sibling ranks); HF_HUB_OFFLINE=0; server never reaches readiness; killed at the 3600s budget. All 8 passed in the prior 1-3 scheduled runs (or, row 7, the file is chronically red but the download-timeout symptom is new).
• Top hypothesis: [HIGH] transient HF Xet/network stall on cold/partial runner caches — not sglang code (stacks are entirely in third-party download code; HEAD 592f6c84 doesn't touch the load path). Disconfirming: silent stalls (no HTTP error logged) → cannot fully rule out a ci_download_with_validation_and_retry deadlock.
• In-flight fix: ❌ none directly; #25108 (HF-cache fallback/warn) and #29419 (sgl-data mount) are tangential.
• Suggested triage: pre-stage weights on the MI30x/MI35x runners (warm /sgl-data/hf-cache) + re-run; add a download-level timeout/retry so a stalled xet_get fails fast instead of wedging the full hour.

Cluster: HF Hub 429 rate-limit (rows 9-11)

• Shared facts: 429 Too Many Requests … 0/3000 requests remaining in current 300s window (plan team) during live weight/dataset download; cache lacked the safetensors so an online fetch was forced. First model in each suite passed.
• Top hypothesis: [FACT] external HF API rate-limit (printed verbatim). In-flight fix: ✅ #23400 "prefetch HF snapshots / skip redundant metadata calls to survive 429s" — chase reviewers.
• Suggested triage: pre-warm caches + land #23400; stagger concurrent AMD jobs that hit HF in the same 5-min window.

Cluster: ROCm mamba extra_buffer assert (rows 12-13) — carry-over, fix ready

• AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA) at server_args.py:4500; auto mamba-radix-cache strategy picks extra_buffer, hard-gated off ROCm. Deterministic since Jun 19, every run; both qwen35 jobs.
• Top hypothesis: [HIGH] #28151 (66a7fd5c) removed the ROCm no_buffer fallback. In-flight fix: ✅ #27141 (ROCm extra_buffer support, open since 2026-06-03); prior fallback #29158 closed unmerged.
• Suggested triage: land #27141, or interim --mamba-radix-cache-strategy no_buffer to restore signal.

Cluster: DSA draft-extend v2 assert (rows 16-17) — fresh regression, no fix

• All 8 ranks: AssertionError: All of them must not be None at dsa_backend.py:721 via eagle_worker_v2._draft_extend_for_decode → prepare_for_draft_extend. Both DSV4.2-MTP jobs (perf exit -9, accuracy exit -9). Passed Jun 26 (c98d31143d), fails first run after the commit landed.
• Top hypothesis: [HIGH] #29413/9214b933 set needs_cpu_seq_lens=False (nulling extend_*_cpu) but moved only the graph path off the mirrors; the eager path (taken on ROCm, where draft-extend graph is disabled) still asserts them. Disconfirming: (none found) — CUDA masks it via the graph path. In-flight fix: ❌ none (#29499 is adjacent but not this assert).
• Suggested triage: revert 9214b933 on a branch + re-run on MI35x; or make the eager is_draft_extend_v2() branch derive lengths from GPU seq_lens/page-table width.

Cluster: hipErrorCapturedEvent in RCCL watchdog (row 18) — carry-over, fix ready

• RCCL watchdog hipEventQuery on an event in a capturing stream during MoE all-reduce HIP-graph capture → scheduler exit -6. Identical 7/7 runs since Jun 20, ROCm 7.2.0 / mi35x.
• Top hypothesis: [MEDIUM] ROCm 7.2.0 event-query race. In-flight fix: ✅ #24151 (bump 7.2.0→7.2.2); also #23581 (SGLANG_USE_AITER_AR=false).

Cluster: DSV4-Pro server SIGKILL on load (rows 14-15)

• Both FP4 launches exit -9 during the ~40-min load; GPU VRAM healthy (~100-118 GB free) → host-side kill; rank-0 loads 3-5× slower than peers. Files added Jun 24; failing file rotates run-to-run (fp8 Jun 26, fp4+fp4_cp Jun 27) and retries pass 3/3 → flaky.
• Top hypothesis: [MEDIUM] Linux OOM-killer during concurrent 8-process load (-9 = SIGKILL). Disconfirming: no dmesg oom-kill captured. In-flight fix: ❌ none on AMD.

Cluster: Mistral/Mixtral gsm8k accuracy (row 19) — carry-over

• Mistral-7B 0.36 vs 0.47 threshold, every run ≥ Jun 24 (Mistral/Mixtral-family-wide). In-flight fix: ✅ #27757 "Fix Mistral GSM8K chat eval". (Today's Llama-70B timeout (row 6) masked the full matrix by aborting early.)

Cluster: Qwen3-30B-A3B GPU mem fault (row 20) — carry-over, intermittent

• Memory access fault by GPU node mid-eval → exit -6; failed 4/5 runs since Jun 24, passed Jun 23. Top hypothesis: [LOW] MoE forward-path fault on gfx950, candidate window ed26a109ee..d6aacd2801 (#28084/#25090/#27833); disconfirming: passed inside the window, inconsistent with a deterministic commit. ❌ no fix; enable /coredumps to capture the faulting kernel.

Cluster: Kimi-K2.6 weight load > timeout (row 21) — carry-over

• Load weight end elapsed=3358s of a 3600s budget; eval reached 99% then deadline. Failed Jun 25-27, passed Jun 24 (load 2380s). [FACT] startup-bound, not accuracy. In-flight fix: ✅ #24076 (vectorize Kimi MoE repack); consider raising est_time.

Cluster: Host VRAM not clearable / zombie KFD (row 22)

• Pre-test ensure_vram_clear.sh failed 3/3 on runner linux-mi35x-gpu-8-zf775-runner-9w45z (GPUs 0-3 stuck at 16%, no holder PID). [HIGH] orphaned KFD context → node reboot/cordon; tracked upstream ROCm/aiter#2061.

Cluster: Perf runner hang after warmup (row 23) / bench_serving gpt2 hang (row 24)

• Row 23: 2-GPU text perf hung ~57 min after warmup /flush_cache (no benchmark code changed in window) → likely flake. Row 24: unit test_bench_serving_functionality.py 15-min silence right where get_tokenizer("gpt2") runs → transient HF stall; no python/sglang/benchmark/ change in window. Both [LOW]/flaky, ❌ no fix — re-run to confirm.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Re-run workflow with pre-warmed /sgl-data/hf-cache (+ land #23400); reboot runner …zf775…9w45z	C1 rows 1-8, C2 rows 9-11, C9 row 21, C11 row 22, C12 row 23, C10 row 24	Largest blast radius (~14 failures) and cheapest — almost all are infra/HF stalls, not code
2	Land in-flight #27141 (ROCm mamba support)	C4 rows 12-13	9-day deterministic block with a fix already open
3	Revert/fix 9214b933 (#29413), re-run DSV4.2-MTP	C3 rows 16-17	Fresh HIGH-confidence regression, no in-flight fix, breaks all ROCm DSA+spec
4	Verify in-flight #24151 (ROCm720 7.2.2 bump), re-run mxfp4 job	C5 row 18	7-day persistent with a matching fix; cheap A/B
5	Capture dmesg/coredump on MI35x for the host-OOM/GPU-fault crashes	C8 rows 14-15, C6 row 20	Need kernel-log evidence before a code path can be implicated

---

Job	Failed Steps	Started
nightly-perf-2-gpu-text-rocm720	Performance Test ROCm 7.2 (2-GPU Text Models)	2026-06-27T17:47
nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720	Accuracy Test MI35x ROCm 7.2 (2-GPU DeepSeek-R1-MXFP4 TP2)	2026-06-27T17:47
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-06-27T17:48
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720	Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4)	2026-06-27T17:47
nightly-test-1-gpu-unit-rocm720	Nightly Unit Test ROCm 7.2 (1-GPU)	2026-06-27T17:49
nightly-8-gpu-deepseek-v31-rocm720	Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.1)	2026-06-27T17:47
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-06-27T17:47
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-06-27T17:48
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP)	2026-06-27T22:04
nightly-accuracy-8-gpu-rocm720	Accuracy Test ROCm 7.2 (8-GPU GPT-OSS)	2026-06-27T17:48
nightly-8-gpu-mi35x-grok2-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Grok2)	2026-06-27T17:48
nightly-8-gpu-qwen35-rocm720	Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-27T17:48
nightly-8-gpu-qwen3-235b-rocm720	Accuracy Test + Performance Test ROCm 7.2 (8-GPU Qwen3)	2026-06-27T17:48
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-06-27T20:55
nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720	Ensure VRAM is clear	2026-06-27T21:00
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)	2026-06-27T20:16
nightly-8-gpu-mi35x-qwen35-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)	2026-06-27T20:12
nightly-8-gpu-kimi-k26-rocm720	Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)	2026-06-27T18:55
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-06-27T17:48
nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720	Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton)	2026-06-27T21:02

Per-Job Analysis

nightly-perf-2-gpu-text-rocm720 — failed step(s): Performance Test ROCm 7.2 (2-GPU Text Models)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: Dockerfile default — dev/patched image build detected, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi30x/test_text_models_perf_amd.py	N/A (whole-file perf runner)	TIMEOUT after 3600s → process killed, exit code 255	Log

Failure Summary

The 2-GPU text-model perf runner test_text_models_perf_amd.py exceeded its 3600s budget and was killed (exit 255). The server for the first model (meta-llama/Llama-3.1-8B-Instruct) started successfully and served a warmup /generate (200 OK at 17:58:37), then a /flush_cache succeeded at 17:59:24 — after which no further server activity is logged for ~57 minutes until kill_process_tree at 18:56:32. The run hung during the benchmark phase rather than failing a numeric perf threshold.

Regression Status

New regression (or a one-off hang). This specific job passed in the last 5 completed scheduled runs.

Recent history of test_text_models_perf_amd.py in job nightly-perf-2-gpu-text-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Error
Jun 27	28296988041	job	❌ Failed	TIMEOUT 3600s (hang after warmup)
Jun 26	28256271771	job	✅ Passed	—
Jun 25	28190877198	job	✅ Passed	—
Jun 24	28119401197	job	✅ Passed	—
Jun 23	28046795462	job	✅ Passed	—
Jun 22	27975850577	job	✅ Passed	—

First observed failure: 2026-06-27. Last known pass: 2026-06-26 (c98d3114). Lookback: 6 most recent scheduled runs. (Note: today's overall workflow run is still in_progress, but THIS job has completed with failure.)

Failure Cluster

Perf benchmark hang/timeout after server warmup (no progress for ~57 min, killed at 3600s)

Facts (verified observations)

• Server reached "fired up and ready to roll" and returned 200 OK on /generate at 17:58:37 (log:417).
• /flush_cache returned 200 OK at 17:59:24 — last server log line before the silent gap.
• A transient health-check warning fired once during aiter kernel JIT build: Health check failed. Server couldn't get a response from detokenizer for last 20 seconds at 17:59:03 — but the server recovered (/health_generate → 200 OK at 17:59:14).
• No server log output between 17:59:24 and kill_process_tree at 18:56:32 (~57 min silence), then ✗ TIMEOUT ... after 3600 seconds, Test Summary: 0/1 passed, exit code 255.
• aiter was JIT-compiling mha_batch_prefill_* kernels during warmup (each ~24–28s); the second build finished at 17:59:07, before the hang window.
• Commit window c98d3114..592f6c84 = 32 commits.

Failure Origin

N/A — this is nightly-test-amd-rocm720.yml, not an AITER Scout run (no AITER_COMMIT_OVERRIDE).

Hypothesised Causes (with confidence)

• [LOW] Flaky/one-off hang on the perf runner (benchmark client stalled after flush_cache).
  • Supporting: test passed 5 consecutive prior runs; the server itself served the warmup request and flush successfully, so basic serving was functional; hangs at the benchmark-client layer are a known intermittent failure mode on loaded MI30x runners.
  • Disconfirming: a single occurrence can't be distinguished from a real regression without a re-run; the 57-min total silence is unusually clean for a transient stall.
  • How to verify: re-run the job at the same SHA; if it passes, classify as flake.
• [LOW] IPC/detokenizer regression from #28688 (be193013 "Convert IPC dataclasses to msgspec.Struct with opt-in msgpack transport"), merged Jun 26 inside the window.
  • Supporting: heavily rewrites io_struct.py (+614/-…), tokenizer_manager.py, and output_streamer.py — the scheduler→detokenizer→tokenizer output path; a "couldn't get a response from detokenizer" health warning appeared right before the hang.
  • Disconfirming (strong): the warmup /generate returned 200 OK (full tokenize→schedule→detokenize→HTTP round-trip worked), and the detokenizer health check recovered to 200 OK; the msgpack transport is opt-in and not enabled by these server_args. A fundamental IPC break would have failed the first request, not only the post-flush benchmark.
  • How to verify: git revert be193013 on a branch and re-run test_text_models_perf_amd.py on MI30x; or re-run current SHA first to rule out a flake.
• No hypothesis reaches MEDIUM confidence. If the failure reproduces on re-run, bisect c98d3114..592f6c84 focusing on IPC/scheduler/HiCache commits (#28688, #28713, #28614).

In-flight Fix Check

• In-flight fix: ❌ none found matching a perf-runner hang on this test. Searched open PRs for detokenizer hang and perf timeout (via search/issues); results (#27254 detokenizer recv-loop optimization, #29203 embedding hang with multi tokenizer-worker) are unrelated to this single-worker 2-GPU text perf path.

Suggested Triage Steps

• Re-run this job at SHA 592f6c84 first — a single hang after 5 green runs is most likely a flake; a second failure promotes it to a real regression.
• If it reproduces: bisect c98d3114..592f6c84 (32 commits), testing the IPC/msgspec change be193013 early given the detokenizer health warning.
• Capture a py-spy dump / stack trace of the hung server + benchmark client on the next occurrence to localize whether the stall is in the benchmark client, the HTTP server, or scheduler↔detokenizer IPC.

Status

• Persistence: 1 failed run (first seen 2026-06-27 17:47 UTC); 5 prior completed runs green.
• Scope: this job only (nightly-perf-2-gpu-text-rocm720); not observed in sibling jobs in this run.
• Blocked work: 2-GPU text-model perf signal on AMD ROCm 7.2 nightly.
• In-flight fix: ❌ none found — needs re-run to confirm flake vs regression before opening a PR.

nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (2-GPU DeepSeek-R1-MXFP4 TP2)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; matches Dockerfile default gfx950 block)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py	TestDeepSeekR1MXFP4TP2MI35x.setUpClass	TimeoutError: Server failed to start within the timeout period	Log

Failure Summary

The single test file in this job, test_deepseek_r1_mxfp4_tp2_mi35x.py, failed in setUpClass (popen_launch_server, line 125) with TimeoutError: Server failed to start within the timeout period after 3609 s. The server reached Load weight begin at 17:56:54 but never finished loading; it remained stuck downloading model weights from HuggingFace for the entire ~55 min window until the launch timeout fired. Ran 0 tests — the GSM8K accuracy assertions never executed.

Regression Status

Infrastructure issue (external network / HF download hang) — distinct from the prior failure on this job.

Recent history of test_deepseek_r1_mxfp4_tp2_mi35x.py (setUpClass) in job nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	this job	❌ Failed	setUpClass	Server launch timeout — HF weight download hang
Jun 26	28256271771	job	✅ Passed	—	—
Jun 25	28190877198	job	❌ Failed (different cause)	—	VRAM cleanup unsuccessful after 3 attempts (runner pre-check; test never ran)
Jun 24	28119401197	job	✅ Passed	—	—

The test passed as recently as Jun 26. The Jun 25 failure was a separate infra issue (VRAM cleanup), not a download hang. Today's symptom is new.

Failure Cluster

HuggingFace weight-download hang during server warmup (Xet backend stalls; server launch timeout)

Facts (verified observations)

• command=sglang serve --model-path amd/DeepSeek-R1-MXFP4-Preview --tp 2 ... launched with CI_OFFLINE: Launching server HF_HUB_OFFLINE=0 model=amd/DeepSeek-R1-MXFP4-Preview → weights fetched live from HF, not from runner cache.
• [TP0] Load weight begin. avail mem=279.14 GB at 17:56:54; no Load weight end line ever appears.
• All 11 py-spy dumps from 18:01:40 → 18:51:46 show the identical stuck stacks:
  • TP1 (pid 3339): xet_get (huggingface_hub/file_download.py:566) ← _download_to_tmp_and_move ← snapshot_download ← download_weights_from_hf (weight_utils.py:593) ← ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955). The ThreadPoolExecutor worker is active in xet_get — the Xet chunk download is stalled.
  • TP0 (pid 3338): _poll_until_acquired (filelock/_api.py:552) ← download_weights_from_hf (weight_utils.py:554) — blocked on the HF cache filelock held by TP1 while TP1 downloads.
• hf-xet==1.5.1 installed (log line 2139).
• Watchdog escalated every 5 min: TokenizerManager watchdog timeout (18:01:36) then Scheduler watchdog timeout (TP0/TP1) repeatedly; py-spy --native failed (failed to get os threadid) because the GPU process was wedged in the native Xet/network call.
• Ran 0 tests in 3609.313s → FAILED (errors=1) → job exit code 255.
• Earlier transient: [local-pull] Error response from daemon ... context deadline exceeded for the internal docker registry (10.245.143.50:5000), fell back to public registry — unrelated to the later weight-download hang but indicates runner-network flakiness in this run.

Hypothesised Causes (with confidence)

• [HIGH] External HuggingFace Xet download stall — the model weights for amd/DeepSeek-R1-MXFP4-Preview were downloaded live and the hf-xet transfer hung indefinitely.
  • Supporting: every py-spy dump for ~50 min shows the active worker frozen in xet_get; no bytes-progress or Load weight end; TP0 merely waits on the cache lock TP1 holds. The job's only test passed yesterday with no code change to the loading path implicated.
  • Disconfirming: no explicit network exception/timeout was emitted in the log (the hang is silent, consistent with a stuck socket rather than a code bug).
  • How to verify: re-run the job; if green, confirms transient HF/Xet network stall. Optionally pre-stage amd/DeepSeek-R1-MXFP4-Preview in the runner HF cache (so HF_HUB_OFFLINE path is used) and check the hang disappears.
• [LOW] hf-xet client-side bug/incompatibility causing the transfer to never time out.
  • Supporting: stuck frame is inside hf-xet’s xet_get; no per-file download timeout is enforced.
  • Disconfirming: same hf-xet==1.5.1 and same loader path passed on Jun 24/26 — points to environment/network, not the client version.
  • How to verify: check whether other HF-downloading nightly jobs on the same runner/date also hung in xet_get.
• No sglang code regression hypothesis reaches MEDIUM: the loading code path (deepseek_weight_loader.py, weight_utils.py, loader.py) is unchanged relative to the Jun 26 passing run, and the stall is in third-party download code.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for xet download and weight download timeout; the 3/16 hits returned (e.g. #29178, #28905, #28584) are unrelated (Kimi-K2.6 MoE loading / MI300 runner CI). No PR addresses the HF Xet download hang or adds a weight-download watchdog/timeout.

Suggested Triage Steps

• Re-run nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720; if it passes, classify as a transient HF/Xet network stall (no code action).
• Pre-stage / cache amd/DeepSeek-R1-MXFP4-Preview weights on the MI35x runners so CI uses the offline path instead of a live HF download, removing this external dependency from the critical startup path.
• Consider a download-level timeout/retry around ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) so a stalled xet_get aborts and retries rather than wedging until the 3600 s server-launch timeout. (Investigation, not a directive — touches shared loader code.)
• If Xet stalls recur, try HF_XET_HIGH_PERFORMANCE=0 / disabling the Xet backend (HF_HUB_DISABLE_XET=1) on the runner to fall back to HTTPS range downloads, and compare.

Status

• Persistence: 1 occurrence (Jun 27 run). Same test passed Jun 26; Jun 25 failure was a different infra cause (VRAM cleanup).
• Scope: this job only (its single test file). The hang is in the external HF download path, not GPU/kernel/aiter code.
• Blocked work: blocks the DeepSeek-R1-MXFP4 TP2 accuracy signal on MI35x ROCm 7.2 for this nightly run only.
• In-flight fix: ❌ none found — transient infra; re-run first, then consider weight pre-staging / download-timeout hardening.

nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: 592f6c84
• aiter: N/A (no [CI-AITER-CHECK] markers in this accuracy job log)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=meta-llama/Llama-3.1-70B-Instruct)	TimeoutError: Server failed to start within the timeout period	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models (model=mistralai/Mistral-7B-Instruct-v0.3)	gsm8k score 0.361 < threshold 0.47 (3 retries: 0.358 / 0.364 / 0.361)	Log

Failure Summary

The accuracy job ran 5 of its usual ~20 models before aborting. meta-llama/Llama-3.1-70B-Instruct (TP=2) raised TimeoutError: Server failed to start — the unittest's reported errors=1 — after its weight download stalled in HuggingFace Xet (xet_get), tripping repeated 300 s scheduler/TokenizerManager watchdog timeouts. Separately, mistralai/Mistral-7B-Instruct-v0.3 scored ~0.36 against a 0.47 threshold on all three retries. The job's other 3 executed models (Llama-3.1-8B, DeepSeek-Coder-V2-Lite, gemma-2-27b) passed.

Regression Status

Mixed: Mistral-7B accuracy = known recurring failure (≥4 days); Llama-3.1-70B server-launch timeout = new today / infra.

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Mistral-7B score	Llama-3.1-70B (TP2)	Job verdict
Jun 27	28296988041	❌ 0.361 (<0.47)	❌ TimeoutError (Xet download hang)	❌ errors=1
Jun 26	28256271771	❌ 0.368 (<0.47)	✅ 0.926	❌ failures=1
Jun 25	28190877198	❌ 0.365 (<0.47)	✅ 0.931	❌ failures=1
Jun 24	28119401197	❌ 0.369 (<0.47)	✅ 0.929	❌ failures=1

• Mistral-7B: failing every completed run in the queryable window (no green found ≥ Jun 24); part of a broader Mistral/Mixtral-family accuracy problem on ROCm720 — in the Jun 26 full run, Mixtral-8x7B-Instruct-v0.1 (0.304/0.69), Mistral-7B-Instruct-v0.3-FP8 (0.296/0.46) and Mixtral-8x7B-Instruct-v0.1-FP8 (0.164/0.69) also failed.
• Llama-3.1-70B: passed Jun 24/25/26, first observed timeout today → environmental, not an accuracy regression. Its early abort is what flipped the unittest verdict from failures=1 to errors=1 and prevented the remaining ~15 models (incl. the usual Mixtral failures) from running today.

Failure Cluster

Two clusters: (1) "Mistral/Mixtral gsm8k accuracy below threshold on ROCm720"; (2) "Server launch TimeoutError from HuggingFace Xet weight-download hang".

Facts (verified observations)

• Llama-70B stack (py-spy): scheduler TP1 stuck in xet_get (huggingface_hub/file_download.py:566); TP0 blocked in filelock._poll_until_acquired; [... TP1] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) fired at 19:05:09, 19:10:10, 19:15:10 before TimeoutError at 19:19:43. Launch used HF_HUB_OFFLINE=0.
• meta-llama/Llama-3.1-70B-Instruct is not in DISABLE_HF_XET_MODELS (only the two Qwen2-57B entries are), so HF_HUB_DISABLE_XET=0 for it — Xet backend was active (test file lines 107-110, 249).
• Mistral-7B threshold is 0.47 ("52.1% - 5%", line 45); observed 0.358/0.364/0.361 — a ~23% relative shortfall, consistent across 3 retries and 4 days.
• The 3 other executed models passed: Llama-3.1-8B 0.818/0.80, DeepSeek-Coder-V2-Lite 0.820/0.81, gemma-2-27b 0.861/0.86.

Hypothesised Causes

• [FACT] Llama-3.1-70B timeout = HuggingFace Xet weight download stalled during server startup (network/infra), not a code defect.
  • Supporting: py-spy shows the scheduler thread parked in xet_get; weights are fetched live (HF_HUB_OFFLINE=0); Llama-70B passed the previous 3 nights with identical config.
  • Disconfirming: none — the stall is directly in the HF download call, not in sglang model-load code.
  • How to verify: re-run the job; if it recurs, add meta-llama/Llama-3.1-70B-Instruct to DISABLE_HF_XET_MODELS (or pre-cache weights) and confirm startup succeeds.
• [LOW] Mistral-7B accuracy — candidate cause is a Mistral chat-template / eval-harness mismatch on this path rather than a kernel regression.
  • Supporting: failure is Mistral/Mixtral-family-wide (base + FP8) while Llama/Qwen/DeepSeek/gemma all pass; scores are stable (~0.36), characteristic of a formatting/template issue, not numerical noise; open PR #27757 "Fix Mistral GSM8K chat eval" targets exactly this.
  • Disconfirming: no green run exists in the queryable window, so a specific regressing commit cannot be bisected from CI history here; threshold (0.47) could itself be miscalibrated for AMD.
  • How to verify: review/merge #27757 and re-run; inspect /tmp/gsm8k_mistralai_Mistral-7B-Instruct-v0.3.html for malformed prompts/answers.

In-flight Fix Check

• Mistral accuracy → ✅ #27757 "Fix Mistral GSM8K chat eval" (open since 2026-06-10, last updated 2026-06-22) — directly on point; chase reviewers, do not duplicate. (keywords: Mistral gsm8k accuracy)
• Llama-70B Xet hang → ❌ none found for HF_HUB_DISABLE_XET in open PRs — needs a config tweak or is a transient infra flake.

Suggested Triage Steps

• Re-run run 28296988041 to confirm the Llama-70B Xet timeout is transient; if it recurs, add meta-llama/Llama-3.1-70B-Instruct to DISABLE_HF_XET_MODELS or pre-stage weights in the runner cache.
• Review and land #27757, then re-run the accuracy job to confirm Mistral-7B (and Mixtral-family) clear threshold; if still red, separately re-validate the 0.47 threshold for the ROCm720 path.
• Note that today's errors=1 masked the routine Mistral/Mixtral failures by aborting early — fixing the Llama-70B startup will restore full-matrix signal.

Status

• Persistence: Mistral-7B accuracy red every completed run since at least Jun 24 (4 days); Llama-3.1-70B timeout first seen today.
• Scope: job nightly-accuracy-2-gpu-rocm720 in nightly-test-amd-rocm720.yml; the whole workflow has been red for ~7 consecutive scheduled runs (Jun 20-27).
• Blocked work: AMD ROCm720 nightly accuracy signal; the Llama-70B abort additionally suppresses results for the remaining ~15 models in the matrix.
• In-flight fix: ✅ #27757 (Mistral); ❌ none for the Llama-70B Xet hang.

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4)

Commit Info

• sglang: 592f6c84 — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved; rebuild skipped per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py	setUpClass	TimeoutError: Server failed to start within the timeout period (3600 s)	Log

Failure Summary

The setUpClass fixture for TestDeepSeekR1MXFP4TP4MI35x raised TimeoutError from popen_launch_server (test line 124) after the full SERVER_LAUNCH_TIMEOUT=3600 s elapsed. The server process never finished startup because all 4 TP workers were blocked in HuggingFace weight download (download_weights_from_hf). Ran 0 tests in 3609.447s, job exited 255.

Regression Status

New regression / Infrastructure issue (weight-download stall) — first failure for this test file in the queryable window.

Recent history of test_deepseek_r1_mxfp4_tp4_mi35x.py (setUpClass) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	job	❌ Failed	setUpClass	TimeoutError: Server failed to start
Jun 26	28256271771	job	✅ Passed	—	—
Jun 25	28190877198	job	✅ Passed	—	—

First observed failure: 2026-06-27. Last known passing: 2026-06-26.

Failure Cluster

Server start timeout — HF weight download stall during model load

Facts (verified observations, no interpretation)

• [FACT] Server launched with --model-path amd/DeepSeek-R1-MXFP4-Preview ... --tp 4 and CI_OFFLINE: Launching server HF_HUB_OFFLINE=0 (online download allowed).
• [FACT] TP0 logged: Local HF snapshot at /sgl-data/hf-cache/hub/models--amd--DeepSeek-R1-MXFP4-Preview/snapshots/a7e208e3.../ has no files matching ['*.safetensors']; will attempt download. (17:57:36).
• [FACT] py-spy native dump at 18:02:19 shows all workers blocked in the download path:

  download_weights_from_hf (model_loader/weight_utils.py:554)
  _prepare_weights (model_loader/loader.py:473)
  _get_weights_iterator (model_loader/loader.py:547)
  do_load_weights (deepseek_common/deepseek_weight_loader.py:158)
  load_weights (deepseek_v2.py:2851)
  

• [FACT] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) fired repeatedly from 18:02:19 through 18:52:25 — workers stuck >50 min, never progressing past weight download.
• [FACT] Last weight-load progress line before the stall was Load weight begin. avail mem=278.x GB (17:57:36); no Load weight end ever printed.
• [FACT] Earlier infra noise in the same job: Local docker registry pull failed; falling back to public registry and registry Client.Timeout exceeded (17:48:20), indicating intermittent network/registry trouble on this runner.
• [FACT] aiter version unchanged from a working baseline (dev image preserved); job passed Jun 25 and Jun 26 on the same workflow.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure: the model weights were not present in the runner's local /sgl-data/hf-cache snapshot (no *.safetensors), forcing a live HuggingFace download that stalled or was too slow to complete within 3600 s.
  • Supporting: explicit log has no files matching ['*.safetensors']; will attempt download; all TP workers parked in download_weights_from_hf; no download-progress or Load weight end line ever appears; same job passed the two prior nights with identical sglang/aiter code paths.
  • Disconfirming: cannot see HF HTTP error/rate-limit lines in the captured log (py-spy Failed to get stack traces on TP0 limits visibility), so a hang vs. slow-download distinction is unconfirmed.
  • How to verify: pre-populate /sgl-data/hf-cache with the MXFP4 snapshot (or set HF_HUB_OFFLINE=1 with a warm cache) and re-run the job; check runner egress/HF reachability at 2026-06-27 ~17:57 UTC.
• [LOW] sglang code regression in the weight loader between Jun 26 and Jun 27.
  • Supporting: failure is new today.
  • Disconfirming: the stack is parked in network download (weight_utils.py:554), not in parsing/sharding logic; the missing-safetensors message points to a cache/network condition, not a code defect; a code regression would typically error fast, not hang for 60 min.
  • How to verify: git log --oneline <Jun26_pass_sha>..592f6c84 -- python/sglang/srt/model_loader/ — inspect any change to download_weights_from_hf / lock handling.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting the weight-download stall. Searched open PRs for DeepSeek-R1-MXFP4 (26 hits, all feature/CI coverage, e.g. #29290, #29419 "ci(amd): make sgl-data mount opt-in") and HF cache download weights (16 hits, none address this hang). #29419 touches the sgl-data cache mount and may be tangentially relevant to cache availability, but is not a fix for this failure.

Suggested Triage Steps

• Confirm whether /sgl-data/hf-cache had the amd/DeepSeek-R1-MXFP4-Preview safetensors on this runner at run time; if missing, pre-seed the cache before the nightly or gate the test behind a cache-warm step.
• Check the runner's HF/network egress around 2026-06-27 17:57–18:57 UTC (the registry pull also fell back to public registry earlier — possible runner-wide network flake).
• Re-run the job once; if it passes with a warm cache, classify as transient infra and track recurrence.
• If it recurs with the cache present, bisect python/sglang/srt/model_loader/ over <Jun26_pass_sha>..592f6c84 to rule out a loader regression.

Status (factual, no priority assignment)

• Persistence: 1 run (new on 2026-06-27); passed 2026-06-25 and 2026-06-26.
• Scope: 1 test file in 1 job (nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720) in nightly-test-amd-rocm720.yml. No evidence of spread to sibling jobs from this analysis.
• Blocked work: DeepSeek-R1 MXFP4 TP4 accuracy signal on MI35x ROCm 7.2 nightly.
• In-flight fix: ❌ none found — likely an infra/cache issue; warm the HF cache and re-run before opening a code PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:33 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

nightly-test-1-gpu-unit-rocm720 — failed step(s): Nightly Unit Test ROCm 7.2 (1-GPU)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, matches Dockerfile default; [CI-AITER-CHECK] reported "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/bench_fn/test_bench_serving_functionality.py	TestBenchServingCustomHeaders.test_custom_headers_sent_to_server	TIMEOUT after 900s (est. 300s); process killed by kill_process_tree	Log

Failure Summary

The unit suite reported 9/10 passed; the single failure is test/registered/bench_fn/test_bench_serving_functionality.py, which timed out after 900s and was force-killed. Within the file, execution reached TestBenchServingCustomHeaders.test_custom_headers_sent_to_server: the mock HTTP server logged one GET /v1/models HTTP/1.1 200 at 18:00:01, then produced no further output for ~15 minutes until the watchdog killed the process at 18:14:58. No exception, stack trace, or GPU fault was printed.

Regression Status

New failure today (single occurrence — likely flaky/infrastructure). The same job nightly-test-1-gpu-unit-rocm720 succeeded on the prior three completed scheduled runs, and this exact test file passed quickly each time.

Recent history of test_bench_serving_functionality.py in job nightly-test-1-gpu-unit-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	nightly-test-1-gpu-unit-rocm720	❌ Failed	test_custom_headers_sent_to_server	TIMEOUT 900s
Jun 26	28256271771	nightly-test-1-gpu-unit-rocm720	✅ Passed (52s)	—	—
Jun 25	28190877198	nightly-test-1-gpu-unit-rocm720	✅ Passed	—	—
Jun 24	28119401197	nightly-test-1-gpu-unit-rocm720	✅ Passed	—	—

First observed failure: 2026-06-27. Last known passing: 2026-06-26 (52s). The job's overall pass/fail was used only as a cross-check; the verdict is from the Test File Status column.

Failure Cluster

Benchmark functionality test hang/timeout during tokenizer setup (no GPU, mock server)

Facts (verified observations, no interpretation)

• Test file timed out at exactly 900s (elapsed=900, estimated_time=300.0), First attempt failed with exit code 255 (log lines 4316–4321, 8423).
• Last output before the hang: GET /v1/models HTTP/1.1 200 at 2026-06-27T18:00:01 (log line 4310). Next output is kill_process_tree at 2026-06-27T18:14:58 (line 4311) — ~15 min of complete silence.
• Code path: run_benchmark() calls model_url = f"{base_url}/v1/models" (serving.py:1947), then tokenizer = get_tokenizer(tokenizer_id) (serving.py:2036), then get_dataset(args, ...) (serving.py:2037). The /v1/models call completed; the /generate call (which the mock server would have logged) never arrived — so the hang is between these, in get_tokenizer("gpt2") or get_dataset (test config: tokenizer="gpt2", dataset_name="random").
• get_tokenizer("gpt2") is a HuggingFace Hub download of a repo not otherwise cached in this image.
• The same job passed test_bench_serving_functionality.py in 52s on 2026-06-26 (10/10), verified from log.
• Commit window c98d31143d..592f6c849b = 32 commits; none touch python/sglang/benchmark/ (verified via git log -- python/sglang/benchmark/).
• Other tests in this same job that download models (LoRA suite on Qwen3-0.6B, test_soft_watchdog) all passed in the same run, indicating general model/network access was functional.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Transient network/HuggingFace Hub stall while downloading the gpt2 tokenizer in get_tokenizer() (serving.py:2036), hanging with no timeout.
  • Supporting: 15-min silence begins immediately after /v1/models returns, which is exactly where get_tokenizer("gpt2") runs next; no benchmark code changed in the window; the file passed in 52s the previous day.
  • Disconfirming: other tests in the same job that pull models from HF (Qwen3-0.6B) passed in the same run, so general HF connectivity was up; no network error was printed.
  • How to verify: re-run the job; if it passes, confirms a transient. To repro deterministically, run python3 test/registered/bench_fn/test_bench_serving_functionality.py in container rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260627 with HF cache cleared and observe whether get_tokenizer("gpt2") blocks.
• [SPECULATION] Hang in get_dataset random-dataset generation.
  • Supporting: it is the step right after tokenizer load.
  • Disconfirming: random dataset generation is CPU-only and fast; a 15-min hang there is implausible.
  • How to verify: add per-step timing/logging in run_benchmark between /v1/models, get_tokenizer, and get_dataset.

No hypothesis reaches MEDIUM confidence; the silent 15-min hang with no code change in the regression window and a prior-day 52s pass points to a flaky/infrastructure stall rather than a code regression.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for bench_serving+timeout and test_bench_serving_functionality; no matching fix PR. (Queries: repo:sgl-project/sglang is:pr is:open bench_serving timeout and ...test_bench_serving_functionality.)

Suggested Triage Steps

• Re-run job nightly-test-1-gpu-unit-rocm720 for run 28296988041; a green re-run confirms a transient/flaky network stall and closes this out.
• If it recurs, repro locally: python3 test/registered/bench_fn/test_bench_serving_functionality.py in container rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260627, watching whether get_tokenizer("gpt2") (serving.py:2036) blocks; pre-seed the gpt2 tokenizer into the image HF cache to remove the network dependency.
• Consider a per-test timeout/offline-cache for gpt2 in this functionality test so a download stall fails fast instead of consuming the full 900s file budget (test-hardening, not a fix for the underlying stall).
• Bisect c98d31143d..592f6c849b is low value here since no commit touches python/sglang/benchmark/; defer unless the failure reproduces consistently.

Status (factual)

• Persistence: First failure on 2026-06-27; passed on the prior 3 completed scheduled runs (Jun 24–26). Not yet recurring.
• Scope: 1 test file in 1 job (nightly-test-1-gpu-unit-rocm720); the other 9 files in the suite passed.
• Blocked work: Blocks the rocm720 nightly unit signal for this run only (suite reports Fail due to this single timeout).
• In-flight fix: ❌ none found.

nightly-8-gpu-deepseek-v31-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.1)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, dev/patched build detected so no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_deepseek_v31_eval_amd.py	N/A (whole-file runner timeout — server never finished startup)	TIMEOUT after 3600s → exit code 255	Log

Failure Summary

The Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.1) step ran test_deepseek_v31_eval_amd.py, which timed out after the 3600 s budget and exited with code 255 (step 6). The server never reached readiness because every TP rank was still blocked inside HuggingFace weight download for deepseek-ai/DeepSeek-V3-0324 when the timeout fired. No accuracy assertion was ever evaluated.

Regression Status

New failure for this test file (infrastructure/network, not a code regression). The same job passed in the previous three completed scheduled runs.

Recent history of test_deepseek_v31_eval_amd.py in job nightly-8-gpu-deepseek-v31-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	nightly-8-gpu-deepseek-v31-rocm720	❌ Failed	N/A	TIMEOUT after 3600s (weight download stalled)
Jun 26	28256271771	nightly-8-gpu-deepseek-v31-rocm720	✅ Passed	—	—
Jun 25	28190877198	nightly-8-gpu-deepseek-v31-rocm720	✅ Passed	—	—
Jun 24	28119401197	nightly-8-gpu-deepseek-v31-rocm720	✅ Passed	—	—

First observed failure: 2026-06-27 17:47 UTC. Last known pass: 2026-06-26 (run 28256271771). The parent runs on Jun 24–26 show overall failure, but that was due to other jobs in the matrix — this specific job/test passed.

Failure Cluster

HuggingFace weight download stall during model load (DeepSeek-V3-0324)

Facts (verified observations, no interpretation)

• Step 6 Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.1) = failure; step 7 Performance Test = skipped. Job conclusion failure, duration 17:47:50→18:56:14 UTC (~68 min).
• Runner log: ✗ TIMEOUT: .../test_deepseek_v31_eval_amd.py after 3600 seconds (line 9092), then First attempt failed with exit code 255 / Exit code 255 indicates test failure (not network issue), not retrying / ##[error]Process completed with exit code 255 (lines 9107–9109).
• First py-spy watchdog dump at 18:01:47 (300 s soft Scheduler watchdog timeout). All TP ranks stuck in model weight loading:
  • TP2 (PID 3327): download_weights_from_hf (weight_utils.py:593) → ci_download_with_validation_and_retry (ci_weight_validation.py:1955) → snapshot_download → thread_map (actively downloading).
  • TP4 (PID 3329): download_weights_from_hf (weight_utils.py:554) → filelock _poll_until_acquired (filelock/_api.py:552) (waiting on the shared download lock held by TP2).
• Final py-spy dump at 18:51 (10 min before timeout) shows TP2 STILL in snapshot_download → hf_hub_download → xet_get (file_download.py:566) on an active ThreadPoolExecutor — i.e. the download had not completed after ~55 min and never progressed past weight fetch.
• Earlier benign note (not the cause): Failed to load generation config for deepseek-ai/DeepSeek-V3-0324 ... Proceeding without generation config.
• This is not an AITER Scout run; aiter version unchanged from the Dockerfile dev default (no override).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] HuggingFace weight download for deepseek-ai/DeepSeek-V3-0324 stalled/was too slow (xet backend), so the 8-GPU server never finished startup within the 3600 s budget.
  • Supporting: py-spy stacks at 18:01, 18:06…18:51 all show the rank holding the filelock parked in snapshot_download/hf_hub_download/xet_get; sibling ranks blocked on the download filelock. No forward progress in ~55 min of weight fetch. The accuracy harness never ran.
  • Disconfirming: the runner banner prints Exit code 255 indicates test failure (not network issue) — but that is a generic harness label keyed only on the exit code, not an actual network probe, so it does not contradict the network/IO root cause. (none other found)
  • How to verify: re-run the job; check whether weights are pre-cached on the runner; inspect HF/xet throughput at run time. Compare to the Jun 26 pass log where the same download completed and the server reached warmup.
• [LOW] sglang code regression in the model-load path between c98d31143d (Jun 26 pass) and 592f6c849b (Jun 27 fail).
  • Supporting: HEAD commit 592f6c84 ("Split 2/4: mem-cache / HiCache / sparse KV pool", #28713) touches memory-cache code in the load region.
  • Disconfirming (strong): the stack is parked entirely in third-party huggingface_hub/filelock/xet download code, not in any sglang load logic; the symptom is a network/IO stall, not a crash or assertion. A code regression would not manifest as an indefinite xet_get. Hypothesis is very unlikely.
  • How to verify: only worth pursuing if a re-run with pre-cached weights still hangs.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for weight download timeout (16 hits, all unrelated: MoE/FP8 load bugs, CI sandbox PRs). No open PR addresses HF/xet download stalls in nightly AMD CI. This is an infra/network condition, not a clear code defect.

Suggested Triage Steps

• Re-run nightly-8-gpu-deepseek-v31-rocm720 once; if it passes, this was a transient HF/xet download stall — no code action needed.
• Confirm whether deepseek-ai/DeepSeek-V3-0324 weights are expected to be pre-cached on the mi30x runner; if so, investigate why the cache missed and a fresh download was triggered (a cold cache is the likely trigger of the 1 h overrun for a model this large).
• If download stalls recur, consider raising the per-test timeout for this 8-GPU DeepSeek-V3.1 accuracy test, or pre-staging weights, or pinning the hf_transfer/xet path — discuss with CI owners (triage, not a directive).
• Only if a re-run with warm weights still hangs: bisect c98d31143d..592f6c849b focusing on the model-loader path — currently [LOW] likelihood given the third-party download stack.

Status

• Persistence: 1 run (first failure 2026-06-27 17:47 UTC). Passed the prior 3 scheduled runs (Jun 24–26).
• Scope: this job/test only (nightly-8-gpu-deepseek-v31-rocm720, test_deepseek_v31_eval_amd.py); Performance Test step skipped as a consequence.
• Blocked work: nightly DeepSeek-V3.1 accuracy + perf signal on AMD ROCm 7.2 for this run.
• In-flight fix: ❌ none found — likely transient infra/network (HF/xet weight download); re-run before treating as a code issue.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen3_moe_eval_mi35x.py	test_qwen3_moe_accuracy (model Qwen/Qwen3-30B-A3B)	Memory access fault by GPU node → Fatal Python error: Aborted → scheduler crash exit -6	Log
test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy (model openai/gpt-oss-120b)	httpx.HTTPStatusError: Client error '429 Too Many Requests' during HF weight download	Log
test/registered/amd/accuracy/mi35x/test_qwen3_coder_next_eval_mi35x.py	test_qwen3_coder_next_accuracy (model Qwen/Qwen3-Coder-Next)	httpx.HTTPStatusError: Client error '429 Too Many Requests' during HF weight download	Log

Failure Summary

The job ran 3 accuracy test files (0/3 passed). test_qwen3_moe_eval_mi35x.py crashed with a GPU memory access fault during the GSM8K eval forward pass on Qwen/Qwen3-30B-A3B (scheduler subprocess aborted with exit -6). The other two files (test_gpt_oss_eval_mi35x.py, test_qwen3_coder_next_eval_mi35x.py) failed on their second model only, because the HF Hub returned HTTP 429 (rate limit) while downloading weights — their first model passed accuracy in each case (gpt-oss-20b 0.525, Qwen3-Coder-Next basic 0.970). These are two unrelated failure modes.

Regression Status

• test_qwen3_moe_eval_mi35x.py (test_qwen3_moe_accuracy): Recurring / intermittent failure — GPU memory access fault. Last passed Jun 23 (3/3); failed Jun 22, 24, 25, 26, 27.
• test_gpt_oss_eval_mi35x.py & test_qwen3_coder_next_eval_mi35x.py: New today (Jun 27) — infrastructure HF 429 rate-limit, NOT a code regression. Both passed Jun 25 and Jun 26.

Recent history of job nightly-accuracy-8-gpu-mi35x-rocm720 (branch main, event schedule, completed runs only):

Date	Run	qwen3_moe	gpt_oss	qwen3_coder_next	Note
Jun 27	28296988041	❌ GPU mem fault	❌ HF 429	❌ HF 429	0/3 — this run
Jun 26	28256271771	❌ GPU mem fault	✅	✅	2/3
Jun 25	28190877198	❌ GPU mem fault	✅	✅	2/3
Jun 24	28119401197	❌ GPU mem fault	✅	✅	2/3
Jun 23	28046795462	✅	✅	✅	3/3 (last all-green)
Jun 22	27975850577	❌	✅	❌ (basic)	—

Failure Cluster

Two distinct clusters in this job:

1. GPU memory access fault during Qwen3-30B-A3B eval (qwen3_moe).
2. HF Hub 429 rate-limit during model weight download (gpt_oss-120b + Qwen3-Coder-Next).

Facts (verified observations)

• test_qwen3_moe crash sequence (verbatim): Memory access fault by GPU node-2 … Reason: Unknown. (×4 nodes) → GPU coredump: Directory "/coredumps not writable → Fatal Python error: Aborted → Subprocess scheduler_0 (pid=20296) crashed with exit code -6. Top non-fatal thread frame: python/sglang/srt/utils/watchdog.py:147.
• Immediately preceding the fault: many [aiter] … not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 lines (bf16 GEMM, e.g. M:789,N:768,K:2048). This is log-line proximity, not proven causation.
• test_gpt_oss failing model trace: httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/openai/gpt-oss-120b' raised from download_weights_from_hf → weight_utils.py:568. Message: 429 Too Many Requests: you have reached your 'api' rate limit … quota of 3000 api requests per 5 minutes period … (current: team).
• test_qwen3_coder_next failing model trace: identical 429 for https://huggingface.co/api/models/Qwen/Qwen3-Coder-Next.
• 4 distinct 429 Too Many Requests occurrences in the log.
• Accuracy of models that did run today: gpt-oss-20b 0.525 (thr 0.47) ✅, Qwen3-Coder-Next basic 0.970 (thr 0.9) ✅ — so the 429s are download-time, not accuracy regressions.
• aiter version unchanged across passing (Jun 23) and failing runs (dev image g7d604afe5).
• Not an amd-aiter-scout.yml run (no aiter override; AITER_COMMIT_OVERRIDE empty) — Failure Origin / Baseline A/B check not applicable.

Hypothesised Causes (with confidence)

Cluster 1 — qwen3_moe GPU memory access fault

• [LOW] Intermittent GPU memory access fault in the Qwen3-30B-A3B (MoE) forward path on mi35x/gfx950, possibly tied to a commit in the ed26a109ee..d6aacd2801 window (55 commits, last-pass Jun 23 → first-fail-of-streak Jun 24). AMD/MoE-touching candidates in window: #28084 "[AMD] Fuse topk padded-token masking into a single Triton kernel", #25090 "[AMD] Support triton backend decode context parallel for Qwen3.5", #27833 "[AMD] Enable BCG on ROCm + route aiter prefill via MHA".
  • Supporting: fault occurs mid-eval (server loaded, forward pass running); these PRs touch AMD MoE / attention capture paths exercised by this model.
  • Disconfirming: the test is intermittent — it PASSED Jun 23 (inside/after this window) and FAILED Jun 22 (before it), which is inconsistent with a single deterministic regression commit. No commit diff has been tied to the faulting kernel.
  • How to verify: re-run test_qwen3_moe_eval_mi35x.py on an MI35x (gfx950) runner several times to measure flake rate; if reproducible, bisect ed26a109ee..d6aacd2801 reverting AMD-MoE candidates one at a time.

Cluster 2 — HF 429 rate limit

• [FACT] gpt-oss-120b and Qwen3-Coder-Next failed because HF Hub returned 429 during weight download (3000 api req / 5 min quota exhausted). Purely an environment/infra condition; not caused by any sglang or aiter change. The runner appears not to use a local HF cache/mirror for these two large models, so it hit the live API.

In-flight Fix Check

• qwen3_moe GPU fault: In-flight fix: ❌ none found — searched open PRs Qwen3-30B+memory+fault (1 hit, #25692, NVIDIA/unrelated) and qwen3_moe+mi35x (0 hits).
• HF 429: In-flight fix: ❌ none found — infra/runner config issue, not a code fix.

Suggested Triage Steps

• qwen3_moe (real CI signal): re-run the test ~5× on an MI35x runner to quantify flakiness; if consistently reproducible, bisect ed26a109ee..d6aacd2801 starting with the AMD-MoE candidates (#28084, #25090, #27833); enable /coredumps (currently not writable) on the runner to capture a GPU coredump for the faulting kernel.
• HF 429 (infra): pre-stage openai/gpt-oss-120b and Qwen/Qwen3-Coder-Next in the runner's local HF cache / internal mirror and set HF_HUB_OFFLINE=1 for these accuracy tests, or supply an authenticated higher-quota HF_TOKEN, or add download retry/back-off on 429. This will clear the two new failures independently of the qwen3_moe issue.

Status

• Persistence: test_qwen3_moe_eval_mi35x.py GPU memory access fault — failed 4 of last 5 completed scheduled runs (last green Jun 23); intermittent. test_gpt_oss + test_qwen3_coder_next 429s — first seen today (Jun 27), both green Jun 25–26.
• Scope: single job nightly-accuracy-8-gpu-mi35x-rocm720 (workflow nightly-test-amd-rocm720.yml). The 429 cluster may affect any AMD nightly job downloading these models concurrently in the same 5-min window.
• Blocked work: AMD MI35x (gfx950) ROCm 7.2 nightly accuracy signal for Qwen3-30B-A3B is unreliable; GPT-OSS-120B and Qwen3-Coder-Next accuracy unverified today.
• In-flight fix: ❌ none found for either cluster.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: 592f6c84 ("[minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool" #28713)
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5) — source: dev/patched image preserved ([CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	TestDeepSeekV32EvalMI35x.test_deepseek_v32_accuracy	✗ TIMEOUT after 3600s — server never finished loading model weights	Log

Failure Summary

The single test in this job (test_deepseek_v32_accuracy, model deepseek-ai/DeepSeek-V3.2 (basic), TP=8) timed out after the harness wrapper's 3600s limit. The server process started at 17:57, finished torch-distributed init and reached Load weight begin at 17:57:30, then spent the next ~55 minutes in HuggingFace weight download and never reached "Load weight end". The test-runner wrapper killed it at 18:56 (elapsed=3600, exit code 255).

Regression Status

Known recurring failure (test file Never-passed in queryable window) — but with a NEW failure mode today. The test file has failed in every completed scheduled run in the lookback window, but the prior failures were accuracy-below-threshold (model loaded and benchmarked fine); today's failure is a weight-download timeout (server never came up). These are two distinct clusters on the same chronically-red test.

Recent history of test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Failure mode	Detail
Jun 27	28296988041	❌ Failed	Weight-download timeout (NEW)	TIMEOUT after 3600s, stuck in xet_get
Jun 26	28256271771	❌ Failed	Accuracy < threshold	accuracy=0.890 threshold=0.93 (elapsed 2298s)
Jun 25	28190877198	❌ Failed	Accuracy < threshold	accuracy=0.835 threshold=0.93 (elapsed 2069s)
Jun 24	28119401197	❌ Failed	Accuracy < threshold	elapsed 2287s
Jun 23	28046795462	❌ Failed	Accuracy (overall job red)	—

First observed failure for this test file: ≥ Jun 23 (no green run in queryable window → Never-passed). Today's download-timeout symptom: first seen Jun 27.

Failure Cluster

DeepSeek-V3.2 weight download timeout during server launch (MI35x 8-GPU)

Facts (verified observations)

• Load weight begin. avail mem=276.88 GB at 17:57:30 (all 8 ranks). No "Load weight end" anywhere in the log.
• At 18:52:16–17 (≈55 min later) the soft watchdog fired (Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True)) and dumped py-spy stacks:
  • TP0 (active): xet_get (huggingface_hub/file_download.py:566) ← _download_to_tmp_and_move ← hf_hub_download ← snapshot_download ← ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) ← download_weights_from_hf (weight_utils.py:593). TP0 was the active downloader.
  • TP3 / TP5 / others (idle): blocked in filelock _poll_until_acquired (filelock/_api.py:552) ← download_weights_from_hf (weight_utils.py:554) — i.e. waiting on the rank-0 download lock. Expected multi-rank coordination, not an independent fault.
• Harness wrapper terminated at 18:56:37: ✗ TIMEOUT … after 3600 seconds, First attempt failed with exit code 255, Exit code 255 indicates test failure (not network issue), not retrying.
• Timeout-config mismatch [FACT]: the test's own ModelConfig.timeout=5400 and --watchdog-timeout 1200 (test file lines 72, 84-85; confirmed watchdog_timeout=1200.0 in server_args), but the per-file runner wrapper enforced a hard 3600s limit (estimated_time=5400.0, killed at elapsed=3600). The wrapper limit is shorter than the model's expected load+eval budget.
• In prior runs the same weight load + full GSM8K eval completed within ~2000–2300s total, implying weights are normally cached on the runner; today's run paid a full (or stalled) fresh download.
• No HIP/CUDA error, no kernel fault, no aiter call in the hang path. aiter version unchanged (dev image preserved).

Hypothesised Causes (with confidence)

• [MEDIUM] Slow or stalled HuggingFace weight download (cold/partial HF cache on this runner) pushed model load past the 3600s wrapper limit.
  • Supporting: TP0 py-spy stack is actively inside xet_get 55 min into the run; no "Load weight end"; prior runs that hit a warm cache finished load+eval in ~2000s.
  • Disconfirming: cannot tell from a single py-spy snapshot whether bytes were still flowing (true network stall) vs. merely slow; no download-progress / MB/s lines in the log to confirm forward progress.
  • How to verify: re-run the job and watch for download progress; check runner HF cache state for deepseek-ai/DeepSeek-V3.2; pre-stage/warm the model on the MI35x runner before the eval.
• [LOW] Wrapper timeout (3600s) is mis-sized vs. the model's own 5400s budget, so a cold-cache run is guaranteed to fail even if the download would eventually succeed.
  • Supporting: est_time=5400 in register_amd_ci, ModelConfig.timeout=5400, but wrapper killed at exactly 3600s.
  • Disconfirming: this is a contributing config issue, not the proximate trigger — a warm cache still passes within 3600s.
  • How to verify: confirm the per-file wrapper timeout source in the runner script; align it with the 5400s est_time.
• [SPECULATION] Today's HEAD commit caused the hang. No evidence — 592f6c84 (#28713) touches mem-cache/HiCache/sparse-KV-pool code, which is not in the download_weights_from_hf → xet_get hang path. Not aiter-related (hang is in HF download, no aiter frames).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DeepSeek-V3.2 + accuracy + mi35x (7 results, none address this test's accuracy or download timeout) and weight download timeout (16 results, all unrelated AMD/MoE/CI items). Neither the chronic accuracy gap nor today's download timeout has a matching open PR.

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 once; if it loads from a warm HF cache and only the accuracy gap remains, today's timeout was a transient cold-cache/network event.
• Verify the MI35x runner's HF cache for deepseek-ai/DeepSeek-V3.2 and consider pre-staging weights so the eval doesn't pay download cost inside the timed window.
• Reconcile the per-file wrapper timeout (3600s) with the test's declared est_time=5400 / ModelConfig.timeout=5400 so a legitimately-slow load isn't killed early.
• Track the accuracy-below-threshold failure (0.835–0.890 vs 0.93) separately — it is the persistent, code/model-side issue that will remain once the download/timeout flake is resolved; bisect or investigate DeepSeek-V3.2 DSA tilelang backend accuracy on MI35x for that one.

Status

• Persistence: test file red in every completed scheduled run since ≥ Jun 23 (Never-passed in window). Today's download-timeout symptom is new (first seen 2026-06-27 18:56 UTC); the prior 4+ runs were accuracy-below-threshold.
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720), 1 test file/function. Note the sibling …-mtp-rocm720 accuracy job also failed Jun 26 but passed Jun 23–25 — separate signal.
• Blocked work: DeepSeek-V3.2 basic accuracy signal on AMD MI35x (ROCm 7.2) — no green baseline.
• In-flight fix: ❌ none found in open PRs.

nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720 — failed step(s): Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5, source: Dockerfile default — dev image preserved, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch	Exception: Server process exited with code -9 (root: AssertionError: All of them must not be None at dsa_backend.py:721)	Log

Failure Summary

The DeepSeek-V3.2 MTP perf server (TP8, attention_backend='dsa', EAGLE spec decode) crashed during benchmarking on MI35x/ROCm720. All 8 TP schedulers raised AssertionError: All of them must not be None at dsa_backend.py:721 inside the is_draft_extend_v2() branch of init_forward_metadata, reached via eagle_worker_v2._draft_extend_for_decode → base_spec_worker.prepare_for_draft_extend. The schedulers died, the server was SIGKILLed (exit -9), and test_bench_one_batch failed.

Regression Status

New regression. The exact job nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720 passed in the prior three scheduled runs and fails for the first time in this run — which is the first scheduled run to include 9214b933 (#29413, landed 2026-06-27 06:53 UTC).

Recent history of test_deepseek_v32_mtp_perf_mi35x.py (test_bench_one_batch) in job nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 27	28296988041	❌ Failed	test_bench_one_batch	AssertionError @ dsa_backend.py:721 → exit -9
Jun 26	28256271771	✅ Passed	—	—
Jun 25	28190877198	✅ Passed	—	—
Jun 24	28119401197	✅ Passed	—	—

First observed failure: 2026-06-27. Last known pass: 2026-06-26 (28256271771, head c98d31143d).

Failure Cluster

AssertionError "All of them must not be None" in DSA draft-extend-v2 eager metadata init

Facts (verified observations)

• Stack trace (verbatim, all 8 TP ranks): eagle_worker_v2.py:1171 forward_batch_generation → eagle_worker_v2.py:850 _draft_extend_for_decode → base_spec_worker.py:163 prepare_for_draft_extend → dsa_backend.py:721 init_forward_metadata → AssertionError: All of them must not be None.
• The failing assert (current source dsa_backend.py:717-722) requires extend_seq_lens_cpu, extend_seq_lens, and extend_prefix_lens_cpu to all be non-None in the is_draft_extend_v2() branch.
• 9214b933 (#29413) added needs_cpu_seq_lens = False to DeepseekSparseAttnBackend (dsa_backend.py:309). decide_needs_cpu_seq_lens (overlap_utils.py:21) uses this flag to skip computing the host CPU mirrors → extend_*_cpu become None. Before this commit DSA inherited the base default needs_cpu_seq_lens = True (base_attn_backend.py:90).
• The same commit migrated the graph-replay draft-extend-v2 path to derive lengths from the static page-table width (metadata.page_table_1.shape[1]) instead of seq_lens_cpu, so it no longer needs the CPU mirrors — but the eager init_forward_metadata draft-extend-v2 branch (line 717-722) was left asserting they exist.
• Draft-extend CUDA graph is gated: supports_cuda_draft_extend_graph = (_is_cuda or _is_musa) and graph_supported_backend (eagle_worker_v2.py:447-449). On ROCm/HIP both _is_cuda and _is_musa are False, so draft-extend falls into the eager path and hits the assert.
• Config: attention_backend='dsa', speculative_algorithm='EAGLE', speculative_num_draft_tokens=4, TP8, model DeepSeek-V3.2.
• aiter unchanged across passing/failing runs (Dockerfile default dev build v0.1.14rc1.dev240+g7d604afe5); not a scout run.

Hypothesised Causes (with confidence)

• [HIGH] 9214b933 (#29413, "[DSA] Enable draft-extend CUDA graph") set needs_cpu_seq_lens=False on the DSA backend (nulling extend_*_cpu host mirrors) and moved only the graph draft-extend path off those mirrors, leaving the eager draft-extend-v2 branch (dsa_backend.py:717-722) still asserting them. On ROCm the draft-extend graph is disabled (_is_cuda or _is_musa is False), so DSA draft-extend always takes the eager path and the assert fires.
  • Supporting: commit is the sole DSA/eagle change introducing needs_cpu_seq_lens=False; lands exactly in the regression window c98d31143d..592f6c84; the modified files + exact branch appear in the failing stack trace; job passed in the run immediately before the commit; the commit comment ("DSA is CUDA-only") shows the author did not account for ROCm exercising DSA without the graph path.
  • Disconfirming: (none found) — the eager path on CUDA is masked because draft-extend uses the graph there, which is why CUDA CI did not catch it; ROCm is the exposed configuration.
  • How to verify: on MI35x/ROCm720 in the CI image, git revert 9214b933 on a branch and re-run test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py; expect the server to start and the bench to pass. Alternatively confirm _is_cuda/_is_musa are both False on this runner.
• [LOW] ee77a7d3 (#29379, "DSA: size cudagraph page_table to req_to_token width") also lands in the window and touches dsa_backend.py.
  • Supporting: DSA change in the regression window.
  • Disconfirming: it modifies cudagraph page_table sizing, not the needs_cpu_seq_lens flag or the eager draft-extend-v2 assert path; does not explain None CPU mirrors.
  • How to verify: only if reverting #29413 does not resolve the assert.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for DSA draft extend and extend_prefix_lens_cpu; no open PR addresses this eager draft-extend-v2 assertion. (#29499 "[DSA] Optimize DSA CUDA graph replay metadata generation" is DSA-graph-related but does not touch the eager assert path.)
• API queries run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+DSA+draft+extend and ...+extend_prefix_lens_cpu.

Suggested Triage Steps

• Bisect/confirm on the regression window c98d31143d..592f6c84; only two DSA commits are present, with #29413 the prime suspect.
• Revert 9214b933 on a branch and re-run test_deepseek_v32_mtp_perf_mi35x.py on MI35x/ROCm720 to confirm.
• If confirmed, candidate fix direction: make the eager is_draft_extend_v2() branch in dsa_backend.py derive lengths from GPU seq_lens/page-table width (as the graph path now does) instead of asserting the nulled extend_*_cpu mirrors — i.e. extend the needs_cpu_seq_lens=False handling to the eager ROCm path, not just the graph path.

Status

• Persistence: First failure 2026-06-27 (this run); 1/1 completed run since 9214b933 landed. Prior 3 scheduled runs passed.
• Scope: nightly-perf-8-gpu-mi35x-deepseek-v32-mtp-rocm720; all 8 TP ranks crash → server SIGKILL. Likely affects any ROCm DSA + EAGLE draft-extend path (other DeepSeek-V3.2 MTP DSA jobs on AMD may share it).
• Blocked work: DeepSeek-V3.2 MTP perf signal on AMD MI35x/ROCm720.
• In-flight fix: ❌ none found — needs a new PR.

nightly-accuracy-8-gpu-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5f (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — matches Dockerfile default; not relevant to this failure)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gpt_oss_eval_amd.py	test_gpt_oss_accuracy (model lmsys/gpt-oss-120b-bf16)	AssertionError: Failed models: ['lmsys/gpt-oss-120b-bf16']	Log

Failure Summary

The 8-GPU GPT-OSS accuracy suite tests two models. lmsys/gpt-oss-20b-bf16 passed (accuracy=0.545, threshold=0.45). lmsys/gpt-oss-120b-bf16 produced no score and failed the assertion. All 8 TP scheduler ranks for the 120b server hung in init_tokenizer → hf_hub_download, stuck on a Hugging Face Hub WeakFileLock (filelock._poll_until_acquired), triggering soft scheduler watchdog timeouts at 18:05 and 18:10 UTC (log).

Regression Status

New regression for this test file (job-level: this accuracy job passed the previous 3 scheduled runs; the daily workflow's overall failure came from other jobs).

Recent history of test_gpt_oss_eval_amd.py (test_gpt_oss_accuracy, model gpt-oss-120b-bf16) in job nightly-accuracy-8-gpu-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	job	❌ Failed	test_gpt_oss_accuracy (120b)	120b stuck on HF Hub filelock → 0 score
Jun 26	28256271771	job	✅ Passed	—	120b accuracy=0.810 (cache hit)
Jun 25	28190877198	job	✅ Passed	—	—
Jun 24	28119401197	job	✅ Passed	—	—

First observed failure: Jun 27 2026. Last known passing: Jun 26 2026 (run 28256271771).

Failure Cluster

HF Hub filelock hang during model tokenizer download (scheduler watchdog timeout)

Facts (verified observations)

• 20b passed, 120b failed. Today: accuracy=0.545 threshold=0.45 ✅ PASS for 20b (log:6863); AssertionError: Failed models: ['lmsys/gpt-oss-120b-bf16'] (log:6857).
• Cache present for 20b, absent for 120b (today). Found local HF snapshot for lmsys/gpt-oss-20b-bf16 ... skipping download (log:3872). There is no equivalent "Found local HF snapshot for ...120b" line and no [CI Download] lock line for 120b anywhere in today's log.
• Yesterday 120b cache was warm. Jun 26 log: Found local HF snapshot for lmsys/gpt-oss-120b-bf16 at .../snapshots/3609e677...; skipping download → accuracy=0.810 threshold=0.75 ✅ PASS, file elapsed 671s.
• Stuck call stack (all 8 TP ranks): init_tokenizer (scheduler.py:705) → get_tokenizer → from_pretrained → cached_file (transformers/utils/hub.py:278) → hf_hub_download (file_download.py:1016) → WeakFileLock (huggingface_hub/utils/_fixes.py:99) → _poll_until_acquired (filelock/_api.py:552) (log:6807-6829). Server launched with HF_HUB_OFFLINE=0.
• Watchdog soft-timeouts at 18:05:03 and 18:10:03 UTC on TP0-TP7 (watchdog_timeout=300, soft=True); the 120b model never produced output → file elapsed 1086s → exit code 1 → step exit 255.
• 20b accuracy stable across runs (0.545 today vs 0.555 yesterday) — no model-quality drift.

Hypothesised Causes (with confidence)

• [HIGH] Infrastructure / runner HF cache miss for gpt-oss-120b-bf16, forcing a live HF Hub download that hung on the filelock.
  • Supporting: 120b local snapshot was found (and download skipped) on Jun 26 but is absent today; with the cache gone and HF_HUB_OFFLINE=0, all 8 TP ranks raced on the same WeakFileLock and one holder blocked on a slow/hung network fetch — exactly the stack captured in the pyspy dumps. 20b (cache hit) passed in the same run.
  • Disconfirming: (none found) — the same head SHA's 20b test passed, and the failure is entirely in HF Hub download/locking code, not sglang model/kernel code.
  • How to verify: re-run nightly-accuracy-8-gpu-rocm720 after warming /sgl-data/hf-cache/hub/models--lmsys--gpt-oss-120b-bf16 on the target runner; expect 120b to pass as on Jun 26.
• [SPECULATION] sglang code regression in the 592f6c849b window (job passed Jun 26 at c98d31143d).
  • Supporting: head SHA changed between the last pass and this fail.
  • Disconfirming: failure is in third-party HF Hub filelock acquisition during download, gated entirely on local-cache presence; the cache-miss vs cache-hit difference fully explains the divergence without any code change. The commit delta is not relevant.
  • How to verify: not actionable — pursue the cache hypothesis first.

In-flight Fix Check

• In-flight fix: ⚠️ candidate #25108 "[AMD] Mount HF cache from sgl-data fallback + warn on missing" — directly targets the missing-HF-cache failure mode; worth checking whether it would warn/fallback for the 120b cache miss. Not an exact fix for the filelock hang.
• In-flight fix: ❌ none found for the filelock-hang/watchdog symptom specifically.
• API queried: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+HF+cache+download (and gpt-oss+accuracy, filelock+watchdog).

Suggested Triage Steps

• Inspect the runner's /sgl-data/hf-cache/hub/models--lmsys--gpt-oss-120b-bf16 — confirm whether the snapshot was evicted/missing, and re-warm it, then re-run the job.
• Consider setting HF_HUB_OFFLINE=1 (or a download timeout) for cached nightly models so a cache miss fails fast instead of hanging 8 ranks on a filelock for ~18 min.
• Evaluate whether #25108 (HF cache fallback + warn-on-missing) covers this path; if so, chase reviewers rather than opening a duplicate.
• If the cache is present and it still hangs, capture network connectivity to huggingface.co from the runner during 120b launch.

Status

• Persistence: 1 run (Jun 27 2026); this test passed Jun 24-26.
• Scope: nightly-accuracy-8-gpu-rocm720 job only; only the 120b model (20b passed same run). The Grok1-FP8 step was skipped due to the GPT-OSS step failing.
• Blocked work: 8-GPU GPT-OSS-120b accuracy signal on ROCm 7.2.
• In-flight fix: candidate #25108; no exact fix for the filelock hang.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:33 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 3/4)

nightly-8-gpu-mi35x-grok2-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Grok2)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5; source: Dockerfile gfx950 default — dev/patched image version detected and preserved, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_grok2_eval_mi35x.py	TestGrok2EvalMI35x.test_grok2_accuracy	TIMEOUT after 3600s — server never finished launching; all TP ranks stuck downloading xai-org/grok-2 weights from HuggingFace	Log

Failure Summary

The single accuracy test test_grok2_accuracy timed out after 3600s (the action's 60-minute hard limit), reported as 0/1 passed. The server never came up: py-spy dumps of every scheduler rank (TP0–TP7) show all of them blocked inside HuggingFace snapshot_download → hf_hub_download → xet_get, downloading xai-org/grok-2 weights via grok.py:960 _prepare_presharded_weights. Soft watchdog timeouts began firing at 18:02 (TokenizerManager + all schedulers) and the download was still in progress when the job was killed at 18:56.

Regression Status

New failure today (this specific nightly-8-gpu-mi35x-grok2-rocm720 test file passed the previous 4 scheduled runs).

Recent history of test_grok2_eval_mi35x.py (test_grok2_accuracy) in job nightly-8-gpu-mi35x-grok2-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	grok2-mi35x	❌ Failed	test_grok2_accuracy	TIMEOUT 3600s (HF weight download hang)
Jun 26	28256271771	grok2-mi35x	✅ Passed	—	—
Jun 25	28190877198	grok2-mi35x	✅ Passed	—	—
Jun 24	28119401197	grok2-mi35x	✅ Passed	—	—
Jun 23	28046795462	grok2-mi35x	✅ Passed	—	—

First observed failure: 2026-06-27. Last known passing: 2026-06-26 (run 28256271771).

Failure Cluster

HuggingFace weight-download hang during server warmup (xet_get stall → 60-min job timeout)

Facts (verified observations, no interpretation)

• Log shows ✗ TIMEOUT: .../test_grok2_eval_mi35x.py after 3600 seconds and Test Summary: 0/1 passed, then ##[error]The action 'Accuracy Test MI35x ROCm 7.2 (8-GPU Grok2)' has timed out after 60 minutes (log).
• Every scheduler rank's py-spy dump (TP0–TP7) ends in the same stack: _prepare_presharded_weights (grok.py:960) → download_weights_from_hf (weight_utils.py:593/554) → snapshot_download → hf_hub_download → xet_get (huggingface_hub/file_download.py:566) (active thread ThreadPoolExecutor-2_0).
• Soft watchdog timeouts (watchdog_timeout=300, soft=True) fired for TokenizerManager at 18:02:26 and for all schedulers at 18:02:30 — soft=True only logs, it does not kill, so the hung download continued until the action-level 60-min timeout.
• The same test file passed in the 4 prior scheduled runs (Jun 23–26) on main.
• AITER unchanged: dev image v0.1.14rc1.dev240+g7d604afe5 preserved ([CI-AITER-CHECK] markers); no override.
• Test runs xai-org/grok-2 via popen_launch_server (test file line 131); model path GROK2_MODEL_PATH = "xai-org/grok-2".

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Infrastructure / network stall in the HuggingFace xet download backend — the grok-2 weight download never progressed past xet_get on any rank within the full hour.
  • Supporting: all 8 ranks blocked in the identical xet_get frame; no model code executed beyond weight loading; the test passed yesterday on near-identical code, so it is not a deterministic code break.
  • Disconfirming: cannot from the log alone prove the stall is HF-side vs. runner-side network (no explicit network error printed); (none found that points to an sglang code defect).
  • How to verify: re-run the job; check runner egress to huggingface.co / xet CAS endpoints; inspect whether HF_HUB_DISABLE_XET=1 (force classic download) avoids the hang.
• [LOW] Soft watchdog cannot rescue a stuck download — soft=True means a wedged weight load consumes the entire 60-min budget instead of failing fast.
  • Supporting: watchdog fired at 18:02 but job ran until 18:56.
  • Disconfirming: this governs failure latency, not the root cause of the stall.
  • How to verify: n/a (design observation).
• [LOW] No sglang code regression in the c98d3114..592f6c84 window is implicated — the failure occurs entirely in third-party download code before any model forward pass.
  • Disconfirming: HEAD commit 592f6c84 touches mem-cache/HiCache/sparse KV pool, none of which is reached.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for grok2+timeout, xet+download, weight+download+hang; no matching fix for a grok-2 / xet download hang. (Searched via GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>.)

Suggested Triage Steps

• Re-run nightly-8-gpu-mi35x-grok2-rocm720 — if it passes, this is a transient HF/xet download stall (no code action needed).
• If it recurs, check runner network egress to huggingface.co and the xet CAS endpoints from the MI35x runner, and confirm a warm local weight cache exists for xai-org/grok-2.
• Consider testing with HF_HUB_DISABLE_XET=1 (fall back to classic resolve/download) to isolate the xet backend; and/or pre-stage grok-2 weights on the runner to avoid live downloads during the timed test.
• Consider whether a stuck weight load should hard-fail faster than 3600s (the soft watchdog at 300s does not abort), so the job fails in minutes rather than consuming the full hour.

Status (factual, no priority assignment)

• Persistence: 1 run (first failure 2026-06-27 ~18:02 UTC); passed the 4 prior scheduled runs (Jun 23–26).
• Scope: 1 job / 1 test file (nightly-8-gpu-mi35x-grok2-rocm720 → test_grok2_eval_mi35x.py) in nightly-test-amd-rocm720.yml. Note the sister nightly-8-gpu-grok2-rocm720 (non-mi35x) job also failed on Jun 26, but this analysis is scoped to the mi35x grok2 accuracy test.
• Blocked work: AMD MI35x ROCm 7.2 grok-2 accuracy signal for this nightly.
• In-flight fix: ❌ none found in open PRs.

nightly-8-gpu-qwen35-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: 592f6c849b
• aiter: N/A — no [CI-AITER-CHECK] markers in this workflow's log; AITER_COMMIT_OVERRIDE empty (Dockerfile default). Not relevant: failure is a config-validation assertion that aborts before any kernel loads.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_qwen35_eval_amd.py	TestQwen35EvalAMD.setUpClass	Exception: Server process exited with code 1 → AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).	Log

Failure Summary

setUpClass of TestQwen35EvalAMD failed because the SGLang server (Qwen/Qwen3.5-397B-A17B, --tp 8 --attention-backend aiter) exited with code 1 during launch. The crash is an AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). raised in _validate_mamba_extra_buffer at server_args.py:4500. 0 tests ran.

Regression Status

New regression (regressed ~June 19; failing every completed run since).

Recent history of test_qwen35_eval_amd.py (TestQwen35EvalAMD.setUpClass) in job nightly-8-gpu-qwen35-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 27	28256271771	❌ Failed	setUpClass	extra_buffer needs CUDA/MUSA/NPU
Jun 26	28190877198	❌ Failed	setUpClass	same
Jun 25	28119401197	❌ Failed	setUpClass	same
Jun 20	27879239829	❌ Failed	setUpClass	same
Jun 19	27841393725	❌ Failed (first w/ this assertion)	setUpClass	same
Jun 18	27780200720	❌ Failed (different cause: server launched OK then Scheduler watchdog timeout)	—	watchdog timeout
Jun 17	27710084518	✅ Passed	—	1/1 passed
Jun 16	27640284588	✅ Passed	—	1/1 passed

First observed with this assertion: Jun 19 (head ca88b7f1d2). Last clean pass: Jun 17 (head 8aaca72c21). On Jun 18 (head f7632ef860) the server still launched successfully (assertion not yet present) before timing out — so the assertion regression landed in f7632ef860..ca88b7f1d2.

Failure Cluster

Server launch abort — AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA) on ROCm for hybrid-mamba model.

Facts (verified observations)

• Verbatim crash from log: AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA)., raised via __post_init__ → _handle_model_specific_adjustments → _handle_mamba_radix_cache → _validate_mamba_extra_buffer (server_args.py:4500).
• Launch command (from log): sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter ... --device cuda (--device cuda is the SGLang generic GPU device name; the runner is ROCm/HIP MI30x, so is_cuda()/is_musa()/is_npu() are all False).
• The assertion guard is_cuda() or is_musa() or is_npu() was created by commit 66a7fd5c0b in _validate_mamba_extra_buffer (verified via the commit diff).
• Reproducibility: identical assertion in every completed run Jun 19→27 (6 runs); the sister job nightly-8-gpu-mi35x-qwen35-rocm720 also fails in the same runs.

Hypothesised Causes (with confidence)

• [HIGH] Commit 66a7fd5c0b — refactor: mamba radix cache server args initialize (#28151) — changed the mamba_radix_cache_strategy="auto" default and removed the ROCm guard, so AMD now selects extra_buffer and hits the CUDA/MUSA/NPU-only assertion.
  • Supporting: diff shows the pre-refactor "auto" path set mamba_scheduler_strategy = "no_buffer" (with a # TODO: when extra_buffer is more verified note) and contained an explicit # On ROCm, extra_buffer is unsupported → self.disable_radix_cache = True guard. Post-refactor, "auto" picks extra_buffer whenever wants_overlap or wants_paging and the arch supports it (overlap is on by default), and the ROCm guard is gone. The exact failing assertion extra_buffer needs CUDA/MUSA/NPU (FLA) is introduced by this same commit. Commit sits inside the regression window f7632ef860..ca88b7f1d2.
  • Disconfirming: I could not run git show locally (the local clone history is squashed; SHAs in the window are not present), so commit placement/diff was confirmed via the GitHub compare + commit API rather than local git. The Jun 18 run failed for an unrelated reason (watchdog timeout), so the immediately-preceding run was not green — but its server launched without the assertion, which supports (not weakens) the window.
  • How to verify: launch Qwen/Qwen3.5-397B-A17B on MI30x with --attention-backend aiter --tp 8 on a build with vs. without 66a7fd5c0b; or set --mamba-radix-cache-strategy no_buffer (or --disable-radix-cache) and confirm the assertion disappears.

In-flight Fix Check

• In-flight fix: ✅ #27141 — [ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5 (open since 2026-06-03, last updated 2026-06-25, non-draft). Directly targets enabling this path on ROCm. Chase reviewers; do NOT open a duplicate.
• API used: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+mamba+extra_buffer+ROCm.

Suggested Triage Steps

• Confirm by re-running test_qwen35_eval_amd.py on MI30x with --mamba-radix-cache-strategy no_buffer (or --disable-radix-cache) added in the test launch args — if it passes, 66a7fd5c0b is confirmed as the trigger.
• Prioritize landing #27141 (ROCm extra_buffer support) — it is the intended fix for this exact path.
• As an interim option, consider restoring a ROCm-aware fallback in _handle_mamba_radix_cache (auto → no_buffer when not is_cuda()/is_musa()/is_npu()), matching the pre-#28151 behavior, until #27141 merges.

Status

• Persistence: Failing with this assertion since 2026-06-19 (~9 days), every completed schedule run since.
• Scope: 2 jobs in this workflow (nightly-8-gpu-qwen35-rocm720 + nightly-8-gpu-mi35x-qwen35-rocm720); blocks all Qwen3.5 accuracy signal on AMD ROCm 7.2.
• In-flight fix: ✅ #27141 (open since 2026-06-03) — awaiting merge.

nightly-8-gpu-qwen3-235b-rocm720 — failed step(s): Accuracy Test + Performance Test ROCm 7.2 (8-GPU Qwen3)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — v0.1.14rc1.dev240+g7d604afe5 detected in CI image, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct.py	N/A (server never finished loading)	timeout after 3600s — server stuck in weight download	Log

Sister test in the same job, test/registered/amd/test_qwen3_instruct_fp8.py, passed (accuracy 0.960, 1/2 passed).

Failure Summary

The nightly-8-gpu-qwen3-235b-rocm720 job ran two test files; test_qwen3_instruct.py (Qwen3-235B-A22B-Instruct-2507, TP8) timed out after 3600 s and the harness exited with code 255. The server never reached readiness because model-weight loading stalled during the HuggingFace download phase. The FP8 sister test passed.

Regression Status

Infrastructure issue (weight-download hang) — not a code regression.

Recent history of test_qwen3_instruct.py in job nightly-8-gpu-qwen3-235b-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	nightly-8-gpu-qwen3-235b-rocm720	❌ Timeout	N/A	timeout after 3600s (download hang)
Jun 26	28256271771	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—
Jun 25	28190877198	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—
Jun 24	28119401197	nightly-8-gpu-qwen3-235b-rocm720	✅ Passed	—	—

First observed failure: Jun 27 2026. Last passing: Jun 26 2026. (The current run 28296988041 is still in_progress at snapshot time, but this job has completed/failed.)

Failure Cluster

HuggingFace weight-download hang during model load (TP holds filelock, download stalls → 3600s timeout)

Facts (verified observations, no interpretation)

• Test harness ran run_suite.py --hw amd --suite nightly-8-gpu-qwen3-235b --nightly --timeout-per-file 3600 (log line 3627).
• Final result: ✗ FAILED: test/registered/amd/test_qwen3_instruct.py (timeout after 3600s); ✓ PASSED: test_qwen3_instruct_fp8.py; Test Summary: 1/2 passed; Process completed with exit code 255 (log lines 12575–12578).
• TP7 (PID 3331) ACQUIRED the download lock at 17:57:42 for Qwen/Qwen3-235B-A22B-Instruct-2507 (log lines 3844–3846).
• TP7's py-spy stack (lines 4096–4130) shows it idle in: snapshot_download (huggingface_hub/_snapshot_download.py:467) → thread_map → result_iterator → wait (threading.py:320), reached via ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → download_weights_from_hf (weight_utils.py:593). The download itself is blocked waiting on its worker-thread futures.
• All other schedulers (TP1/TP2/TP3/TP5) are idle in _poll_until_acquired (filelock/_api.py:552) inside download_weights_from_hf (weight_utils.py:554) — i.e. blocked waiting for the lock TP7 holds (lines 3964–3997, 4030+).
• Soft watchdog (watchdog_timeout=300, soft=True) fired starting 18:02:12 on TokenizerManager and every scheduler, but being soft it only logged and did not kill the process; the hang persisted to the 3600 s file timeout.
• py-spy could not dump TP7's native stack repeatedly (Failed to get stack traces / failed to get os threadid, lines 3993, 4556, 5021, … every ~5 min) — consistent with a process wedged in a download worker.
• The job's other test (test_qwen3_instruct_fp8.py) loaded its model and ran to completion (accuracy 0.960), so the cluster/runner GPUs were healthy.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] HuggingFace hub weight download for Qwen/Qwen3-235B-A22B-Instruct-2507 stalled while TP7 held the shared download filelock, blocking all 8 TP workers from finishing model load within the 3600 s budget.
  • Supporting: TP7's verified py-spy stack sits in huggingface_hub snapshot_download thread-pool wait; the other 7 TPs are verified blocked on the filelock TP7 owns; the FP8 sister test in the same job loaded fine, so it is not a GPU/runner fault.
  • Disconfirming: no explicit network/ReadTimeout/HTTP error was logged (the stall is silent), so we cannot 100% rule out a deadlock inside ci_download_with_validation_and_retry rather than a pure network stall.
  • How to verify: re-run the job; if it passes (weights now cached / network recovered) this confirms a transient download stall. If it reproduces, inspect model_loader/ci_weight_validation.py:1955 and the snapshot_download max_workers/retry path for a possible hang.
• [LOW] Code-level regression in the weight-download/validation path.
  • Supporting: none specific.
  • Disconfirming: the same job + test file passed in the previous 3 completed scheduled runs (Jun 24–26) on nearby SHAs; the failure mode is a silent I/O stall, not an exception/traceback in sglang code.
  • How to verify: not actionable unless the hang reproduces across multiple reruns.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs (repo:sgl-project/sglang is:pr is:open download hang timeout, 4 results, none relevant to HF weight-download hangs). This is most likely transient infrastructure, not a code defect needing a PR.

Suggested Triage Steps

• Re-run nightly-8-gpu-qwen3-235b-rocm720 (the job passed on Jun 24–26); a green rerun confirms a transient HF download stall and closes this out.
• If the hang reproduces, capture whether TP7 is stuck on network I/O vs. a thread-pool deadlock by inspecting model_loader/ci_weight_validation.py:1955 (ci_download_with_validation_and_retry) and the snapshot_download worker/retry config.
• Consider pre-staging/caching Qwen/Qwen3-235B-A22B-Instruct-2507 weights on the MI30x runner volume (/sgl-data/hf-cache) so nightly runs skip the live HF download, removing this flake class.
• Consider whether the soft watchdog (soft=True) should escalate to a hard kill on prolonged load-phase stalls so the file-level 3600 s timeout isn't the only backstop.

Status (factual, no priority assignment)

• Persistence: First failure on Jun 27 2026; the same job passed in the 3 prior completed scheduled runs (Jun 24–26).
• Scope: 1 job (nightly-8-gpu-qwen3-235b-rocm720), 1 of its 2 test files (test_qwen3_instruct.py); the FP8 sister test passed.
• Blocked work: Blocks the Qwen3-235B BF16 nightly accuracy/speed signal on AMD MI30x ROCm 7.2 for this run only.
• In-flight fix: ❌ none found — likely transient infra; rerun to confirm.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports v0.1.14rc1.dev240+g7d604afe5 "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) — server crash during HIP graph capture	Log

Failure Summary

The server for amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 (--tp 4 --ep 2 --attention-backend aiter) crashed during startup, so setUpClass failed and Ran 0 tests. The Rank 0 scheduler aborted with exit code -6 (SIGABRT) while capturing the decode HIP graph for the Qwen3-MoE forward pass. The crash originates in the ProcessGroupNCCL (RCCL) watchdog thread, which threw HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent).

Regression Status

Known recurring failure (Never-passed in queryable window). Every completed schedule run of this job in the lookback window (Jun 20 → Jun 26, 7 runs) failed with the identical hipErrorCapturedEvent symptom on this same test file. No green run exists in the queryable window; earlier runs (Jun 16–19) were cancelled and not comparable.

Recent history of test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
Jun 27	28296988041	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 26	28256271771	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Jun 25	28190877198	❌ Failed	setUpClass	same
Jun 24	28119401197	❌ Failed	setUpClass	same
Jun 23	28046795462	❌ Failed	setUpClass	same
Jun 22	27975850577	❌ Failed	setUpClass	same
Jun 21	27912828989	❌ Failed	setUpClass	same
Jun 20	27879239829	❌ Failed	setUpClass	hipErrorCapturedEvent (verified)

First observed failure (queryable window): Jun 20 2026. No known passing date in the window. Lookback range: Jun 16 → Jun 27 (Jun 16–19 cancelled, excluded). The current Jun 27 run is the workflow's only in_progress run; this job within it has already completed and failed.

Failure Cluster

HIP hipErrorCapturedEvent in RCCL watchdog during MoE all-reduce graph capture (ROCm 7.2.0)

Facts (verified observations, no interpretation)

• Verbatim crash (log step at 21:12:42): terminate called after throwing an instance of 'c10::DistBackendError' / [PG ID 6 PG GUID 27 Rank 1] Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream → Search for 'hipErrorCapturedEvent'.
• Crashing thread is the RCCL watchdog: stack frames c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() → isCompleted() → ProcessGroupNCCL::Watchdog::runLoop() (ProcessGroupNCCL.cpp:2063). This thread polls a HIP event for completion (effectively hipEventQuery).
• The concurrent main thread is capturing a HIP graph: stack decode_cuda_graph_runner.py:690 capture → full_cuda_graph_backend.py:111 capture_one → qwen3_moe.py:1002/847/299 forward → qwen3_moe.py:341 forward_normal → communication_op.py:79 moe_tensor_model_parallel_all_reduce → parallel_state.py:660 all_reduce → parallel_state.py:169/777 inplace_all_reduce → torch all_reduce (distributed_c10d.py:2935). Source lines verified: communication_op.py:79, parallel_state.py:660.
• Resulting process state: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6); Ran 0 tests in 496.001s; FAILED (errors=1); job exit code 255.
• Environment: image rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260627, compiled against /opt/rocm-7.2.0/include (ROCm 7.2.0), GPU_ARCH=mi35x/gfx950, aiter dev v0.1.14rc1.dev240+g7d604afe5.
• Reproducibility: identical symptom in 7/7 completed schedule runs over 7 days; affects multiple ranks (PG ID 5 and 6, ranks 0/1/2 all print the same error).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] ROCm 7.2.0 does not permit hipEventQuery on an event recorded in a stream that is mid graph-capture; the RCCL watchdog thread queries the all-reduce work event concurrently with the main thread capturing the MoE decode graph, raising hipErrorCapturedEvent and aborting the scheduler.
  • Supporting: crash is in the watchdog's finishedGPUExecutionInternal/isCompleted (an event query), the error text is explicitly "event last recorded in a capturing stream"; image is ROCm 7.2.0; in-flight PR #24151 "Bump ROCm720 base image 7.2.0→7.2.2 to fix hipEventQuery race" targets exactly this API/symptom.
  • Disconfirming: no A/B run on 7.2.2 is available to confirm the bump resolves it; failure is also consistent with an sglang-side capture/watchdog-coordination gap rather than a pure ROCm bug.
  • How to verify: build/run this job on the 7.2.2 base image (per #24151) and re-run test_qwen3_instruct_mxfp4.py; or set AMD_SERIALIZE_KERNEL=3 and capture a coredump to confirm the watchdog event-query is the faulting call.
• [LOW] aiter all-reduce path (SGLANG_USE_AITER_AR) interferes with HIP graph capture on this MoE all-reduce.
  • Supporting: the captured op is moe_tensor_model_parallel_all_reduce under --attention-backend aiter; in-flight PRs #23581 (default SGLANG_USE_AITER_AR=false to avoid HIP graph capture invalidation) and #27650 (PCG capture fail with aiter fused all-reduce+RMSNorm) address aiter-AR capture problems.
  • Disconfirming: the actual faulting frame is the RCCL watchdog doing an event query, not the aiter AR kernel itself; the error is the generic capturing-stream event restriction, which points at the watchdog/ROCm interaction rather than aiter code specifically.
  • How to verify: re-run with SGLANG_USE_AITER_AR=0 and check whether the capture completes.

In-flight Fix Check

• In-flight fix: ✅ #24151 (open since 2026-04-30) — "Bump ROCm720 base image 7.2.0→7.2.2 to fix hipEventQuery race"; directly matches the symptom and the ROCm 7.2.0 environment. Ping reviewers / verify it lands; do NOT open a duplicate. (Searched: repo:sgl-project/sglang is:pr is:open capturing stream and ... watchdog graph.)
• In-flight fix: ✅ #23581 (open since 2026-04-23) — default SGLANG_USE_AITER_AR=false to avoid HIP graph capture invalidation (covers the [LOW] aiter-AR hypothesis).
• Related: #27650 (open since 2026-06-09) — aiter fused all-reduce+RMSNorm PCG capture fix.

Suggested Triage Steps

• Verify in-flight PR #24151 (ROCm720 7.2.0→7.2.2) builds cleanly, then re-run nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 against the 7.2.2 image to test the [MEDIUM] hypothesis.
• If still failing on 7.2.2, re-run with SGLANG_USE_AITER_AR=0 (per #23581) to test the [LOW] aiter-AR hypothesis.
• Capture a device coredump with AMD_SERIALIZE_KERNEL=3 to confirm the watchdog hipEventQuery is the faulting call vs. the aiter AR kernel.
• If neither fixes it, consider whether the RCCL watchdog can be quiesced during graph capture (coordination gap) and disable this test on rocm720/mi35x until resolved.

Status (factual, no priority assignment)

• Persistence: Failing every completed schedule run for 7 days (since 2026-06-20; Never-passed in the queryable window).
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720) in nightly-test-amd-rocm720.yml; crash affects all ranks (PG 5 & 6). ROCm 7.2.0 / mi35x specific.
• Blocked work: Blocks Qwen3-235B MXFP4 nightly accuracy signal on AMD MI35x / ROCm 7.2.0.
• In-flight fix: ✅ #24151 (ROCm720 7.2.2 bump) — strongest candidate; also #23581.

nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 592f6c84
• aiter: N/A (job died in the host-level pre-test step ensure_vram_clear.sh; no container started, no dependency install, so no [CI-AITER-CHECK] markers. AITER_COMMIT_OVERRIDE was empty.)

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure)	N/A	FAILED: VRAM cleanup unsuccessful after 3 attempts → exit code 1	Log

No test ever ran. The job failed in the pre-test step bash scripts/ci/amd/ensure_vram_clear.sh rocm, before the Docker container was pulled or built.

Failure Summary

The job failed at the host-level VRAM pre-clean step on runner linux-mi35x-gpu-8-zf775-runner-9w45z. GPUs 0–3 were each stuck at 16% VRAM allocated (above the 5% threshold) and stayed there across all 3 cleanup attempts (SIGTERM → SIGKILL → 30s wait, ×3). Because the pre-step exited 1, Docker pull/build and the entire DeepSeek-R1 HiCache test suite never executed.

Regression Status

Infrastructure issue (new occurrence today for this failure mode).

Recent history of the VRAM pre-clean step in job nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 (same branch main, same event schedule, completed runs only):

Date	Run	Job	VRAM-Clear Step	Job Outcome	Note
Jun 27	28296988041	job	❌ Failed (3/3 attempts)	❌ infra fail, no tests ran	this run
Jun 26	28256271771	job	✅ Passed	❌ failed later (in-test)	VRAM cleared, build+tests ran
Jun 25	28190877198	job	✅ Passed	❌ failed later (in-test)	VRAM cleared, build+tests ran
Jun 24	28119401197	job	✅ Passed	❌ failed later (in-test)	VRAM cleared, build+tests ran

The VRAM-clear pre-step passed in the previous three scheduled runs (they proceeded to Docker pull and test execution, failing later for separate in-test reasons). Today's VRAM-clear failure is therefore a new, distinct infrastructure failure, not the recurring test-level failure seen on Jun 24–26.

Failure Cluster

Host VRAM not clearable at pre-test step (zombie KFD context, GPUs 0–3 stuck at 16%).

Facts (verified observations, no interpretation)

• [FACT] Pre-test step exited with ##[error]Process completed with exit code 1. (line 545).
• [FACT] GPUs 0,1,2,3 each reported GPU Memory Allocated (VRAM%): 16 at the start and after all 3 cleanup attempts; GPUs 4–7 were at 0% (lines 167–171).
• [FACT] Cleanup ran 3 attempts (SIGTERM, then SIGKILL, then 30s wait each); VRAM remained at 16% after every attempt (✗ VRAM still not clear after attempt 1/2/3, lines 262/371/480).
• [FACT] Diagnostics showed no host processes / containers / KFD PIDs holding the GPUs: No KFD PIDs currently running, No docker containers found on host, empty zombie list (lines 264–277, 380–387, 489–496).
• [FACT] The script's own hint attributes this to a zombie KFD context and recommends a node reboot, citing ROCm/aiter#2061 (lines 540–543).
• [FACT] Runner: linux-mi35x-gpu-8-zf775-runner-9w45z (line 2).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Zombie KFD/HIP context on runner linux-mi35x-gpu-8-zf775-runner-9w45z is holding ~16% VRAM on GPUs 0–3 from a prior job that crashed without releasing its GPU allocation.
  • Supporting: VRAM stays pinned at exactly 16% on 4 GPUs across all 3 kill+wait cycles, yet no KFD PID, no host process, and no container is reported as holding the GPUs — the classic signature of an orphaned kernel-driver context that only a reboot clears. The script itself diagnoses this and cites ROCm/aiter#2061.
  • Disconfirming: (none found) — the diagnostics actively rule out the alternative (a live host process holding the device); no competing explanation is visible in the log.
  • How to verify: reboot/cordon node linux-mi35x-gpu-8-zf775-runner-9w45z, then re-run the job; confirm rocm-smi --showmemuse reports 0% on GPUs 0–7 before the run.
• [LOW] A code change in 592f6c84 caused a leak.
  • Supporting: it is the HEAD commit under test.
  • Disconfirming: the failure happens before any sglang code runs (no container, no import) — the stuck VRAM predates this job's checkout and is pure host state left by a previous job on the shared runner. Commit content cannot influence a pre-test host cleanup step.
  • How to verify: not actionable; the failure is independent of the checked-out SHA.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a runner-state/infra problem (node needs reboot), not a code defect, so a PR is not the appropriate remedy. Searched open PRs:
  • curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/repos/sgl-project/sglang/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+ensure_vram_clear" — (no matching code-fix PR; tracked upstream as ROCm/aiter#2061).

Suggested Triage Steps

• Reboot (or cordon + drain) runner linux-mi35x-gpu-8-zf775-runner-9w45z to clear the zombie KFD context on GPUs 0–3, then re-run run 28296988041.
• After reboot, confirm rocm-smi --showmemuse shows 0% across all 8 GPUs before dispatching the next job to this node.
• Separately track the recurring in-test failure of this job (Jun 24–26 runs passed VRAM-clear but still failed) — it is a distinct issue from today's infra failure and needs its own analysis once the node is healthy.
• Consider whether the prior job on this runner crashed without GPU cleanup; if a specific test reliably leaves orphaned KFD contexts, that test is the upstream source of the zombie state.

Status (factual, no priority assignment)

• Persistence: First VRAM-clear pre-step failure in the last 4 scheduled runs (Jun 24–27); the prior 3 runs cleared VRAM successfully.
• Scope: This job on runner linux-mi35x-gpu-8-zf775-runner-9w45z. Other jobs scheduled on the same node will also fail at the pre-clean step until it is rebooted.
• Blocked work: Blocks the entire DeepSeek-R1 HiCache (8-GPU MI35x, ROCm 7.2) nightly signal for this run — no tests executed.
• In-flight fix: ❌ none found — needs node reboot (infra action), tracked upstream as ROCm/aiter#2061.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: 592f6c84
• aiter: v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild"; Dockerfile default was 7d604afe5)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.setUpClass	Exception: Server process exited with code -9 (server crashed during warmup/decode)	Log

Failure Summary

The DeepSeek-V3.2 MTP accuracy test on MI35x (ROCm 7.2) failed in setUpClass because the server process died (exit code -9 / SIGQUIT). All 8 TP scheduler ranks crashed simultaneously with AssertionError: All of them must not be None, raised in dsa_backend.py:721 inside the is_draft_extend_v2() branch of init_forward_metadata, reached via the speculative MTP draft-extend path (eagle_worker_v2._draft_extend_for_decode → base_spec_worker.prepare_for_draft_extend). The test never ran any eval; the server crashed during startup/warmup.

Regression Status

New regression (this specific crash signature first appears today, 2026-06-27).

Recent history of test_deepseek_v32_mtp_eval_mi35x.py in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	accuracy-mtp-rocm720	❌ Failed	setUpClass	Server exit -9 — AssertionError: All of them must not be None (dsa_backend.py:721)
Jun 26	28256271771	accuracy-mtp-rocm720	❌ Failed (different)	eval assertion	AssertionError: np.float64(0.94) not greater than 0.94 (accuracy miss, server stayed up — NOT this crash)
Jun 25	28190877198	accuracy-mtp-rocm720	✅ Passed	—	—
Jun 24	28119401197	accuracy-mtp-rocm720	✅ Passed	—	—

The Jun 26 failure was a soft accuracy mismatch (0.94 not > 0.94), a distinct failure from today's hard server crash. For the crash signature, the last passing run is Jun 25 (9495737d82) and the last run without this crash is Jun 26 (c98d31143d) → regression window c98d31143d..592f6c849b (32 commits).

Failure Cluster

DSA spec-v2 draft-extend assertion crash (extend_*_cpu is None) during DeepSeek-V3.2 MTP

Facts (verified observations, no interpretation)

• All 8 TP ranks (TP0–TP7) crash with identical traceback ending AssertionError: All of them must not be None at python/sglang/srt/layers/attention/dsa_backend.py:721 (verified log lines 21266–21466).
• Call chain (verbatim from traceback): scheduler.run_batch → model_worker.forward_batch_generation → eagle_worker_v2.forward_batch_generation:1171 → _draft_extend_for_decode:850 → base_spec_worker.prepare_for_draft_extend:163 → dsa_backend.init_forward_metadata:721.
• The asserted condition (current source, dsa_backend.py:717-722) requires extend_seq_lens_cpu, extend_seq_lens, and extend_prefix_lens_cpu to all be non-None in the is_draft_extend_v2() branch; line 721 is the extend_prefix_lens_cpu is not None clause.
• Server exited code -9 → setUpClass raised Server process exited with code -9 (log line 21484), test reported FAILED (errors=1), job exit 255.
• aiter unchanged vs passing runs (dev image preserved; not an override).
• Crash is reproducible across all 8 ranks in this single run; this is the only run exhibiting this signature so far.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Commit 9214b933 ("[DSA] Enable draft-extend CUDA graph for DeepSeek Sparse Attention", #29413) introduced a needs_cpu_seq_lens=False path that nulls the CPU host mirror for spec-v2 relay batches, but left the hard is_draft_extend_v2() assertion in init_forward_metadata intact.
  • Supporting: The commit modifies the exact failing file (dsa_backend.py, +78 lines) and the exact code path (eagle_worker_v2.py _draft_extend_for_decode, +24 lines); its own diff removes sibling assert ... seq_lens_cpu is not None checks and replaces them with None-tolerant fallbacks, with the comment "needs_cpu_seq_lens=False nulls the host mirror for spec-v2 relay batches; ... only this eager (e.g. over-capture-bs) fallback needs a length here." The surviving assertion at line 721 fires when extend_prefix_lens_cpu is the nulled mirror on the eager (non-graph) draft-extend path. Timing: landed 2026-06-27 06:53 UTC (Jun 26 23:53 PDT), absent from the Jun 26 run head (c98d31143d), present in the Jun 27 run head — i.e. it lands exactly between the last non-crash run and the first crash run.
  • Disconfirming: No revert+rerun A/B performed yet; the commit's stated goal is enabling the CUDA-graph path, and this crash is on the eager fallback (graph-disabled / over-capture-bs), so it is plausibly an incomplete-coverage bug in that commit rather than a regression in its main path — but this does not exonerate the commit.
  • How to verify: git revert 9214b933 on a branch, rebuild, run test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py on an MI35x 8-GPU ROCm 7.2 runner; confirm the server no longer crashes in init_forward_metadata.
• [LOW] Other DSA-area commits in the window — ee77a7d3 (#29379, DSA cudagraph page_table sizing) or 73741f70 (#29454, GLM DSA layer-type bypass).
  • Supporting: both touch DSA code and land in the window.
  • Disconfirming: neither modifies the is_draft_extend_v2 branch or the extend_*_cpu host-mirror nulling; the assertion text and call path point squarely at the 9214b933 rework.
  • How to verify: only investigate if the 9214b933 revert does not resolve.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for draft-extend DSA, extend_prefix_lens_cpu, DSA draft extend (https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...). The closest open PR is #29499 ("[DSA] Optimize DSA CUDA graph replay metadata generation") which touches the same area but is an optimization, not a fix for this assertion. Needs a new PR or investigation.

Suggested Triage Steps

• Bisect / spot-check the regression window c98d31143d..592f6c849b, starting with 9214b933.
• Revert 9214b933 on a branch and re-run test_deepseek_v32_mtp_eval_mi35x.py on MI35x 8-GPU ROCm 7.2 to confirm the assertion is the introduced regression.
• If confirmed, the fix direction is to make the is_draft_extend_v2() branch in dsa_backend.py:717-722 tolerate the nulled CPU host mirror (mirror the None-tolerant pattern the same commit added at dsa_backend.py:647-661), or to ensure prepare_for_draft_extend populates extend_prefix_lens_cpu on the eager draft-extend path.
• Ping the author of #29413 (Khoa Pham) since the failing path is the one that PR reworked.

Status

• Persistence: First seen this run (2026-06-27 20:57 UTC). Distinct from the Jun 26 accuracy miss; last crash-free run was Jun 26.
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp-rocm720), all 8 TP ranks. Likely affects any DSA + spec-v2 (MTP) eager draft-extend path on DeepSeek-V3.2; the companion nightly-perf-...-deepseek-v32-mtp job should be checked for the same crash.
• Blocked work: DeepSeek-V3.2 MTP accuracy signal on AMD MI35x / ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:33 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 4/4)

nightly-8-gpu-mi35x-qwen35-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: 592f6c84
• aiter: N/A — no [CI-AITER-CHECK] markers in this job log; AITER_COMMIT_OVERRIDE empty (Dockerfile default). aiter is irrelevant: the server crashes during arg validation, before any kernel runs.

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	test_lm_eval (TestQwen35EvalMI35x)	AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA). → server exits code 1	Log

Note: the sister test test/registered/amd/test_qwen3p5_triton_dcp.py in the same job passed (Test Summary: 1/2 passed).

Failure Summary

The accuracy test test_lm_eval for Qwen/Qwen3.5-397B-A17B (a Mamba / linear-attn hybrid model) fails because the SGLang server crashes on startup. During ServerArgs.__post_init__, _handle_mamba_radix_cache auto-selects the extra_buffer radix-cache strategy and then _validate_mamba_extra_buffer asserts the platform is CUDA/MUSA/NPU. On ROCm this assertion fails, so the server never launches and the test errors after retries.

Regression Status

Known recurring failure (persistent, every completed scheduled run in the queryable window).

Recent history of test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	job	❌ Failed	test_lm_eval	extra_buffer needs CUDA/MUSA/NPU (FLA)
Jun 26	28256271771	job	❌ Failed	test_lm_eval	same assertion
Jun 25	28190877198	job	❌ Failed	test_lm_eval	same assertion
Jun 23	28046795462	job	❌ Failed	test_lm_eval	same assertion

No passing run for this test file found in the queryable window (lookback: last 8 scheduled runs, Jun 20–27 — all failure). The earliest log I inspected (Jun 23) already shows the identical assertion, so the failure predates the queryable window.

Failure Cluster

ROCm Mamba extra_buffer radix-cache assertion crash on server startup (Qwen3.5 hybrid).

Facts (verified observations, no interpretation)

• Verbatim crash (step:6): File ".../server_args.py", line 4500, in _validate_mamba_extra_buffer → AssertionError: extra_buffer needs CUDA/MUSA/NPU (FLA).
• Server launch command: sglang serve --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend aiter ... --device cuda — --device cuda is the SGLang alias used on ROCm; the platform is still HIP.
• is_cuda() (python/sglang/srt/utils/common.py:163-164) = torch.cuda.is_available() and torch.version.cuda is not None. On ROCm torch.version.cuda is None (it is torch.version.hip), so is_cuda() → False; is_musa() and is_npu() are also False → the assert at server_args.py:4500-4502 fails. [FACT]
• Control flow: _handle_mamba_radix_cache (server_args.py:4512) with default mamba_radix_cache_strategy="auto" and overlap-schedule enabled (default) selects extra_buffer because _support_mamba_cache_extra_buffer returns True for Qwen3_5MoeForConditionalGeneration (server_args.py:4471-4483). It then calls _validate_mamba_extra_buffer, which asserts CUDA/MUSA/NPU. [FACT]
• Same assertion reproduced in 4 consecutive scheduled runs (Jun 23/25/26/27) on the same test file. [FACT]
• Sister test test_qwen3p5_triton_dcp.py passes in the same job → not a job-wide infra/image issue. [FACT]

Hypothesised Causes (with confidence)

• [HIGH] The Mamba radix-cache auto strategy selects extra_buffer, which is hard-gated to CUDA/MUSA/NPU, but the job runs on ROCm — so the server cannot start for this hybrid model on AMD.
  • Supporting: deterministic assert at server_args.py:4500-4502; is_cuda()/is_musa()/is_npu() all False on ROCm (verified by reading common.py:163); reproduced identically across 4 runs; the FLA (extra_buffer) path has no ROCm implementation.
  • Disconfirming: (none found) — the crash is platform-deterministic and independent of the commit under test.
  • How to verify: launch the same server command in an ROCm container with --mamba-radix-cache-strategy no_buffer (and --disable-overlap-schedule); the assert should not fire.
• [MEDIUM] Introduced by the mamba-radix-cache refactor in #28151 (66a7fd5c, merged 2026-06-18) which made auto default to extra_buffer when overlap/paging is wanted, without a ROCm fallback.
  • Supporting: failure first observed in this window aligns with the refactor landing Jun 18; the refactor owns the _handle_mamba_radix_cache / _validate_mamba_extra_buffer logic. The dedicated fix attempt #29158 names exactly this assertion and these nightly jobs.
  • Disconfirming: local clone is shallow (cannot enumerate the full pre-Jun-18 history), so I could not bisect to a green baseline; the line-history tool mis-attributes the block to an unrelated graft-boundary commit (efbe67d2), so introduction date is inferred from PR metadata, not a verified green→red run pair.
  • How to verify: on a full clone, find the last scheduled run where test_qwen35_eval_mi35x passed and git log <pass_sha>..<fail_sha> around 2026-06-18.

In-flight Fix Check

• In-flight fix: ✅ #27141 "[ROCm] Support radix-cache with mamba-extra-buffer for Qwen3.5" — open (since 2026-06-03, updated 2026-06-25, not draft). This is the proper fix (adds ROCm support for the extra-buffer/FLA path). Chase reviewers; do NOT open a duplicate.
• Note: a prior simpler fix #29158 ("[AMD] Fall back to mamba no_buffer radix cache on ROCm") was closed unmerged on 2026-06-24 — the ROCm fallback was apparently superseded by the support-it approach in #27141.
• API calls used: search/issues?q=...is:pr+is:open+mamba+no_buffer+ROCm and ...qwen35+mamba.

Suggested Triage Steps

• Verify #27141 still applies cleanly to head 592f6c84 and re-run nightly-8-gpu-mi35x-qwen35-rocm720; if green, this single PR resolves the cluster.
• As an interim signal-restoring option (maintainer's call), confirm whether the no_buffer fallback from the closed #29158 would unblock the nightly while #27141 lands — i.e. have _handle_mamba_radix_cache pick no_buffer when not CUDA/MUSA/NPU instead of asserting in _validate_mamba_extra_buffer.
• On a full clone, pin the exact regression commit with git log <last_pass_sha>..<first_fail_sha> around the #28151 merge (2026-06-18) to confirm the introduction point.

Status

• Persistence: Failing every completed scheduled run in the lookback window (Jun 23→27, ≥4 consecutive runs; predates queryable history). Deterministic, not flaky.
• Scope: This test file/function (test_qwen35_eval_mi35x.py::test_lm_eval) on ROCm 7.2 MI35x. Per fix PR #29158's description, the sibling nightly-8-gpu-qwen35 job is affected by the same assertion. Sister test in the same job passes.
• Blocked work: Qwen3.5 8-GPU accuracy signal on AMD MI35x ROCm 7.2 nightly.
• In-flight fix: ✅ #27141 open (since 2026-06-03) — needs review/merge; prior fallback #29158 closed unmerged.

nightly-8-gpu-kimi-k26-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_kimi_k26_eval_amd.py	TestKimiK26EvalAMD.test_kimi_k26_gsm8k_accuracy	✗ TIMEOUT after 3600 seconds (Time elapsed 3605.11s; exit code 255)	Log

Failure Summary

The single test file in the nightly-amd-accuracy-8-gpu-kimi-k26 suite hit the per-file 3600s timeout. Model weight loading for Kimi-K2.6 (8-GPU TP, compressed-tensors, ~73.6 GB/GPU) took 3359s (Load weight end. elapsed=3358.97 s), leaving only ~85s for the GSM8K eval, which reached 99% (1305/1319) before the deadline fired. This is a timeout driven by slow startup, not an accuracy failure or kernel crash.

Regression Status

Known recurring failure — 3 consecutive scheduled runs (Jun 25, 26, 27). Last passing run: Jun 24, where weight load took only 2375–2380s (~40 min) and the eval completed.

Recent history of test_kimi_k26_eval_amd.py in job nightly-8-gpu-kimi-k26-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Test File Status	Detail
Jun 27	28296988041	❌ Failed	TIMEOUT; weight load 3359s, eval reached 99% then deadline
Jun 26	28256271771	❌ Failed	TIMEOUT; weight load never completed (no Load weight end)
Jun 25	28190877198	❌ Failed	TIMEOUT; weight load never completed
Jun 24	28119401197	✅ Passed	weight load 2380s; eval completed

(The current run is the in-progress workflow 28296988041; this job itself is complete.)

Failure Cluster

Kimi-K2.6 8-GPU weight load exceeds per-file timeout (startup-bound TIMEOUT, not accuracy)

Facts (verified observations)

• ✗ TIMEOUT: .../test_kimi_k26_eval_amd.py after 3600 seconds; Fail. Time elapsed: 3605.11s; ##[error]Process completed with exit code 255 (verbatim from log).
• Weight load window: Load weight begin 19:06:17 → Load weight end. elapsed=3358.97 s 20:02:16, type=KimiK25ForConditionalGeneration, quant=compressed-tensors, mem usage=73.58 GB per GPU.
• KV cache allocated 20:02:18; eval method [CI Test Method] TestKimiK26EvalAMD.test_kimi_k26_gsm8k_accuracy started 20:03:47; eval progressed 0→99% (1305/1319) by 20:05:11; deadline at 20:05:17.
• TokenizerManager watchdog timeout (300s, soft=True) and repeated Pyspy failed dumps during 19:10–19:30 are the soft watchdog firing while weights load — diagnostic noise, not a hang/crash (process stack stuck in _wait_for_scheduler_ready → load).
• Weight-load time is highly variable across runs: 2380s (pass Jun 24) → never-finished (Jun 25/26) → 3359s (Jun 27). The per-file budget is exactly 3600s (--timeout-per-file 3600); the test's own est_time=3600 leaves zero margin.
• AITER_COMMIT_OVERRIDE is empty — this is a regular nightly schedule run, not a scout run (no Baseline A/B required).

Hypothesised Causes (with confidence)

• [FACT] The job failed because Kimi-K2.6 weight loading consumed ~3359s of the 3600s per-file budget, leaving insufficient time for the eval to finish.
  • Supporting: Load weight end. elapsed=3358.97 s; eval only started at 20:03:47 and reached 99% before the 3600s deadline.
  • Disconfirming: (none — directly observed)
• [MEDIUM] Slow/variable Kimi-K2.6 compressed-tensors MoE weight load + post-load repack on ROCm is the underlying driver; the test budget has no headroom over the load time.
  • Supporting: even the passing Jun 24 run spent 2380s (~66% of budget) on weight load alone; multiple open PRs target exactly this path (#24076 vectorizes Kimi MoE post-load repack; #29178/#28905 fix Kimi-K2.6 compressed-tensors MoE w2 scale loading on ROCm).
  • Disconfirming: the load did complete in 2380s on Jun 24, so the path is not strictly broken — the failure is timing/variance, possibly compounded by HF download or disk/network jitter (no explicit download log captured this run).
  • How to verify: re-run with weight-load instrumentation; compare load elapsed across 3–4 runs; test whether #24076 reduces Load weight end. elapsed.
• [LOW] Per-GPU model download (first-pull) variance inflates total file time on slow-cache runners.
  • Supporting: load time swings >1000s run-to-run, consistent with cache/IO variance.
  • Disconfirming: Load weight begin appears promptly (~19:06), suggesting weights were largely local; variance is within the load phase, not pre-load download.

In-flight Fix Check

• In-flight fix: ✅ #24076 (open) "Vectorize Kimi MxInt4 MoE post-load repack" — directly targets the slow weight-load/repack path; verify it reduces load elapsed below budget.
• In-flight fix: ✅ #29178 / #28905 — Kimi-K2.6 compressed-tensors MoE w2 scale loading fixes on ROCm (same load path; may affect load behavior/time).
• No open PR found that raises this suite's timeout-per-file / est_time. (Searched kimi k26 timeout, kimi weight load.)

Suggested Triage Steps

• Quantify load variance: pull Load weight end. elapsed from the last ~6 nightly-8-gpu-kimi-k26-rocm720 runs to confirm the load time trend vs the 3600s budget.
• Evaluate #24076 on this runner — if it cuts Load weight end. elapsed materially, it likely resolves the recurring timeout.
• As an interim CI-signal measure, consider raising --timeout-per-file / the test's est_time (currently 3600s with zero margin over a ~40–56 min load) so eval time isn't starved — maintainer decision.
• Confirm weights are warm-cached on the MI325 runner image to remove first-pull download variance from the load window.

Status

• Persistence: Failing 3 consecutive scheduled runs (Jun 25, 26, 27 2026); last pass Jun 24.
• Scope: nightly-8-gpu-kimi-k26-rocm720 job, single test file test_kimi_k26_eval_amd.py. Sister mi35x job (different HW) passed Jun 26, failed Jun 25 — not directly comparable.
• In-flight fix: #24076 (load-speed) and #29178/#28905 (load-correctness) are open and touch the implicated Kimi-K2.6 load path.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, matches Dockerfile default, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen2.5-VL-7B-Instruct)	Exception: Server process exited with code -9 — scheduler raised huggingface_hub.errors.HfHubHTTPError: 429 Too Many Requests during weight download	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (model=Qwen/Qwen2-VL-7B-Instruct)	Failed to load MMMU datasets (3/3 attempts failed)	Log

Failure Summary

The VLM MMMU accuracy suite (test_vlms_mmmu_eval_amd.py) failed during model serving. The Qwen2.5-VL-7B-Instruct server process was killed (exit -9) when the scheduler tried to download model weights and HuggingFace Hub returned 429 Too Many Requests (0/3000 requests remaining in current 300s window). A second model, Qwen2-VL-7B-Instruct, failed to load the MMMU evaluation dataset after 3 attempts. The job exited with code 255 and was not retried.

Regression Status

Known recurring failure — but today's specific symptom (HF 429 rate-limit) is distinct from prior runs.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function / Symptom	Error
Jun 27	28296988041	job	❌ Failed	test_mmmu_vlm_models (Qwen2.5-VL / Qwen2-VL)	HF 429 Too Many Requests on weight + MMMU dataset download
Jun 26	28256271771	job	❌ Failed	test_mmmu_vlm_models	FAILED (failures=1) — accuracy/assertion failure (different symptom)
Jun 25	28190877198	job	❌ Failed	test_mmmu_vlm_models (deepseek-vl2-small)	FAILED (errors=1) — weight-download hang in download_weights_from_hf (different symptom)

This test file has failed in every recent completed scheduled run (Jun 20–26 all failure), so it is a Never-greened job in the queryable window. However, each run fails on a different model / different cause; do not treat the prior failures as the same bug as today's 429.

Failure Cluster

HuggingFace Hub 429 rate-limit during model-weight / dataset download

Facts (verified observations, no interpretation)

• [FACT] Local HF cache was incomplete: Local HF snapshot at /sgl-data/hf-cache/hub/models--Qwen--Qwen2.5-VL-7B-Instruct/... has no files matching ['*.safetensors', '*.bin', '*.pt']; will attempt download. (log line ~4521).
• [FACT] The forced download hit the rate limit: httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/Qwen/Qwen2.5-VL-7B-Instruct' followed by 429 Too Many Requests: you have reached your 'api' rate limit. ... 0/3000 requests remaining in current 300s window (current plan: team).
• [FACT] Stack trace path: scheduler.py:788 init_tp_model_worker → tp_worker.py:378 → model_runner.py:1443 load_model → loader.py:473 _prepare_weights → weight_utils.py:568 download_weights_from_hf → hf_file_system.py ls → hf_api.repo_info → 429.
• [FACT] Server killed: Received sigquit from a child process → kill_process_tree → Server process exited with code -9 (SIGKILL).
• [FACT] Qwen2-VL-7B-Instruct MMMU dataset load failed 3/3 attempts (Attempt 1/2/3 failed with error: Failed to load MMMU datasets).
• [FACT] aiter unchanged vs Dockerfile default (dev image v0.1.14rc1.dev240+g7d604afe5 preserved; AITER_COMMIT_OVERRIDE empty). [CI-AITER-CHECK] markers confirm no rebuild.
• [FACT] Job exited 255; harness logged Exit code 255 indicates test failure (not network issue), not retrying.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] Infrastructure / external dependency: HuggingFace Hub API rate-limit (429). The 429 is printed verbatim in the failing stack trace; the server died as a direct consequence.
  • Supporting: 0/3000 requests remaining in current 300s window; both the weight download and the MMMU dataset load failed against huggingface.co; local cache had no weight files so a live download was mandatory.
  • Disconfirming: none — this is directly observable, not inferred. (Note the underlying trigger — why the local cache lacked the weight files — is not visible in this log.)
  • How to verify: re-run the job off-peak or with a pre-warmed /sgl-data/hf-cache containing the safetensors; confirm HF_HUB_OFFLINE/local-snapshot path resolves without a network call.
• [LOW] Stale/incomplete CI HF cache for Qwen2.5-VL-7B-Instruct. The snapshot dir existed but contained no weight files, forcing the online path that then got rate-limited.
  • Supporting: ... has no files matching ['*.safetensors', '*.bin', '*.pt']; will attempt download.
  • Disconfirming: cannot confirm from this log whether the cache is normally complete; may be a transient cache-population race rather than a persistent gap.
  • How to verify: inspect the runner's /sgl-data/hf-cache contents for this model; check whether other models in the same run resolved from cache.

No sglang code commit is implicated — the failure is in the external HF download path, not in changed sglang logic. No git-bisect is warranted for today's symptom.

In-flight Fix Check

• In-flight fix: ✅ #23400 "ci: prefetch HF snapshots and skip redundant metadata calls to survive 429s" (open since 2026-04-21, last updated 2026-04-23) — directly targets this exact 429 failure mode. Chase reviewers; do NOT open a duplicate. (Searched repo:sgl-project/sglang is:pr is:open HF_HUB_OFFLINE OR rate limit OR MMMU download.)

Suggested Triage Steps

• Verify #23400 still applies cleanly and would cover the VLM accuracy suite's weight + dataset fetch paths; if so, prioritize merging it to stop the recurring 429s.
• Pre-warm / repair the runner HF cache so Qwen2.5-VL-7B-Instruct safetensors resolve locally (eliminates the live download entirely).
• Consider setting HF_HUB_OFFLINE=1 (the log shows CI_OFFLINE: Launching server HF_HUB_OFFLINE=0) once caches are guaranteed complete, so a missing file fails fast instead of hammering the API.
• Because each recent run fails on a different model/cause, after the 429 infra issue is resolved, re-run to surface the next real failure (Jun 26 was an accuracy failures=1) — that one likely needs separate code-level investigation.

Status (factual, no priority assignment)

• Persistence: test_vlms_mmmu_eval_amd.py has failed in every completed scheduled run Jun 20–27; today's 429 symptom is first observed in this run (Jun 27).
• Scope: 1 job (nightly-accuracy-2-gpu-vlm-rocm720); the 429 rate-limit pattern is environment-wide and could affect any AMD/NVIDIA job that performs live HF downloads.
• Blocked work: VLM MMMU accuracy signal on AMD ROCm 7.2 (2-GPU) is unavailable.
• In-flight fix: ✅ #23400 (open since 2026-04-21) — awaiting merge.

nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 — failed step(s): Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp4.py	TestDeepseekV4ProFp4.setUpClass	Exception: Server process exited with code -9 (SIGKILL during model load)	Log
test/registered/amd/test_deepseek_v4_pro_fp4_cp.py	TestDeepseekV4ProFp4CPInterleave.setUpClass	Exception: Server process exited with code -9 (offline then online retry both -9)	Log

(test_deepseek_v4_pro_fp8.py passed today — Test Summary: 1/3 passed.)

Failure Summary

Both FP4 DeepSeek-V4-Pro server launches (tp8, and tp8 + attn_cp8 interleave) failed in setUpClass because the server subprocess was killed with signal 9 (SIGKILL) during the ~40-minute model-loading phase. Ranks 1–7 reached the post-load monitoredBarrier and then raised ValueError: TP rank N could finish the model loading, but there are other ranks that didn't finish loading … (e.g., OOM) or a slow node, because rank 0 vanished (was -9 killed) before reaching the barrier.

Regression Status

Flaky test (intermittent server kill during load) — first failure for these specific files is today.

Recent history of test_deepseek_v4_pro_fp4.py / test_deepseek_v4_pro_fp4_cp.py in job nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27	28296988041	…deepseek-v4-pro-rocm720	❌ Failed	setUpClass (both fp4 + fp4_cp)	Server exited -9
Jun 26	28256271771	…deepseek-v4-pro-rocm720	✅ Passed	— (fp8 failed instead that day)	—
Jun 25	28190877198	…deepseek-v4-pro-rocm720	✅ Passed (3/3)	—	—

Test files were added Jun 24 (9fd6d0ec, #28953); Jun 25 is the first scheduled run containing them. The job's overall conclusion has been failure on Jun 26 and Jun 27, but the failing test file rotates (fp8 on Jun 26, fp4+fp4_cp on Jun 27) — the hallmark of a flake, not a fixed regression. Lookback: 3 completed scheduled runs (the entire queryable history of these files).

Failure Cluster

DeepSeek-V4-Pro server SIGKILL (-9) during 8-way fp8 weight load on MI35X ROCm 7.2.

Facts (verified observations)

• Exception: Server process exited with code -9 at both setUpClass failures (log lines 24407 and 61591); fp4_cp also logged CI_OFFLINE: Offline launch failed (Server process exited with code -9), retrying with online mode... and the online retry also died -9.
• Rank-0 weight loading is far slower than peers, in both passing and failing runs:
  • Jun 27 (fail): TP0 Load weight end. elapsed=1941.03 s vs TP1–7 ≈ 645–676 s; ATTN_CP0 elapsed=1567.63 s vs CP1–7 ≈ 294–320 s.
  • Jun 25 (pass): TP0 elapsed=2213.86 s / 3868.11 s; ATTN_CP0 elapsed=1872.66 s — slower than the failing run, yet the suite passed 3/3.
• GPU VRAM is healthy at the time of death: Load weight end … avail mem=117.78 GB (fp4) / 100.12 GB (fp4_cp). No out of memory / hipError in the log → the -9 is host-side, not GPU-VRAM OOM.
• Weights load from a local mount: Found local HF snapshot … at /sgl-data/hf-cache/hub/models--deepseek-ai--DeepSeek-V4-Pro …; skipping download.
• Soft watchdog timeouts (300 s) fire repeatedly during load but are self.soft=True (advisory only) — not the kill source.
• Different files fail across runs and retries pass 3/3 (Test Summary: 3/3 passed appears later in both the Jun 26 and Jun 27 job logs), confirming intermittency.

Hypothesised Causes (with confidence)

• [MEDIUM] Host-memory pressure / Linux OOM-killer kills the rank-0 server process (SIGKILL = -9) during the concurrent 8-process fp8 weight load of the very large DeepSeek-V4-Pro checkpoint.
  • Supporting: exit code is exactly -9 (SIGKILL, the OOM-killer's signal); GPU VRAM is healthy (~100–118 GB free) so it is not a CUDA/HIP OOM; the failure is intermittent and the killed rank is the slowest loader (rank 0), consistent with it holding the most transient host memory longest.
  • Disconfirming: no oom-kill/Out of memory: Killed process dmesg line is captured in the job log (CI does not surface kernel logs), so host-OOM is inferred from the signal, not directly observed; a CI-side launch/health timeout sending SIGKILL cannot be fully excluded.
  • How to verify: re-run the job and capture dmesg//var/log/kern.log for an oom-kill entry; monitor host RSS during load; try --weight-loader-disable-mmap or staggering rank startup to lower peak host memory.
• [LOW] Slow /sgl-data mount I/O makes rank-0 load exceed a launch/health deadline.
  • Supporting: rank-0 load is pathologically slow (up to 3868 s) and weights come from the /sgl-data/hf-cache mount; an in-flight PR #29419 touches this mount.
  • Disconfirming: the Jun 25 passing run had even slower rank-0 loads and still succeeded, so slow load alone does not produce the -9.
• No hypothesis reaches HIGH. No commit in the Jun 25→Jun 26 window (9495737d82..c98d31143d, 44 commits) was found touching the weight-load / memory path for this model; the CI-config commits in range (b73e5721 adds an unrelated miles workflow) do not affect this job.

In-flight Fix Check

• In-flight fix: ❌ none found for an AMD MI35X ROCm-7.2 DeepSeek-V4-Pro server--9 flake. Related but not a direct fix: #29502 "[CI] Fix GB300 DSV4 Pro FP4 nightly" (targets NVIDIA GB300, not AMD) and #29419 "ci(amd): make sgl-data mount opt-in" (touches the weight mount). Searched search/issues?q=…is:pr+is:open+deepseek-v4-pro+nightly (8 open PRs, none matching this symptom on AMD).

Suggested Triage Steps

• Re-run job 83838515346 once or twice — if fp4/fp4_cp pass (or a different file fails), the flake classification is confirmed.
• On the MI35X runner, capture host dmesg/kernel log during the DeepSeek-V4-Pro load to confirm/deny an oom-kill SIGKILL on the rank-0 PID.
• Measure peak host RAM during the 8-way fp8 load; if near the limit, try lowering host-memory peak (e.g. --weight-loader-disable-mmap, fewer weight_loader_prefetch_num_threads, or staggered rank init).
• Investigate why rank 0 loads 3–5× slower than peers (GPU0 host-NUMA/affinity or /sgl-data mount contention — note TP0 CPU affinity [0,5..15]); coordinate with #29419 on the /sgl-data mount.

Status

• Persistence: Job-level failure on Jun 26 and Jun 27; but at the fp4/fp4_cp test-file level today (Jun 27) is the first failure — the failing file rotates run-to-run and retries reach 3/3 passed, i.e. intermittent.
• Scope: 1 job (nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720), 2 test files today. Sister job …-deepseek-v4-pro-mtp-rocm720 passed Jun 25 & Jun 26.
• Blocked work: DeepSeek-V4-Pro FP4 nightly signal on AMD MI35X ROCm 7.2 is unreliable.
• In-flight fix: ❌ none found directly; see related #29502 / #29419.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:33 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — 31 failure(s)

Run started (UTC): 2026-06-27 17:49 UTC
Last scanned (UTC): 2026-06-28 00:58 UTC
Commits: sglang 592f6c8

Summary

Counts: 31 jobs analyzed · 10 clusters · 1 never-passed code bug + 1 with in-flight fix · ~16 jobs are pure orchestration cancellations (no test failure) · all in run 28297015673, same-SHA rerun 28297867681 success

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Concurrency/fast-fail cancellation (no own test failure)	16 jobs (see cluster detail)	—	—	one-off; superseded by same-SHA run that passed	[FACT] concurrency cancel-in-progress + fast-fail cascade; not a code defect
2	HuggingFace 429 rate-limit during weight download (server exit -9)	extra-a-1gpu-large	test/registered/sessions/test_streaming_session_extra.py	TestStreamingSessionEagle.setUpClass	transient; same SHA passed on rerun	[FACT] HF Hub 429; in-flight #23400
3	HuggingFace 429 rate-limit	extra-a-1gpu-large	test/registered/quant/test_fp8kv_triton.py	TestFP8KVCacheTritonBackend.setUpClass	transient	[FACT] HF Hub 429
4	HuggingFace 429 rate-limit	stage-a-1gpu-small	test/registered/core/test_basic_sanity_eagle3.py	TestBasicSanityEagle3.setUpClass	transient; same SHA passed	[FACT] HF Hub 429
5	HuggingFace 429 rate-limit	stage-a-1gpu-small	test/registered/core/test_basic_sanity.py	TestBasicSanity.setUpClass	transient	[FACT] HF Hub 429
6	HuggingFace 429 rate-limit	stage-b-1gpu-small (12)	test/registered/spec/dflash/test_dflash.py	TestDFlashServerBase.setUpClass	transient; same SHA passed	[FACT] HF Hub 429
7	HuggingFace 429 rate-limit	stage-b-1gpu-small (6)	test/registered/models/test_compressed_tensors_models.py	TestCompressedTensorsLlama3FP8.setUpClass	transient	[FACT] HF Hub 429
8	HuggingFace 429 rate-limit	stage-b-1gpu-small (7)	test/registered/perf/test_vlm_perf_5090.py	TestVLMPerf5090.test_vlm_offline_throughput	transient; same SHA passed	[FACT] HF Hub 429; in-flight #23400
9	HuggingFace 429 rate-limit	stage-b-1gpu-small (10)	test/registered/dllm/test_llada2_mini.py, core/test_srt_engine.py, openai_server/features/test_json_mode.py	setUpClass / engine init	transient	[FACT] HF Hub 429 (3 tests)
10	HuggingFace weight-download stall → watchdog/timeout (not 429)	dsv4-pro-fp4	test/registered/amd/test_deepseek_v4_pro_fp4.py	server launch (filelock stall)	transient; same SHA passed	[HIGH] HF download stall, 300s soft-watchdog
11	HuggingFace weight-download stall	stage-b-1gpu-small-mi35x	test/registered/quant/test_quark_mxfp4.py	TestOnlineQuantizationMemoryLoadMOE.setUpClass	1× (job usually skipped)	[HIGH] HF xet_get stall → 1200s timeout
12	HuggingFace weight-download stall	stage-b-1gpu-small (1)	test/registered/moe/test_torch_compile_moe.py	N/A (cancelled mid-capture after 24min xet hang)	1×	[HIGH] HF xet_get hang, then concurrency cancel
13	torch.compile/Inductor AssertionError on ROCm (diffusion T2V)	multimodal-2gpu (1)	test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_*_t2v_*] (3 params)	no baseline (job usually skipped)	[LOW] Inductor decompose_triton_kernel_wrapper_functional assert
14	torch.compile/Inductor AssertionError on ROCm	multimodal-1gpu (0)	multimodal_gen/test/server/test_server_1_gpu.py	test_diffusion_generation[qwen_image_edit_2509_ti2i, wan2_1_t2v_1.3b_teacache_enabled]	no baseline	[MEDIUM] triton-custom specialize_impl ImportError → layernorm compile abort
15	Incomplete HF model download (diffusion: model_index/transformer/vae missing)	multimodal-1gpu (2)	multimodal_gen/test/single_test_file/test_generate_zimage_turbo_cli.py	TestZImageTurboCLI.test_output_file_path_alias, test_single_gpu	no baseline	[HIGH] Z-Image-Turbo cache incomplete (infra)
16	Incomplete HF model download (diffusion)	multimodal-2gpu (1)	test/server/test_server_2_gpu.py	[flux2_*, ideogram4_*, wan2_2_i2v_*] (3 errors)	no baseline	[FACT] FLUX.2/ideogram/Wan2.2 cache incomplete (infra)
17	causal Conv3D cat/pad fusion unsupported on ROCm	multimodal-2gpu (1)	test/server/test_server_2_gpu.py	[mova_360p_tp2, wan2_1_i2v_14b_480P/720P_2gpu] (3 params)	likely never-passed on ROCm	[LOW] Triton fallback CUDA-gated in parallel_conv.py:57; pre-existing ROCm gap
18	Inplace update to inference tensor (update_weights_from_disk)	multimodal-1gpu (3)	multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	5 failed + 4 setup errors	Never-passed (first scheduled exec)	[MEDIUM] non-offload param_data.copy_ missing inference_mode(False) (linear.py:575)
19	msgspec struct contract: missing input_embeds	stage-b-1gpu-small (6)	test/registered/utils/test_type_based_dispatcher.py	test_type_dispatcher_e2e_performance	Never-passed (shard usually skipped); deterministic	[HIGH] test omits required input_embeds; in-flight #29376
20	Perf throughput threshold shortfall (triton attention)	stage-b-1gpu-large (0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	no baseline	[LOW] 3176 < 3500 tok/s; flake vs ~9% regression unresolved
21	Scheduler watchdog timeout during CUDA graph replay	stage-b-1gpu-small (10)	test/registered/profiling/test_start_profile.py	test_start_profile_2	1× (intermittent)	[LOW] HIP CUDAGraph::replay stall → 300s watchdog

Cluster: Concurrency/fast-fail cancellation (no own test failure)

• Affected (16): extra-a-1gpu-small, dsv4-flash-fp4-fp8, stage-c-mi35x (0), stage-c-mi35x (1), stage-c-8gpu (0), stage-c-8gpu (1), stage-c-8gpu (2), stage-c-8gpu (3), stage-c-4gpu, stage-b-2gpu-large (0), stage-b-2gpu-large (1), stage-b-1gpu-large (1), stage-b-1gpu-small (9), multimodal-2gpu (0), multimodal-2gpu (2), multimodal-1gpu (1), nondeterministic.
• Shared facts: Run 28297015673 = 16 success / 10 failure / 23 cancelled; all cancellations clustered 18:24:17–18:24:32Z. Newer same-SHA schedule run 28297867681 created 18:24:16Z and concluded success. pr-test-amd-rocm720.yml:117-123 sets cancel-in-progress: true for schedule events; stage-c jobs additionally fast-fail-gated on stage-b.
• Top hypothesis: [FACT] two concurrent same-SHA schedule runs (17:48 + 18:24) → concurrency supersede + fast-fail cascade. No test-level defect in these jobs. Disconfirming: none.
• In-flight fix: N/A (orchestration, not a code bug).
• Suggested triage: investigate why two schedule runs fire ~36 min apart on the same SHA (same pattern on 06-26); consider cancel-in-progress: false or de-duplicating cron so scarce 8-GPU stage-c isn't discarded after 35 min queue.

Cluster: HuggingFace 429 rate-limit during weight download (largest genuine cluster, 6 jobs)

• Affected: rows 2-9 above (stage-a, extra-a, stage-b-small shards 6/7/10/12). Models: Llama-3.1-8B, Qwen2.5-VL-3B, RedHatAI FP8/FP8-KV, EAGLE3, LLaDA2-mini, Llama-3.2-1B.
• Shared facts: identical httpx ... 429 Too Many Requests ... 3000 api requests per 5 minutes (current: team) in download_weights_from_hf → hf_file_system.ls; server -9. Same SHA passed on rerun. Root: 49 concurrent jobs sharing one HF team quota.
• Top hypothesis: [FACT] transient HF API rate-limit (infra), not code. Disconfirming: none — same-SHA rerun green.
• In-flight fix: ✅ #23400 (prefetch HF snapshots / drop redundant metadata calls to survive 429s) — chase reviewers.
• Suggested triage: land #23400; pre-seed runner HF cache for CI models or use HF_HUB_OFFLINE=1 + higher-quota token; stagger job starts.

Cluster: HuggingFace weight-download stall/hang → watchdog/timeout (3 jobs, distinct from 429)

• Affected: rows 10-12 (dsv4-pro-fp4 filelock stall; mi35x test_quark_mxfp4 + test_torch_compile_moe xet_get hang).
• Shared facts: schedulers parked in snapshot_download/xet_get/_poll_until_acquired; no HTTP error, no Python exception — silent stall tripping 300s soft watchdog / 1200s file timeout.
• Top hypothesis: [HIGH] cold HF xet transfer stall for large checkpoints. Disconfirming: download eventually succeeds; could be runner disk/network contention vs pure HF latency.
• In-flight fix: ❌ none found.
• Suggested triage: pre-bake/cache large models on runners; consider HF_HUB_DISABLE_XET=1; add per-download wall-clock timeout+retry in ci_weight_validation.py.

Genuine code-level clusters (not infra)

• Inplace inference-tensor update (row 18, multimodal-1gpu (3)): Never-passed on first scheduled execution. [MEDIUM] non-offload load_weights_into_model copy at linear.py:575 lacks torch.inference_mode(False) (offload path at layerwise_offload.py:288 has it). Offload variants also fail (500) → possibly a second gap. Plus independent FLUX.2 fixture bug (No weights dir for transformer). In-flight: ❌ none. Triage: wrap copy in inference_mode(False)+no_grad, fix fixture path.
• input_embeds struct contract (row 19): [HIGH] test omits required field after msgspec conversion. In-flight: ✅ #29376 — unblock & merge.
• Diffusion Inductor assert / Conv3D-on-ROCm / incomplete diffusion downloads (rows 13-17): all in usually-skipped multimodal-gen jobs → no baseline. Conv3D (row 17) likely a pre-existing ROCm unsupported-feature gap (Triton fallback CUDA-gated in parallel_conv.py), not a regression.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Land in-flight #23400 (HF 429) + #29376 (input_embeds)	rows 2-9, 19	Fixes already exist; resolves the largest genuine cluster + the deterministic test bug
2	Fix concurrency/cron so two same-SHA schedule runs don't supersede each other	cluster 1 (16 jobs)	Eliminates the bulk of the noise; no code change to product
3	Pre-seed/cache large CI models + add download wall-clock timeout	rows 10-12, 15-16	Removes both 429 and silent-stall exposure
4	Reproduce test_update_weights_from_disk on MI325; verify inference_mode(False) fix	row 18	Only never-passed code bug with no in-flight fix
5	Confirm whether diffusion Inductor/Conv3D paths are ROCm-supported; reproduce with TORCHDYNAMO_VERBOSE=1	rows 13-14, 17	No baseline; needs a clean uncancelled run to classify regression vs pre-existing gap

---

Job	Failed Steps	Started
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang)	Run test	2026-06-27T17:49
call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	Run test	2026-06-27T17:49
dsv4-pro-fp4-amd-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4)	2026-06-27T18:10
dsv4-flash-fp4-fp8-amd-rocm720	N/A	2026-06-27T17:49
stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang)	Run test	2026-06-27T17:49
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-06-27T17:49
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	N/A	2026-06-27T17:49
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	Run test	2026-06-27T18:02
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-27T17:49
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	N/A	2026-06-27T17:49
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-06-27T18:02
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run test	2026-06-27T17:49
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0)	N/A	2026-06-27T17:49
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	N/A	2026-06-27T17:49
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-27T17:49
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-06-27T17:49
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0)	N/A	2026-06-27T17:49
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3)	N/A	2026-06-27T17:49
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-27T17:49
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-06-27T17:49
stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang)	Run test	2026-06-27T17:50
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-06-27T17:49
multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2)	N/A	2026-06-27T17:49
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-27T17:50
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	Run test	2026-06-27T17:50
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-27T17:49
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7)	Run test	2026-06-27T17:49
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1)	Run test	2026-06-27T17:50
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9)	Run test	2026-06-27T18:00
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)	Run test	2026-06-27T17:50
stage-c-test-4-gpu-amd-rocm720	N/A	2026-06-27T18:24

Per-Job Analysis

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/sessions/test_streaming_session_extra.py	TestStreamingSessionEagle.setUpClass	Exception: Server process exited with code -9 (preceded by HF 429 Too Many Requests loading lmsys/sglang-EAGLE3-LLaMA3.1-Instruct-8B)	Log
test/registered/quant/test_fp8kv_triton.py	TestFP8KVCacheTritonBackend.setUpClass	Exception: Server process exited with code -9 (preceded by HF 429 Too Many Requests loading RedHatAI/Meta-Llama-3-8B-Instruct-FP8-KV)	Log

(A third file, test/registered/spec/test_spec_standalone_extra.py, passed → Test Summary: 1/3 passed.)

Failure Summary

Two test files failed in setUpClass while launching their model servers. In both cases the scheduler hit a HuggingFace 429 Too Many Requests while fetching model metadata/weights, after which the server process was killed (exit code -9). No test assertions ran (Ran 0 tests). This is an environment/network failure during model fetch, not a test-logic failure.

Regression Status

Flaky / Infrastructure issue — HF Hub rate limiting (HTTP 429).

Recent history of the two failing test files in job call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-large-amd-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27 18:24	28297867681	job	✅ Passed	—	—
Jun 27 17:48	28297015673	job	❌ Failed	both setUpClass	HF 429 → -9
Jun 27 12:26	28289152816	run	✅ Passed	—	—
Jun 27 07:03	28281981946	run	✅ Passed	—	—

Decisive evidence: the very next scheduled run 28297867681 ran on the identical head SHA 592f6c84 and this same job passed. Same code, opposite result → not a code regression.

Failure Cluster

HuggingFace 429 rate-limit during server warmup model fetch (server killed -9)

Facts (verified observations)

• httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/lmsys/sglang-EAGLE3-LLaMA3.1-Instruct-8B/revision/main' (line 4171).
• httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/RedHatAI/Meta-Llama-3-8B-Instruct-FP8-KV' (line 4316).
• Both schedulers raised the 429 at Load weight begin and then Received sigquit from a child process; popen_launch_server reported Server process exited with code -9.
• The offline-cache fallback also failed for at least one model: LocalEntryNotFoundError ... outgoing traffic has been disabled / OSError: We couldn't connect to 'https://huggingface.co' (line 4443), so the model was neither online-fetchable (429) nor present in the local cache.
• aiter version unchanged (dev image preserved, no override) — irrelevant to a network 429.
• Same SHA passed in the subsequent scheduled run (see Regression Status).

Hypothesised Causes (with confidence)

• [FACT] HuggingFace Hub returned HTTP 429 (rate limit) during model metadata/weight fetch in setUpClass, killing the server before any test ran.
  • Supporting: explicit 429 stack traces for both models immediately before each -9 exit; offline cache miss for these models.
  • Disconfirming: (none) — this is directly observed, not inferred.
• [FACT] Not an sglang/aiter code regression.
  • Supporting: identical head SHA passed in the next scheduled run 28297867681; failure mode is purely network/environment.
  • Disconfirming: (none found).

In-flight Fix Check

• In-flight fix: ✅ #23400 ("ci: prefetch HF snapshots and skip redundant metadata calls to survive 429s") — directly targets this 429 class; chase reviewers rather than opening a duplicate. (Search: repo:sgl-project/sglang is:pr is:open 429 huggingface.)

Suggested Triage Steps

• No code action needed for this run — re-run is sufficient; the same SHA already passed on retry. Treat as transient infra.
• Track #23400 to reduce 429 exposure; it would prefetch/cache lmsys/sglang-EAGLE3-LLaMA3.1-Instruct-8B and RedHatAI/neuralmagic FP8-KV models so warmup doesn't hit the Hub metadata API under rate limiting.
• Investigate why the offline cache (HF_HUB_OFFLINE=1 fallback) lacked these two models — pre-warming the runner cache for these model IDs would make the offline fallback succeed when the Hub 429s.

Status (factual)

• Persistence: Single occurrence; the prior and immediately-following scheduled runs of this job passed (same SHA on the following run).
• Scope: 2 test files in one job in run 28297015673 (which was overall cancelled).
• Blocked work: None ongoing — transient HF rate limit, not blocking CI signal.
• In-flight fix: ✅ #23400 (open) addresses the 429 root environment cause.

call-pr-test-amd-extra-rocm720 / extra-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 592f6c849bf972370782d915c7a6528c158c2c80
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved; rebuild skipped. Dockerfile default also AITER_COMMIT=7d604afe5..., so override and default coincide.)

Failed Tests

Test File	Test Function	Error	Log
N/A — no test failed	N/A	Job externally cancelled at file 21/30 (test_self_unit_sweep_plan_builder.py) before it could run	Log

All 21 test files that started (0/30…20/30) completed with OK. No assertion, crash, or timeout was observed.

Failure Summary

This job did not fail on a test. The entire scheduled run 28297015673 was cancelled (run conclusion cancelled), and this job's "Run test" step was terminated mid-execution at file 21/30 with ##[error]The operation was canceled.. Every kv_canary/mock_model test file that executed before the cancellation passed (Ran N tests ... OK), including the canary-raise e2e test (file 5) whose expected RuntimeError: kv_canary violation is part of normal test behaviour, not a failure.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

A newer scheduled run of the same workflow, 28297867681 (created 2026-06-27 18:24:16 UTC, same head_sha 592f6c849b), started ~14 s before this run's job was cancelled at 18:24:30 UTC. The concurrency group cancelled the older in-progress run. The newer run completed success, and its sister job (extra-a-test-1-gpu-small-amd-rocm720) passed.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date (UTC)	Run	Conclusion	Head SHA
Jun 27 18:24	28297867681	✅ success	592f6c849b
Jun 27 17:48	28297015673	⚪ cancelled (this run)	592f6c849b
Jun 27 12:26	28289152816	✅ success	3306233961
Jun 27 07:03	28281981946	✅ success	9214b9338f
Jun 27 00:37	28273097445	✅ success	13b5bd962a

Failure Cluster

Run cancelled by concurrency (superseded by a newer scheduled run on the same SHA) — orchestration artifact, not a code/test fault.

Facts (verified observations, no interpretation)

• Run 28297015673: status=completed, conclusion=cancelled, event=schedule, head_sha 592f6c849b (verified via API).
• This job (83838596125): conclusion=cancelled; step 6 "Run test" conclusion cancelled; started 17:49:27 UTC, completed 18:24:32 UTC.
• Log line 93388: ##[error]The operation was canceled. immediately after Begin (21/30): test_self_unit_sweep_plan_builder.py.
• Files 0/30–20/30 all printed OK (e.g. line 4683 file 0 OK, line 24428-range files, line 93358 file 20 Ran 4 tests ... OK).
• Line 21469: RuntimeError: kv_canary violation: launch_tag=TAIL_K_FULL ... verify_real_kv_hash — occurred inside file 5 (test_self_e2e_perturb_raise.py), which then reported Ran 1 test in 43.505s / OK (line 21561-21563). This is the test exercising the canary RAISE path; it is expected and passed.
• Newer run 28297867681: created 18:24:16 UTC, same head_sha, conclusion=success; its extra-a-test-1-gpu-small job 83843620979 conclusion=success (verified via API).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by GitHub Actions concurrency because a newer schedule run on the identical head_sha started ~14 s before the cancel.
  • Supporting: newer run 28297867681 created 18:24:16 UTC; this job cancelled 18:24:30 UTC; both runs share head_sha 592f6c849b. Step 6 conclusion is cancelled, not failure. All executed test files passed.
  • Disconfirming: (none found) — no test error, crash, or timeout precedes the cancel marker in the log.
  • How to verify: inspect the workflow's concurrency: block in pr-test-amd-rocm720.yml (cancel-in-progress on schedule), and note the newer same-SHA run that succeeded.

In-flight Fix Check

Not applicable — no code/test failure to fix. No fix PR is warranted for a concurrency-driven cancellation. (No API search performed because there is no failing test/library to key on.)

Suggested Triage Steps

• No action needed for code. Treat this job as a no-signal cancellation, superseded by the successful newer run 28297867681 on the same SHA.
• If these duplicate same-SHA scheduled runs are undesirable, review the cron schedule / concurrency settings in pr-test-amd-rocm720.yml to avoid two overlapping scheduled runs launching ~35 min apart on the same commit.

Status (factual, no priority assignment)

• Persistence: One-off cancellation in run 28297015673; the immediately following run on the same SHA passed.
• Scope: Run-wide — 23 jobs in this run ended cancelled (incl. this one), consistent with a single concurrency cancel of the whole run; not isolated to this test file.
• Blocked work: None — the kv_canary extra-a-test-1-gpu-small suite is verified green on this SHA by run 28297867681.
• In-flight fix: N/A (no failure).

dsv4-pro-fp4-amd-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; matches Dockerfile default for gfx950)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v4_pro_fp4.py	N/A (server launch — model load phase)	Scheduler watchdog timeout (watchdog_timeout=300, soft=True) during weight load → run cancelled	Log

This is not a test-assertion failure. The job runs a single entrypoint, python3 registered/amd/test_deepseek_v4_pro_fp4.py (model deepseek-ai/DeepSeek-V4-Pro, tp_size=8, SGLANG_DSV4_ACCURACY_ONLY=1). It never reached the GSM8K accuracy check — it hung during server startup while loading model weights.

Failure Summary

All 8 TP scheduler processes (TP0–TP7) blocked at the identical frame _poll_until_acquired (filelock/_api.py:552) → acquire → download_weights_from_hf (model_loader/weight_utils.py:554) while loading deepseek-ai/DeepSeek-V4-Pro weights. The soft scheduler watchdog (300 s) fired at 2026-06-27 18:24:22 (model init started ~18:19:13), the runner then printed ##[error]The operation was canceled. at 18:24:30. The model-weight download from HuggingFace stalled past the watchdog window; no safetensors load-progress lines were ever emitted.

Regression Status

Flaky / infrastructure issue — not a code regression.

Recent history of test_deepseek_v4_pro_fp4.py in job dsv4-pro-fp4-amd-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27 18:24	28297867681	dsv4-pro-fp4-amd-rocm720	✅ Passed	—	—
Jun 27 17:48	28297015673	dsv4-pro-fp4-amd-rocm720	❌ Cancelled (watchdog)	N/A	weight-load watchdog timeout
Jun 27 12:26	28289152816	dsv4-pro-fp4-amd-rocm720	✅ Passed	—	—
Jun 27 07:03	28281981946	dsv4-pro-fp4-amd-rocm720	✅ Passed	—	—

Decisive evidence: the next scheduled run 28297867681, launched ~35 min later on the exact same head SHA 592f6c84, passed this job. A same-commit pass rules out a code regression in this commit.

Failure Cluster

DeepSeek-V4-Pro weight-download stall → scheduler watchdog timeout during server warmup

Facts (verified observations, no interpretation)

• All 8 TP schedulers share the identical idle stack top: select (libc.so.6) → _poll_until_acquired (filelock/_api.py:552) → acquire (filelock/_api.py:506) → __enter__ (filelock/_api.py:581) → download_weights_from_hf (weight_utils.py:554) → _prepare_weights (loader.py:473) → load_weights (deepseek_v4.py:2178).
• Watchdog config from server_args: watchdog_timeout=300, soft_watchdog_timeout=300; [18:19:13] Set soft_watchdog_timeout since in CI.
• [2026-06-27 18:24:22 TP0..TP7] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True); 18:24:30 ##[error]The operation was canceled.
• No Loading safetensors / download-progress / it/s lines appear anywhere in the log — weight load never advanced past the filelock acquire.
• aiter version unchanged vs passing runs (dev image preserved, matches Dockerfile default).
• Same-SHA re-run 28297867681 passed; two prior scheduled runs on different SHAs also passed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Transient HuggingFace weight-download stall for deepseek-ai/DeepSeek-V4-Pro. One TP rank holds the HF download filelock and downloads (or stalls), the other 7 ranks block on _poll_until_acquired; the download did not complete inside the 300 s soft-watchdog window, so the scheduler timed out and CI cancelled the job.
  • Supporting: all 8 ranks parked on the HF download filelock; zero download-progress output; identical-SHA re-run passed; this is a network/cache-dependent path independent of the diff.
  • Disconfirming: cannot see why the download was slow (no network error printed) — could in principle be HF-hub latency, a stale lock, or runner disk/cache contention rather than pure network slowness. (none point to a code defect.)
  • How to verify: re-run the job (already done — 28297867681 passed); or pre-stage the DeepSeek-V4-Pro weights in the runner cache / set download_dir so the first cold pull doesn't race the watchdog.
• [LOW] Code regression in 592f6c84 or the window before it.
  • Supporting: failure landed on this SHA.
  • Disconfirming: the same SHA passed on re-run 28297867681 — a deterministic code regression would fail both times. Strongly disconfirmed.
  • How to verify: not actionable; already disproven by the same-SHA pass.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs is:pr is:open watchdog download weights (11 hits, all unrelated: e.g. #29178 Kimi-K2.6 INT4 MoE load, #28584 MI300 runner test). No PR targets the HF-download-stall / weight-load watchdog timeout.

Suggested Triage Steps

• Treat as a flake: the failure self-resolved on a same-SHA re-run, so no code action is needed for 592f6c84.
• To reduce recurrence, pre-warm / cache deepseek-ai/DeepSeek-V4-Pro weights on the mi35x runners (or point download_dir at a persistent cache) so a cold HF pull cannot race the 300 s soft watchdog at warmup.
• If the stall recurs, capture HF-hub network/latency on the runner during weight download (the log shows no explicit network error — need instrumentation to distinguish slow network vs. stale filelock).
• Consider whether the model-load phase should be exempt from / given a longer soft-watchdog budget on very large models, so a slow first download surfaces as a timeout error with diagnostics rather than a CI cancel.

Status (factual, no priority assignment)

• Persistence: single occurrence — run 28297015673 (2026-06-27 17:48Z); the immediately following same-SHA run and the two prior scheduled runs all passed.
• Scope: one job (dsv4-pro-fp4-amd-rocm720) in one run; not seen in sibling jobs.
• Blocked work: none currently — DeepSeek-V4-Pro FP4 GSM8K signal on ROCm 7.2 is green again on the same commit.
• In-flight fix: ❌ none found — flake; no fix PR required.

dsv4-flash-fp4-fp8-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: N/A (no log was produced — job was cancelled before any step ran, so no [CI-AITER-CHECK] markers exist)

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled while queued — never executed (0 steps, no runner assigned, log blob BlobNotFound)	Job

Failure Summary

The job dsv4-flash-fp4-fp8-amd-rocm720 did not fail a test — it was cancelled while still queued. The job has 0 steps, no runner was ever assigned, and its log blob returns BlobNotFound, all of which indicate it never reached execution. It waited ~35 min for a linux-mi35x-gpu-8 runner (scarce 8-GPU MI35x hardware) and was then cancelled at the workflow level.

Regression Status

Infrastructure issue (concurrency cancellation) — not a test regression.

The run 28297015673 (created 17:48:59 UTC) was cancelled at 18:24:17 UTC. A newer scheduled run 28297867681 on the identical head SHA 592f6c84 was created at 18:24:16 UTC — one second before this run's cancellation — superseding it via the workflow's concurrency group. In that newer run, the same dsv4-flash-fp4-fp8-amd-rocm720 job ran to completion successfully (8 steps, completed 20:55:30 UTC).

Recent history of job dsv4-flash-fp4-fp8-amd-rocm720 (pr-test-amd-rocm720.yml, same branch main, event=schedule, completed runs only):

Date	Run	Job Status	Note
Jun 27 18:24	28297867681	✅ Passed	Superseding run, same SHA 592f6c84
Jun 27 17:48	28297015673	⏹️ Cancelled (queued, never ran)	This run
Jun 27 12:26	28289152816	✅ Passed	—
Jun 27 07:03	28281981946	✅ Passed	—
Jun 27 00:37	28273097445	✅ Passed	—

Failure Cluster

Queued job cancelled by concurrency supersession (no execution)

Facts (verified observations, no interpretation)

• Job 83838596532 conclusion is cancelled, steps: 0, runner_name empty; requested labels ['linux-mi35x-gpu-8'].
• Log fetch returns <Error><Code>BlobNotFound</Code> — no log was ever written.
• Job started_at 2026-06-27T17:49:05Z, completed_at 2026-06-27T18:24:17Z (~35 min, entirely queue/wait time with no step execution).
• Run 28297015673 overall conclusion cancelled: of 49 jobs → 16 success, 10 failure, 23 cancelled.
• Newer run 28297867681 created 2026-06-27T18:24:16Z on the same head SHA 592f6c84, overall conclusion success; its dsv4-flash-fp4-fp8-amd-rocm720 job passed (8 steps, completed 20:55:30Z).
• Both runs were event=schedule, branch=main, triggering actor bingxche.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by concurrency supersession — a newer run on the same SHA started 1 second before cancellation, evicting this run's queued jobs.
  • Supporting: newer run 28297867681 created 18:24:16Z, this run cancelled 18:24:17Z (1s gap); both same SHA 592f6c84, same event/branch/actor; the same job passed in the newer run.
  • Disconfirming: (none found) — the exact timing coincidence and identical SHA make any other cause unlikely.
  • How to verify: inspect the concurrency: group config in pr-test-amd-rocm720.yml and confirm cancel-in-progress: true is set for scheduled/dispatched runs.
• [LOW] Runner scarcity for linux-mi35x-gpu-8 left the job queued ~35 min, widening the window in which a superseding run could cancel it.
  • Supporting: job never got a runner (0 steps, empty runner_name) over a 35-min queue; 8-GPU MI35x is scarce hardware.
  • Disconfirming: the same job completed fine in the superseding run, so the runner was eventually available; scarcity alone did not fail the job.
  • How to verify: check MI35x-8gpu runner availability/queue-depth metrics for the 17:49–18:24Z window.

In-flight Fix Check

• In-flight fix: ❌ none needed — no test or code failure occurred. This is a workflow-orchestration cancellation, not a defect. (No PR search performed because there is no failing test/library to key on.)

Suggested Triage Steps

• No action required for this job — it did not fail; it was superseded and the same job passed on the identical SHA in run 28297867681.
• If these phantom "cancelled" rollups are noisy in the tracker, confirm pr-test-amd-rocm720.yml concurrency settings auto-cancel duplicate scheduled runs by design, and consider filtering concurrency-cancelled (0-step, no-runner) jobs out of failure reporting.
• Separately, the real failures in this run live in other jobs (stage-a-test-1-gpu-small, stage-b-test-1-gpu-small shards 6/7/10, multimodal-gen-test-1-gpu shards 2/3, stage-b-test-1-gpu-small-amd-mi35x) — analyze those individually; they are unrelated to this cancelled job.

Status (factual, no priority assignment)

• Persistence: Not persistent — single cancelled instance; the same job passed in the immediately superseding run and in the 3 prior scheduled runs.
• Scope: This job was 1 of 23 jobs cancelled in run 28297015673 when the run was superseded; the cancellation is workflow-wide, not specific to this job.
• Blocked work: None — coverage for dsv4-flash-fp4-fp8-amd-rocm720 on SHA 592f6c84 was satisfied by the passing superseding run.
• In-flight fix: ❌ none needed — not a defect.

stage-a-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: dev image v0.1.14rc1.dev240+g7d604afe5, preserved — no override, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_basic_sanity_eagle3.py	TestBasicSanityEagle3.setUpClass	Exception: Server process exited with code -9 (HF 429 Too Many Requests)	Log
test/registered/core/test_basic_sanity.py	TestBasicSanity.setUpClass	Exception: Server process exited with code -9 (HF 429 Too Many Requests)	Log

Failure Summary

The stage-a-test-1-gpu-small-amd-rocm720 job ran 5 test files; 3 passed and 2 failed. Both failures occurred in setUpClass when popen_launch_server could not start the server: the scheduler crashed downloading meta-llama/Llama-3.1-8B-Instruct weights because the HuggingFace Hub API returned 429 Too Many Requests (quota of 3000 API requests per 5-minute window exceeded). The server process was then SIGKILL'd (exit -9).

Regression Status

Infrastructure issue (transient HF rate limit) — not a code regression.

Recent history of test_basic_sanity.py / test_basic_sanity_eagle3.py in pr-test-amd-rocm720.yml (branch main, event=schedule, completed runs only):

Date	Run	Head SHA	Status
Jun 27 18:24	28297867681	592f6c84	✅ Success (same SHA, immediate rerun)
Jun 27 17:48	28297015673	592f6c84	❌ This run (429 rate limit)
Jun 27 12:26	28289152816	33062339	✅ Success
Jun 27 07:03	28281981946	9214b933	✅ Success
Jun 27 00:37	28273097445	13b5bd96	✅ Success

The same commit passed on the next scheduled run, conclusively showing this is transient and SHA-independent.

Failure Cluster

HuggingFace Hub 429 rate limit during model weight download (server fails to launch → exit -9).

Facts (verified observations)

• httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/meta-llama/Llama-3.1-8B-Instruct' (log lines 4251, 4393).
• 429 Too Many Requests: you have reached your 'api' rate limit ... quota of 3000 api requests per 5 minutes period ... (current: team) (lines 4307–4310, 4449).
• The 429 is raised inside scheduler.py:run_scheduler_process → model_runner.load_model → weight_utils.download_weights_from_hf → hf_file_system.ls (verified stack trace, lines 4257, 4393–4460).
• Server died with code -9 (SIGKILL via kill_process_tree after sigquit from a child process), causing popen_launch_server to raise (lines 4467, 1027 in test_utils.py).
• 3 sibling tests in the same job PASSED (test_wave_attention_kernels.py, test_fused_rms_fp8_group_quant.py, test_awq_dequant.py) — these do not download gated HF weights, so they were unaffected.
• aiter unchanged (dev image default, no override) — [CI-AITER-CHECK] markers lines 4063–4069.
• Same head SHA 592f6c84 passed on the immediately following scheduled run.

Hypothesised Causes (with confidence)

• [FACT] HuggingFace Hub API rate limit (429). Weight download for the gated meta-llama/Llama-3.1-8B-Instruct repo hit the org-wide 3000-req/5-min quota.
  • Supporting: verbatim 429 errors in both server crashes; only the two tests that fetch HF weights failed; same SHA passed on rerun.
  • Disconfirming: (none found) — there is no code-level evidence implicating any sglang/aiter change.
  • How to verify: already verified — rerun 28297867681 on the same SHA passed.

In-flight Fix Check

Not a code defect, so no source fix is applicable. (For resilience, weight caching / pre-staging of CI models would avoid HF API hits, but that is a CI-infra improvement, not a fix for this commit.)

• In-flight fix: ❌ N/A — transient infrastructure issue, no code change required.

Suggested Triage Steps

• No code action needed — confirm via the green rerun 28297867681 (same SHA 592f6c84).
• If 429s recur frequently in AMD CI: consider pre-caching gated CI models into the runner image / a shared HF cache so stage-a tests run with HF_HUB_OFFLINE=1 and avoid HF API calls during weight load.
• Optionally add retry-with-backoff on 429 in the CI server-launch path (CI-infra, not product code).

Status

• Persistence: Single transient occurrence; resolved on the next scheduled run with the identical commit (~35 min later).
• Scope: 2 of 5 tests in this one job; only the tests that download gated HF weights. No other workflows implicated by evidence.
• Blocked work: None ongoing — CI signal recovered automatically.
• In-flight fix: ❌ N/A — transient HF rate limit, no code fix required.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:58 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/5)

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5; source: dev image preserved — [CI-AITER-CHECK] reports "Dev/patched version detected → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestOnlineQuantizationMemoryLoadMOE (setUpClass / server launch)	✗ TIMEOUT after 1200s — server never became ready; scheduler stuck in HF weight download	Log

Failure Summary

The test file test/registered/quant/test_quark_mxfp4.py ran 2 model classes. The dense class TestOnlineQuantizationMemoryLoadDense (Qwen3-8B) passed both test_gsm8k and test_peak_memory. The MoE class TestOnlineQuantizationMemoryLoadMOE (Qwen/Qwen3-30B-A3B-Instruct-2507) hung during server launch: the model was not in the local HF cache and the weight download from HuggingFace stalled, so the server never reached "ready" and the test runner killed it after the 1200 s per-file timeout. Job exited with code 255.

Regression Status

Infrastructure issue (transient model-weight download stall) — not a code regression.

This job ran in only ONE of the recent scheduled pr-test-amd-rocm720.yml runs; in every other recent scheduled run on main the job was skipped (matrix-conditional), so there is no comparable job-level pass/fail baseline. Same-SHA evidence below shows the failure did not reproduce when the workflow re-ran.

History of job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Job Status	Note
Jun 27 18:24	28297867681	⏭️ Skipped	same head SHA 592f6c84 as the failing run
Jun 27 17:48	28297015673	❌ Failed	this run (test_quark_mxfp4.py timeout)
Jun 27 12:26	28289152816	⏭️ Skipped	—
Jun 27 07:03	28281981946	⏭️ Skipped	—
Jun 27 00:37	28273097445	⏭️ Skipped	—

Failure Cluster

HuggingFace model-weight download stall during server warmup (Qwen3-30B-A3B MoE)

Facts (verified observations, no interpretation)

• The dense model class passed: GSM8K eval completed (Invalid: 0.000) and test_peak_memory ran; first server torn down cleanly ([CI Test Method] TestOnlineQuantizationMemoryLoadDense.test_peak_memory).
• Second server launched at 17:58:24 for Qwen/Qwen3-30B-A3B-Instruct-2507 --quantization quark_mxfp4.
• 17:58:36 Load weight begin. avail mem=287.18 GB; Local HF snapshot ... has no files matching ['*.safetensors']; will attempt download. → weights were NOT cached, a full download started.
• Per-shard download locks were acquired/released every ~4 s from 17:58:36 to the last one at 17:59:07 (effec719... acquired), then no further log output until the watchdog fired.
• Soft watchdog timeouts repeated every 5 min: 18:03:26 (TokenizerManager), 18:03:32 (Scheduler), 18:08:27, 18:08:33, 18:13:27, 18:13:33 — self.watchdog_timeout=300, self.soft=True.
• Final py-spy dump of the scheduler shows the main thread blocked in the weight-download path, and an active worker thread mid-download:
  • __init__ (model_runner.py:579) → load_model (model_runner.py:1443) → load_weights (qwen3_moe.py:1127) → download_weights_from_hf (model_loader/weight_utils.py:593) → ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py:1955) → snapshot_download (huggingface_hub/_snapshot_download.py:467)
  • active thread ThreadPoolExecutor-2_0: xet_get (huggingface_hub/file_download.py:566) → _download_to_tmp_and_move (file_download.py:1861) → hf_hub_download (file_download.py:1016).
• Final outcome: ✗ TIMEOUT: .../test_quark_mxfp4.py after 1200 seconds; Test Summary: 3/4 passed; First attempt failed with exit code 255 ... not retrying; Process completed with exit code 255.
• Also seen earlier (unrelated, infra): Local docker registry pull failed; falling back to public registry at 17:49:47 — environment had network friction in this run.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] HuggingFace weight download for Qwen/Qwen3-30B-A3B-Instruct-2507 stalled (hung in the HF xet_get transfer), so model loading never finished and the watchdog/test-timeout fired.
  • Supporting: weights were not cached (will attempt download); download progressed normally for ~30 s then went silent at 17:59:07; the scheduler stack at all three watchdog dumps is parked inside snapshot_download/xet_get; no Python exception, no GPU/HIP error, no OOM anywhere in the log.
  • Disconfirming: (none found) — there is no stack trace or error pointing at sglang model code; the dense model in the same job loaded and ran to completion.
  • How to verify: re-run the job; or pre-warm the HF cache for Qwen/Qwen3-30B-A3B-Instruct-2507 on the mi35x runner and confirm the MoE class then loads in <300 s.
• [LOW] HEAD commit 592f6c84 (mem-cache / HiCache / sparse KV pool split) introduced a regression.
  • Supporting: it is the head SHA of the failing run.
  • Disconfirming: the hang is in HF download, which executes before any mem-cache/KV-pool code runs; the dense model loaded fine on the same commit; and the same SHA's re-run did not reproduce a failure. No code-level link.
  • How to verify: not actionable — the failing code path (snapshot_download) is upstream of the changed code.

In-flight Fix Check

• In-flight fix: ❌ none found — open-PR searches for xet download and weight download timeout returned only unrelated MoE-weight-loading and runner-migration PRs (#29178, #28905, #28584). This is an infra/network stall, not a code defect, so no source fix is expected.

Suggested Triage Steps

• Re-run the job; a transient HF download stall is expected to clear on retry (the same-SHA workflow re-run 28297867681 did not reproduce it — the job was skipped there).
• Pre-warm / pin the HF cache for Qwen/Qwen3-30B-A3B-Instruct-2507 on the mi35x runner so first-run cold-download latency cannot exceed the 1200 s file timeout.
• Consider making ci_download_with_validation_and_retry (model_loader/ci_weight_validation.py) enforce a per-download wall-clock timeout + retry so a stuck xet_get fails fast instead of consuming the whole test budget. (Investigation direction, not a directive.)
• If cold-download time for a 30B model legitimately approaches the limit, raise the per-file timeout for this test or split the MoE class into its own slot.

Status (factual, no priority assignment)

• Persistence: 1 occurrence — run 28297015673; not reproduced on the same-SHA re-run.
• Scope: single job stage-b-test-1-gpu-small-amd-mi35x-rocm720, single test file (test_quark_mxfp4.py), MoE class only; dense class passed.
• Blocked work: blocks the quark_mxfp4 MoE accuracy/memory signal on mi35x ROCm 7.2 when it triggers, but appears infra-transient.
• In-flight fix: ❌ none found — infra/network stall, no code fix applicable.

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image — Dockerfile default AITER_COMMIT=7d604afe5... matched the pre-baked dev build, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
N/A — target job did not run	N/A	Job cancelled (run-wide fast-fail cascade); 0 steps executed	Job

The target job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0) 83838610237 has conclusion cancelled with zero recorded steps — it never reached its test phase. Its own log blob is gone (BlobNotFound), consistent with a job cancelled before/at startup. The cancellation was triggered by sibling-job failures elsewhere in the run.

Failure Summary

This job did not fail on its own tests. The entire run 28297015673 concluded cancelled (16 success · 10 failure · 23 cancelled across 49 jobs). The real failures that tripped the fast-fail and cancelled this stage-c job were HuggingFace network problems during model-weight download:

• stage-a-test-1-gpu-small-amd-rocm720 83838610220: httpx.HTTPStatusError: 429 Too Many Requests for meta-llama/Llama-3.1-8B-Instruct, → FAILED (errors=1), exit 255 (log line).
• stage-b-test-1-gpu-small-amd-mi35x-rocm720 83838610236: test_quark_mxfp4.py ✗ TIMEOUT after 1200s — server stuck in snapshot_download (ci_weight_validation.py → HF), TokenizerManager watchdog timeout, exit 255 (log line).

Regression Status

Infrastructure issue (transient) — NOT a code regression in this job.

A re-run of the same head SHA 592f6c84 — run 28297867681, 36 min later — concluded success, conclusively exonerating the code at this SHA.

Recent scheduled runs of pr-test-amd-rocm720.yml (same branch main, event=schedule, completed only):

Date (UTC)	Run	Conclusion	Head SHA
06-27 18:24	28297867681	✅ success (same-SHA rerun)	592f6c84
06-27 17:48	28297015673	❌ cancelled (this run)	592f6c84
06-27 12:26	28289152816	✅ success	33062339
06-27 07:03	28281981946	✅ success	9214b933
06-27 00:37	28273097445	✅ success	13b5bd96
06-26 18:35	28257868547	cancelled	c98d3114
06-26 18:04	28256293905	cancelled	c98d3114

Failure Cluster

HuggingFace model-download failure during CI (429 rate-limit / snapshot_download stall) → fast-fail cancellation cascade.

Facts (verified observations, no interpretation)

• Target job 83838610237: conclusion=cancelled, steps=[] (empty), started 17:49:17Z, completed 18:24:17Z. Its own log returns BlobNotFound.
• Run 28297015673: event=schedule, branch=main, conclusion=cancelled; 49 jobs → 16 success, 10 failure, 23 cancelled.
• stage-a 83838610220 log: two 429 Too Many Requests for meta-llama/Llama-3.1-8B-Instruct (lines 4251, 4393), FAILED (errors=1), Process completed with exit code 255.
• stage-b mi35x 83838610236 log: test_quark_mxfp4.py ✗ TIMEOUT ... after 1200 seconds (line 6263); py-spy stack shows snapshot_download → ci_download_with_validation_and_retry (weight_utils.py:593) hung; TokenizerManager watchdog timeout (300s).
• aiter unchanged: dev image v0.1.14rc1.dev240+g7d604afe5 (7d604afe5, matches Dockerfile default) on both mi30x and mi35x runners ([CI-AITER-CHECK] markers).
• Same-SHA rerun 28297867681 = success.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The run was cancelled by a fast-fail cascade triggered by transient HuggingFace download failures (429 rate-limiting in stage-a; snapshot_download stall → 1200s timeout in stage-b mi35x). The target stage-c job was collateral — cancelled before running any step.
  • Supporting: target job has 0 steps and cancelled conclusion; concurrent sibling jobs show verbatim 429 Too Many Requests and a HF snapshot_download hang in the stack trace; both are network/registry-side, not code paths in the diff.
  • Disconfirming: (none found) — same-SHA rerun passing rules out a code regression.
  • How to verify: already verified — rerun 28297867681 of the identical SHA succeeded.
• [LOW] aiter-related cause.
  • Supporting: none.
  • Disconfirming: aiter version identical to passing runs; no aiter symbol in any stack trace; failures are pure HF network errors.

In-flight Fix Check

• In-flight fix: ❌ none needed — transient HF infrastructure failure, already self-resolved by the same-SHA rerun. (Searched is not applicable: no code defect to fix.) Note: a maintainer branch fix/rerun-failed-ci-cancelled-workflows appears in the checkout logs, indicating ongoing work to auto-rerun cancelled CI runs — likely the mechanism that produced the passing rerun.

Suggested Triage Steps

• No code action required for this job — treat as a flaky/infra cancellation; the same-SHA rerun 28297867681 is green.
• To reduce recurrence of the underlying HF 429 / download-stall pattern: confirm CI uses a HF token / local model cache / mirror for meta-llama/Llama-3.1-8B-Instruct and the test_quark_mxfp4 model, so weight downloads don't hit public rate limits.
• If failures recur, consider raising the per-test download timeout or adding download retry/backoff in ci_weight_validation.py so a slow HF response doesn't trip the 300s TokenizerManager watchdog → 1200s suite timeout.

Status (factual, no priority assignment)

• Persistence: Single occurrence at this SHA; same SHA passed on rerun 36 min later. Not persistent.
• Scope: Run-wide cancellation of 28297015673 (23 cancelled + 10 failed jobs); root failures isolated to HF-download-dependent jobs (stage-a, stage-b incl. mi35x, multimodal-gen).
• Blocked work: None ongoing — green same-SHA rerun exists.
• In-flight fix: ❌ none needed (transient HF infrastructure; auto-rerun produced a passing result).

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: 592f6c84 ([minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool #28713)
• aiter: 7d604afe5 v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_kimi_k2_instruct.py	N/A — never reached a verdict	Job cancelled during model weight loading (not a test assertion failure)	Log

This job did not produce a test failure. Step 7 "Run test" was cancelled (##[error]The operation was canceled. at 18:24:30) while the only test in this partition — test_kimi_k2_instruct.py (Kimi-K2-Instruct-0905, DeepSeek-V3 arch, TP=8) — was still loading model weights.

Failure Summary

The run loaded the Kimi-K2 model across 8 TP ranks; all ranks were blocked in download_weights_from_hf on a HuggingFace cache filelock (_poll_until_acquired, weight_utils.py:554) — the normal one-downloader-many-waiters pattern for a first-time download of a very large checkpoint. This crossed the 300 s soft watchdog (soft=True, log-only, does not abort) at 18:17:31 (TokenizerManager) and 18:17:38 (all schedulers). The job was then externally cancelled at 18:24:30, ~14 s after a duplicate scheduled run started — see Regression Status.

Regression Status

Infrastructure / cancellation — NOT a test failure, NOT attributable to the commit.

Two scheduled runs fired on the same head_sha 592f6c84 back-to-back:

Time (UTC)	Run	Conclusion	head_sha	Note
17:48:59	28297015673	❌ cancelled	592f6c84	this run — cancelled 18:24:30
18:24:16	28297867681	✅ success	592f6c84	successor; started 14 s before this was cancelled

[FACT] The successor run started 18:24:16 and this run's "Run test" step was cancelled 18:24:30 — a concurrency-group supersede cancellation (a newer run on the same ref cancels the in-flight one). In the successor run the stage-c-large-8-gpu jobs were skipped, so the Kimi test got no verdict there either.

[FACT] In the regular scheduled runs (28289152816, 28273097445, 28238681767), stage-c-test-large-8-gpu-amd-rocm720 is skipped — it does not normally fan out into partitioned jobs. There is therefore no green/red baseline for test_kimi_k2_instruct.py in the queryable scheduled-run window (last ~6 runs, Jun 25–27). Status for the test itself: Never-evaluated (no completed run with a verdict found).

Facts (verified observations)

• [FACT] Job conclusion = cancelled; only step 7 "Run test" cancelled, all build/setup steps success. Run conclusion = cancelled.
• [FACT] Single enabled test in this partition (3/4, 0-based id=2): test/registered/amd/test_kimi_k2_instruct.py (est 3600 s) — log line 6991.
• [FACT] Soft watchdog (300 s, log-only) fired at 18:17:31 / 18:17:38; all ranks' py-spy dumps show identical stack: run_scheduler_process → ModelRunner.load_model → deepseek_weight_loader.do_load_weights → _prepare_weights → download_weights_from_hf → filelock _poll_until_acquired (log lines 7290–7624, 8138–8187).
• [FACT] 10 sibling jobs in this run failed independently (stage-a/stage-b 1-gpu, multimodal-gen) and 23 were cancelled; those are separate test files, out of scope for this partition's analysis.
• [FACT] aiter unchanged dev build, no override; AITER_COMMIT_OVERRIDE empty throughout.

Hypothesised Causes

• [FACT] Job was cancelled by a concurrency supersede (duplicate same-SHA scheduled run), not by a test failure.
  • Supporting: successor run 28297867681 started 18:24:16 on the same SHA; this run's only running step was cancelled 18:24:30; build steps all succeeded.
  • Disconfirming: (none found) — no assertion error, crash, or non-zero test exit appears in the log; the test never completed loading.
  • How to verify: re-run stage-c-test-large-8-gpu-amd-rocm720 partition 2 in isolation on 592f6c84 and let test_kimi_k2_instruct.py finish weight loading.
• [LOW] Slow/contended HF weight download could make this test exceed its budget even without cancellation (soft watchdog already tripped at ~5 min into load).
  • Supporting: all 8 ranks parked on the HF download filelock for >5 min; Kimi-K2 is a ~1T-param checkpoint, page_size=1.
  • Disconfirming: soft watchdog is log-only and does not abort; the job was killed externally well before any hard timeout (per-file timeout was 5400 s), so we never observed an actual load-timeout failure.
  • How to verify: check whether the HF cache for moonshotai/Kimi-K2-Instruct-0905 is pre-warmed on linux-mi325-8gpu-sglang runners; time a clean load.

In-flight Fix Check

• No fix is warranted — this is a cancellation, not a code defect. (Searched is:pr is:open kimi_k2 / test_kimi_k2_instruct — nothing relevant to a CI cancellation.)
• In-flight fix: ❌ none needed — cancellation, not a regression.

Suggested Triage Steps

• Treat this job as non-signal: it was superseded by a duplicate same-SHA scheduled run; re-run the stage-c large-8-gpu partition standalone to get a real verdict for test_kimi_k2_instruct.py.
• Investigate why two scheduled pr-test-amd-rocm720 runs fired on the same SHA 35 min apart (cron double-fire / manual re-trigger) — the concurrency policy then cancelled the first, wasting an 8-GPU run.
• Separately, the 10 sibling failures (stage-a/stage-b 1-gpu, multimodal-gen) in this run are real and deserve their own per-job analysis — they are unrelated to this partition.
• Confirm whether stage-c-test-large-8-gpu is intended to fan out on schedule (it was skipped in adjacent runs); if it should run, ensure HF weights for Kimi-K2 are pre-cached to avoid the multi-minute download blocking model load.

Status

• Persistence: Single occurrence; this partition is normally skipped, so no recurring history. The cancellation is a one-off concurrency supersede.
• Scope: This job only for the Kimi test; the run-wide cancellation hit 23 jobs after 10 independent sibling failures.
• Blocked work: None from this job — test_kimi_k2_instruct.py produced no verdict (never-evaluated on this SHA).
• In-flight fix: ❌ none needed — cancellation, not a regression.

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 592f6c84 — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: N/A — the job log blob has expired (BlobNotFound), so [CI-AITER-CHECK] markers cannot be read.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure — job was cancelled, not failed	Job

This job did not fail. Its conclusion is cancelled (verified via the jobs API). It recorded zero steps and was terminated ~35 min after starting. There is no failing test to report, and the log blob is no longer retrievable (BlobNotFound).

Failure Summary

Job stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 0) was cancelled at 2026-06-27T18:24:17Z, not failed. The entire parent run 28297015673 (PR Test ROCm 7.2 (AMD), event=schedule) has conclusion cancelled — of its 49 jobs: 16 success, 10 failure, 23 cancelled (this job among them). No test execution output survives.

Regression Status

Infrastructure / orchestration event (not a regression).

This job carries no test-level failure, so a test-file regression table does not apply. Run-level context for this job's workflow (same branch main, event=schedule, completed runs only):

Date	Run	Run Conclusion	This Job's Status	Note
Jun 27 18:24	28297867681	✅ success	⏭️ skipped	Same SHA 592f6c84; started the same second the cancelled run ended
Jun 27 17:48	28297015673	⛔ cancelled	⛔ cancelled	The run under analysis
Jun 27 12:26	28289152816	✅ success	—
Jun 27 07:03	28281981946	✅ success	—
Jun 27 00:37	28273097445	✅ success	—

A re-run on the identical commit 592f6c84 (28297867681) concluded success, indicating no genuine breakage of this job at this SHA.

Facts (verified observations)

• [FACT] Job conclusion is cancelled (jobs API), started 17:49:17Z, completed 18:24:17Z, 0 steps recorded.
• [FACT] Parent run 28297015673 conclusion is cancelled; job-conclusion breakdown: 16 success / 10 failure / 23 cancelled.
• [FACT] The job log is unavailable — the logs endpoint returns <Code>BlobNotFound</Code> (blob expired/purged), so no test output, stack trace, or [CI-AITER-CHECK] aiter version is recoverable.
• [FACT] 10 sibling jobs in the same run genuinely failed; earliest failures completed at 18:00:25Z–18:02:00Z (e.g. stage-a-test-1-gpu-small-amd-rocm720 at 18:00:46Z), ~22 min before this job was cancelled.
• [FACT] A re-run on the same SHA (28297867681) started at 18:24:16Z and concluded success; this job appears there as skipped.

Hypothesised Causes

• [LOW] Fast-fail cascade. 10 jobs failed starting at ~18:00Z; the pr-test-amd fast-fail/finish gating may have cancelled remaining in-progress jobs (including this one) at 18:24Z.
  • Supporting: multiple genuine sibling failures preceded the cancellation; cancellation is the expected fast-fail behavior for downstream/in-progress jobs.
  • Disconfirming: this job ran a full ~35 min before being cancelled rather than being cancelled promptly after the first failure (18:00Z); the timing also coincides exactly with a new run starting.
  • How to verify: inspect the pr-test-amd-rocm720-finish / wait-gate job logic and the run's cancellation event in the Actions UI.
• [LOW] Concurrency / superseding re-run. Run 28297867681 started at 18:24:16Z — the same second this run ended — on the identical SHA, consistent with a concurrency-group cancellation or a "re-run" that cancelled the prior in-flight run.
  • Supporting: exact-second alignment of the new run's start with this run's cancellation; same head SHA; this job is skipped in the new run.
  • Disconfirming: cannot distinguish a manual re-run from an automatic concurrency cancel without the run's event metadata.
  • How to verify: check the run's run_attempt / trigger actor in the Actions UI to see if it was a manual re-run vs. concurrency cancel.

In-flight Fix Check

Not applicable — there is no failing test or error signature to search against. This job was cancelled, not broken.

Suggested Triage Steps

• Treat this job as a non-failure: no action needed for stage-b-test-2-gpu-large-amd-rocm720 itself — it was cancelled and the same-SHA re-run 28297867681 passed.
• Redirect investigation to the 10 jobs that genuinely failed in run 28297015673 (e.g. stage-a-test-1-gpu-small-amd-rocm720, the stage-b-test-1-gpu-small shards 6/7/10, and the multimodal-gen shards 2/3) — those are the real failure signal.
• If accurate per-job logs are needed later, fetch them before the ~log-retention window expires; this run's logs are already purged (BlobNotFound).

Status (factual)

• Persistence: Single cancellation event at this SHA; the immediately following same-SHA run passed — no persistence.
• Scope: 23 of 49 jobs in run 28297015673 were cancelled run-wide; this job is one of them.
• Blocked work: None attributable to this job — the genuine signal is the 10 failed sibling jobs.
• In-flight fix: N/A (no failure to fix for this job).

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — Dockerfile default, no override). Note: this marker is read from the sibling stage-b mi35x job 83838610236; the target job's own log is empty because it never ran.

Failed Tests

The target job itself ran no tests. It was cancelled (conclusion cancelled, zero recorded steps) by a fast-fail cascade, not a test failure. The genuine failures in the same run that caused the cascade are listed below for context.

Job	Test File	Test Function	Error	Log
stage-b-test-1-gpu-small-amd-mi35x (gates target)	test/registered/quant/test_quark_mxfp4.py	(whole-file timeout)	timeout after 1200s (hang; py-spy could not dump stack)	Log
stage-a-test-1-gpu-small (pipeline gate)	test/registered/core/test_basic_sanity.py	server warmup	Server process exited with code -9 (after HF 429 Too Many Requests on meta-llama/Llama-3.1-8B-Instruct)	Log
stage-a-test-1-gpu-small (pipeline gate)	test/registered/core/test_basic_sanity_eagle3.py	server warmup	Server process exited with code -9 (after HF 429 Too Many Requests)	Log

Failure Summary

The target job stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) (83838610246) was cancelled at 18:24:17 UTC with no steps executed — it never reached any test. The whole run 28297015673 concluded cancelled (16 success, 10 failure, 23 cancelled). Two genuine upstream/sibling failures triggered the fast-fail: a 1200 s timeout in test_quark_mxfp4.py on the mi35x stage-b job (which gates this mi35x stage-c job), and HF 429-induced server crashes in stage-a's basic-sanity tests.

Regression Status

Infrastructure issue / fast-fail cascade — target job did not execute. No test-level regression can be attributed to this job.

For the gating failures, no clean baseline exists: the mi35x stage-b/stage-c jobs and stage-a were skipped in all 5 surrounding scheduled runs (28289152816, 28281981946, 28273097445, 28257868547, and the same-SHA retry 28297867681), so test_quark_mxfp4.py has no comparable completed run to compare against.

Date (UTC)	Run	mi35x stage-b status	Note
06-27 17:48	28297015673	❌ ran & failed (quark timeout)	this run; target stage-c cancelled by fast-fail
06-27 18:24	28297867681	⊘ skipped	same SHA 592f6c84; succeeded only because test jobs were skipped
06-27 12:26	28289152816	⊘ skipped	—
06-27 07:03	28281981946	⊘ skipped	—
06-27 00:37	28273097445	⊘ skipped	—

Failure Cluster

Fast-fail cancellation cascade (target job) — driven by (a) test_quark_mxfp4.py 1200 s hang on mi35x and (b) HuggingFace 429 rate-limit → server exit -9 in stage-a.

Facts (verified observations)

• Target job 83838610246: status=completed, conclusion=cancelled, 0 steps, ran 17:49:17→18:24:17 UTC (queued/idle, never executed tests). [FACT]
• Run 28297015673 conclusion cancelled; job tally 16 success / 10 failure / 23 cancelled. [FACT]
• mi35x stage-b 83838610236: ✗ FAILED: test/registered/quant/test_quark_mxfp4.py (timeout after 1200s); 3/4 files passed; repeated Pyspy failed ... Failed to get stack traces on pid 9499. Completed 18:15:44 UTC. [FACT]
• stage-a 83838610220: httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/meta-llama/Llama-3.1-8B-Instruct' followed by Exception: Server process exited with code -9 → test_basic_sanity.py and test_basic_sanity_eagle3.py both ✗ FAILED (exit code 1). Completed 18:00:46 UTC. [FACT]
• aiter on mi35x runner: Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild, no AITER_COMMIT_OVERRIDE. [FACT]
• Same-SHA run 28297867681 "succeeded" but only because all 17 test jobs were skipped (8 success / 17 skipped) — it does not re-validate the failed tests. [FACT]

Hypothesised Causes (with confidence)

• [FACT] The target job's cancelled status is a fast-fail cascade, not a code/test failure. The job emitted zero steps and was terminated at the run-cancellation time (18:24:17). Nothing in this job can be a regression.
• [HIGH] stage-a's basic-sanity failures are infrastructure (HF rate-limiting), not an sglang regression.
  • Supporting: the 429 Too Many Requests on the HF model API directly precedes the exit -9 server crash in the log; model download was blocked.
  • Disconfirming: (none found) — the error is an external HTTP 429, unrelated to repo code.
  • How to verify: re-run stage-a with HF auth/cache warm; the 429 should disappear.
• [LOW] test_quark_mxfp4.py 1200 s timeout on mi35x (gfx950) — candidate hang in MXFP4 quark path or runner contention.
  • Supporting: file hung the full 1200 s budget; py-spy could not even dump native stacks (process unresponsive), consistent with a GPU-side hang rather than a slow-but-progressing test.
  • Disconfirming: no comparable completed baseline exists (mi35x stage-b skipped in all 5 surrounding scheduled runs), so it could equally be a one-off runner/contention timeout; cannot confirm it is a code regression.
  • How to verify: re-run test/registered/quant/test_quark_mxfp4.py standalone on an linux-mi35x-gpu-1 runner; if it consistently exceeds 1200 s, bisect MXFP4/quark commits.

In-flight Fix Check

• HF 429 / rate-limit: In-flight fix: ❌ none found — curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+429" and +huggingface+429 returned no matching open PR. This is transient infra; typically self-resolves.
• quark mxfp4 timeout: In-flight fix: ❌ none found — curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+quark+mxfp4" returned no matching open PR.

Suggested Triage Steps

• Treat the target job as not a real failure — it was fast-fail-cancelled and ran no tests. No action needed on this job specifically.
• For the cascade trigger, re-run the mi35x stage-b job (or just test_quark_mxfp4.py on linux-mi35x-gpu-1) to determine if the 1200 s timeout is reproducible vs a one-off hang/contention.
• If reproducible, bisect commits touching the MXFP4 / quark quantization path and capture a GPU hang trace (rocgdb / amd-smi), since py-spy could not dump.
• For stage-a, retry once HF rate-limit clears, or warm the model cache / use an HF token, to rule out the transient 429.
• For full mi35x signal on this commit, trigger a fresh run that actually executes the mi35x stage-b/stage-c jobs (the same-SHA retry only skipped them).

Status (factual)

• Persistence: single occurrence; the target job was cancelled, not failed. The two cascade-triggering failures appeared once in run 28297015673; no comparable mi35x baseline in the queryable window (all surrounding scheduled runs skipped these jobs).
• Scope: run-wide fast-fail cancellation (23 jobs cancelled, 10 failed) within pr-test-amd-rocm720.yml. The target job exercised none of its own tests.
• Blocked work: mi35x 8-GPU large-test signal for commit 592f6c84 is missing (cancelled before execution).
• In-flight fix: ❌ none found for either the HF 429 or the quark-mxfp4 timeout.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — [CI-AITER-CHECK] reported "Dev/patched version detected … skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_offline_throughput_with_triton_attention_backend	AssertionError: 3176.0154980368743 not greater than 3500 (retries exhausted)	Log

Plus a job-level cancellation (not a test): the Run test step (step 6) was terminated mid-suite — ##[error]The operation was canceled. at 2026-06-27T18:24:30Z, while the next file test/registered/sessions/test_streaming_session.py (file 1/11) was starting. See Log.

Failure Summary

The job ran the stage-b-test-1-gpu-large-amd suite (partition 0/2) under --continue-on-error. The perf test test_offline_throughput_with_triton_attention_backend failed because the measured Llama-3.1-8B offline output throughput was 3176.02 tok/s, below the AMD CI threshold of 3500 (test_bench_serving_1gpu_part1.py:120); the test retried to its maximum and still failed. The suite then advanced to the next file and the entire Run test step was cancelled before the partition completed, so the job's overall conclusion is cancelled, not failure.

Regression Status

Inconclusive — no clean comparable passing run in the queryable window. The stage-b-test-1-gpu-large-amd-rocm720 matrix child is change-detection–gated and is skipped in nearly every scheduled pr-test-amd-rocm720.yml run; when it does run it has been cancelled (concurrency). The perf test therefore rarely completes in scheduled runs.

Recent history of test_bench_serving_1gpu_part1.py (test_offline_throughput_with_triton_attention_backend) in job stage-b-test-1-gpu-large-amd-rocm720 (…, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Job Conclusion	Test File Status	Notes
Jun 27 18:24	28297867681	success	⏭️ Not run	Stage-b child skipped (same head SHA 592f6c849b); not comparable
Jun 27 17:48	28297015673	cancelled	❌ Failed	This run — perf 3176 < 3500, then step cancelled
Jun 26 18:04	28256293905	cancelled	❓ Unknown	Child 83720145730 cancelled; log no longer retrievable
Jun 27 (07:03 / 00:37), Jun 26 12:41, …	multiple	success	⏭️ Not run	Stage-b large child skipped in all

First observed failure for this test file: this run. No green run for this test file found in the lookback window (≈ last 12 scheduled runs / ~2.5 days) — every other run skipped or cancelled the job.

Facts (verified observations)

• Output token throughput (tok/s): 3176.02 in the Serving Benchmark Result block; assertion assertGreater(res["output_throughput"], 3500) at test_bench_serving_1gpu_part1.py:120 (AMD branch). Margin ≈ 9.3% below threshold.
• The test was retried via sglang/test/test_utils.py retry() and raised Exception: retry() exceed maximum number of retries. — i.e. all retries measured below threshold, not a single dip.
• Benchmark completed all 500 requests successfully (Successful requests: 500); this is a throughput-magnitude shortfall, not a crash/correctness error. Mean TTFT 16.1 s, Mean TPOT 144 ms.
• The next scheduled run 28297867681 was created at 18:24:16Z — 14 s before this run's cancellation at 18:24:30Z — and it skipped the stage-b large jobs. This timing is consistent with concurrency-based supersession of the in-progress run.
• aiter version unchanged vs Dockerfile expectation (dev build g7d604afe5 matches AITER_COMMIT=7d604afe5…); [CI-AITER-CHECK] confirms no rebuild.

Hypothesised Causes

• [LOW] Perf-threshold flake / runner-load throughput variance on the triton attention backend (3176 vs 3500, ~9% short).
  • Supporting: pure magnitude shortfall with 100% successful requests and no error/crash; perf benchmarks on shared MI325 runners are sensitive to co-tenant load and clocking; threshold is a hard cutoff.
  • Disconfirming: all retries failed (not a single dip), so a sustained ~9% regression cannot be excluded; no passing baseline exists in the window to prove the test normally clears 3500 here.
  • How to verify: re-run the partition on linux-mi325-1gpu-sglang and record output_throughput across ≥3 runs; if consistently <3500, treat as a real regression and bisect.
• [LOW] Genuine throughput regression in the Llama-3.1-8B triton-backend serving path landed in recent main.
  • Supporting: every retry stayed below threshold this run.
  • Disconfirming: no green baseline for this exact test in the window, so a "regression" cannot be distinguished from a chronically-marginal threshold; aiter is unchanged.
  • How to verify: bisect sglang main against the last run where this test was green (none currently queryable — would require an on-demand run on an older SHA).
• The cancelled conclusion itself is infrastructure, not a code fault: [FACT] the step was externally cancelled (superseding scheduled run), not failed by a test.

No hypothesis reaches MEDIUM: there is no retrievable passing baseline for this test in the scheduled-run history, so regression-vs-flake cannot be separated without a fresh re-run.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs repo:sgl-project/sglang is:pr is:open test_bench_serving throughput (2 hits, neither relevant: #27650 PCG/all-reduce, #24928 CI sandbox).

Suggested Triage Steps

• Re-run stage-b-test-1-gpu-large-amd-rocm720 partition 0 on linux-mi325-1gpu-sglang and capture output_throughput from the GitHub step summary across ≥3 runs to classify flake vs sustained regression for test_offline_throughput_with_triton_attention_backend.
• If consistently 3100–3300 tok/s, this is a marginal-threshold issue: maintainers can decide whether to bisect the triton-backend serving path or re-tune the AMD threshold (currently 3500 at test_bench_serving_1gpu_part1.py:120).
• Separately, investigate why scheduled pr-test-amd-rocm720.yml runs repeatedly cancel/skip the stage-b large matrix children (concurrency supersession by overlapping schedule runs) — this prevents any stable perf baseline for this suite.

Status

• Persistence: First observed this run (28297015673); no comparable completed run executed this test in the lookback window, so persistence is undetermined.
• Scope: 1 test (test_offline_throughput_with_triton_attention_backend) in 1 job; job overall cancelled by a superseding scheduled run before the partition finished.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 v0.1.14rc1.dev240+g7d604afe5 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected … → skipping rebuild, log step 6)

Failed Tests

Test File	Test Function	Error	Log
N/A — job cancelled, no test failed	N/A	##[error]The operation was canceled. (job conclusion = cancelled, not failure)	Log

This is not a test-level failure. The only test that ran to completion in this job, test/registered/perf/test_bench_serving_1gpu_part2.py, passed after one retry; test/registered/lora/test_lora_update.py (TestLoRADynamicUpdate.test_dynamic_lora_update_server) had only just started when the runner was cancelled.

Failure Summary

The Run test step was cancelled mid-execution (stage-b runs a batch of 10 test files; only 0/10 had completed). The job's overall conclusion is cancelled, not failure. The batch's first test (test_bench_serving_1gpu_part2.py) overran badly — elapsed=1514s vs estimated_time=900.0 (step 6) — because the embeddings server crashed once with exit code -9 and had to be relaunched, then passed. The job was then cancelled at the 0/10 → 1/10 test boundary at 18:24:31, while test_lora_update.py was loading.

Regression Status

Infrastructure / orchestration issue (fast-fail cancellation) — not a code regression.

Recent history of job stage-b-test-1-gpu-large-amd-rocm720 (same branch main, same event schedule, completed runs only):

Date	Run	Job Conclusion
Jun 27 17:48	28297015673	⚠️ Cancelled (this run)
Jun 27 12:26	28289152816	⊘ Skipped
Jun 27 07:03	28281981946	⊘ Skipped
Jun 27 00:37	28273097445	⊘ Skipped
Jun 26 18:35	28257868547	⊘ Skipped

This job is normally skipped in scheduled runs; in this run it executed and was cancelled before producing a verdict. No comparable passing/failing test-level baseline exists in the queryable window.

Failure Cluster

Job cancelled by workflow fast-fail cascade (no test-level failure in this job)

Facts (verified observations, no interpretation)

• Job conclusion = cancelled; only Run test step is cancelled, all other steps success (verified via Jobs API). [FACT]
• Run-level picture: of 49 jobs, 16 success · 10 failure · 23 cancelled; this job is one of the 23 cancelled. [FACT]
• Earliest sibling failures landed at 18:00:25 (multimodal-gen-test-1-gpu-amd-rocm720 (…, 2)) and 18:00:46 (stage-a-test-1-gpu-small-amd-rocm720); the workflow finisher pr-test-amd-rocm720-finish concluded failure at 19:06:20. [FACT]
• This job's first batched test took elapsed=1514s vs estimated_time=900.0 (log:27362). [FACT]
• During that test the model-download path hit httpx … 429 Too Many Requests for Alibaba-NLP/gte-Qwen2-1.5B-instruct ("you hit the quota of 3000 api requests per 5 minutes"), which raised in scheduler.py:4291 → … → weight_utils.py:568, killed the scheduler, and produced Server process exited with code -9; the test then retried and the embeddings server came up successfully at 18:00:40 and served 200 OK (log:4152, log:4228). [FACT]
• Cancellation printed at 18:24:30, immediately after Begin (1/10): … test_lora_update.py and [CI Test Method] TestLoRADynamicUpdate.test_dynamic_lora_update_server (log:27376). [FACT]

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] This job did not fail on its own merits — it was cancelled by the workflow's fast-fail mechanism after sibling jobs (e.g. stage-a-test-1-gpu-small, multimodal-gen shards) failed earlier in the same run.
  • Supporting: job conclusion is cancelled not failure; only the in-progress Run test step was cancelled while all setup steps succeeded; 10 sibling jobs concluded failure and 23 (including this one) cancelled — the classic fast-fail pattern. The cancel landed exactly at a test boundary (0/10 → 1/10), consistent with the runner finishing the current test before honoring a cancel signal issued earlier.
  • Disconfirming: the ~22 min gap between the first sibling failure (18:00) and this job's cancel (18:24:31) could also be explained by a workflow/job time budget; the two are not mutually exclusive.
  • How to verify: inspect the pr-test-amd-rocm720.yml job strategy for fail-fast/cancel-in-progress concurrency, and confirm the cancel timestamps of the 23 cancelled jobs cluster after the first sibling failure.
• [FACT] A HuggingFace 429 rate limit transiently crashed the embeddings server (exit -9) for the first batched test.
  • Supporting: verbatim 429 Too Many Requests … quota of 3000 api requests per 5 minutes in the scheduler traceback.
  • Disconfirming: this did not fail the test — it retried and passed; it only inflated runtime (1514s vs 900s), which may have widened the cancellation window but did not itself cause a failure.
  • How to verify: not a root cause for the cancellation; track separately as an HF-rate-limit infra flake (shared HF token quota across concurrent AMD jobs).

This job has no test-level regression to bisect. The actionable signals are (1) the fast-fail cascade originating in sibling jobs, and (2) the HF 429 infra flake.

In-flight Fix Check

• HF rate-limit flake: In-flight fix: searched open PRs for huggingface rate limit / 429 —
  • ❌ none found directly addressing the 429 quota in AMD CI (the issue is shared-token quota exhaustion across the 49 concurrent jobs, an infra/secrets concern rather than a code fix).
• The cancellation itself needs no code fix — it is a consequence of the real failures in sibling jobs (stage-a-test-1-gpu-small-amd-rocm720, multimodal-gen-test-1-gpu-amd-rocm720, several stage-b-test-1-gpu-small shards), which should be triaged in their own per-job analyses.

Suggested Triage Steps

• Treat this job as collateral cancellation, not an independent failure — investigate the 10 genuinely-failed sibling jobs in run 28297015673 to find the real regression(s).
• If full signal is needed for this stage-b-large job specifically, re-run with the bypass-fastfail label so it isn't cancelled by sibling failures (note: consumes more CI resources; use sparingly).
• For the HF 429: consider pre-staging Alibaba-NLP/gte-Qwen2-1.5B-instruct into the CI image/local cache or using a higher-quota HF token, so concurrent AMD jobs don't collectively exceed the 3000-req/5-min API quota.
• Confirm whether pr-test-amd-rocm720.yml intends fast-fail; if not, the mass cancellation of 23 jobs from a few failures may be hiding otherwise-green signal.

Status (factual, no priority assignment)

• Persistence: Single occurrence — this job is normally skipped on the schedule; it executed and was cancelled only in run 28297015673.
• Scope: Run-wide orchestration event — 23 of 49 jobs cancelled, 10 failed; this job is one of the cancelled, with no test-level failure of its own.
• Blocked work: test_lora_update.py and the remaining 8 batched tests in this job never ran, so this job provides no LoRA/perf signal for the head SHA.
• In-flight fix: ❌ none found for the HF-429 infra flake; the cancellation requires no code fix (chase the sibling-job failures instead).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:58 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 3/5)

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: N/A — the job's log blob does not exist (cancelled before/without log upload), so no [CI-AITER-CHECK] markers are queryable.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test failure	N/A	Job cancelled (conclusion=cancelled), no steps recorded, log blob BlobNotFound	Job

This job did not fail a test. It was cancelled while in progress at 18:24:17 UTC after ~35 min of runtime.

Failure Summary

The target job multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 0) (job 83838610254) was cancelled, not failed — its log blob returns BlobNotFound and the job recorded zero steps. The entire run 28297015673 ended cancelled (16 success / 10 failure / 23 cancelled), with all 23 cancelled jobs terminating simultaneously at ~18:24:17 UTC — the signature of an external cancel signal, not natural completion or per-test failure.

Regression Status

Infrastructure / orchestration cancellation — not a test regression for this job.

The job class is otherwise healthy: in the two preceding completed scheduled runs the 2-GPU multimodal-gen job was skipped (change-filter gated), and in the immediately superseding run it was again skipped:

Date (UTC)	Run	multimodal-gen-2gpu Status	Note
Jun 27 18:24	28297867681	⏭️ Skipped	superseding run — overall success
Jun 27 17:48	28297015673	🚫 Cancelled (this run)	superseded mid-flight
Jun 27 12:26	28289152816	⏭️ Skipped	overall success
Jun 27 07:03	28281981946	⏭️ Skipped	overall success

Failure Cluster

Concurrency cancel-in-progress supersession (newer scheduled run cancelled the in-progress run).

Facts (verified observations, no interpretation)

• Job 83838610254: status=completed, conclusion=cancelled, started=17:49:17Z, completed=18:24:17Z, 0 steps. Log fetch returns <Error><Code>BlobNotFound</Code></Error>.
• Run 28297015673: event=schedule, conclusion=cancelled; job tally 16 success / 10 failure / 23 cancelled.
• All 23 cancelled jobs completed within a 18-second window (18:24:17–18:24:35Z), consistent with a single cancel signal rather than independent failures.
• A newer scheduled run 28297867681 was created at 18:24:16Z — one second before our jobs were cancelled — and completed success.
• Workflow concurrency config (pr-test-amd-rocm720.yml:117-123): group: …${{ … github.ref }}, cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} → evaluates true for schedule events, so a new scheduled run on the same ref cancels the in-progress one.
• Genuine test failures in this run were in other jobs, not this one (see disconfirming note below).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by GitHub Actions concurrency cancel-in-progress when the newer scheduled run 28297867681 started at 18:24:16Z and claimed the same concurrency group.
  • Supporting: new run created 18:24:16Z; this run's 23 in-progress jobs all cancelled within 1–18 s after that (18:24:17Z+); workflow concurrency cancel-in-progress is true for schedule events (pr-test-amd-rocm720.yml:123); the superseding run finished success.
  • Disconfirming: (none found) — timing and config align exactly.
  • How to verify: already verified via run-creation timestamps + concurrency config; no further action needed.
• [LOW] (alternative, rejected) Fast-fail cascade from the run's 10 genuine job failures.
  • Supporting: 10 jobs failed earlier (18:00–18:21Z) in the same run.
  • Disconfirming: the precise simultaneity with the new run's 18:24:16Z creation, and the schedule-triggered cancel-in-progress=true, point to concurrency supersession; the genuine failures had already concluded minutes earlier and would not produce a single synchronized cancel wave.

In-flight Fix Check

Not applicable — no code/test failure to fix in this job; the cancellation is expected GitHub Actions concurrency behavior. (For context, the sibling multimodal-gen failures below may warrant their own tracking, but they are outside this job.)

Suggested Triage Steps

• No action for this job — a cancelled job from concurrency supersession is not a CI defect. The superseding run 28297867681 succeeded and is the authoritative signal for this schedule slot.
• If repeated supersession of scheduled runs is undesirable (two schedule runs 36 min apart: 17:48 and 18:24), inspect what triggered the extra 18:24 scheduled run and whether the cron/auto-rerun cadence should be de-duplicated.
• Separately (sibling jobs, not this one), investigate the real multimodal-gen failures observed in the same run:
  • multimodal-gen-1gpu shard 2: test/single_test_file/test_generate_zimage_turbo_cli.py::TestZImageTurboCLI::{test_output_file_path_alias,test_single_gpu} — model Tongyi-MAI/Z-Image-Turbo download incomplete (missing model_index.json/transformer//vae/); looks like a model-availability/cache issue.
  • multimodal-gen-1gpu shard 3: test/single_test_file/test_update_weights_from_disk.py — 5 failed with AssertionError: Update failed: … Inplace update to inference tensor outside InferenceMode is not allowed (a real code-level error), plus a FLUX.2-klein-base-4B HF repo-id path error.

Status (factual, no priority assignment)

• Persistence: One-off cancellation in run 28297015673; not recurring for this job (prior and next scheduled runs skipped this job, overall success).
• Scope: 23 jobs in this single run were cancelled by the same concurrency supersession; this target job is one of them.
• Blocked work: None — the superseding scheduled run 28297867681 provided a clean success signal for the same slot.
• In-flight fix: N/A — no defect in this job.

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: N/A — target job log unavailable (cancelled job, blob expired); no [CI-AITER-CHECK] markers recoverable.

Failed Tests

Test File	Test Function	Error	Log
N/A — no test failed in this job	—	Job was cancelled (conclusion: cancelled), not failed. It produced zero test results.	Job

Failure Summary

The target job stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) did not fail — it was cancelled at 2026-06-27T18:24:17Z, the exact same instant 22 other in-progress jobs in the run were cancelled. This is a fast-fail cascade: sibling jobs began failing at 18:00:25Z, the run's fast-fail threshold was hit, and all remaining in-flight jobs (including this one) were force-cancelled mid-execution. The job's log blob returns BlobNotFound because cancelled jobs don't persist step logs.

Regression Status

Infrastructure issue (fast-fail cascade) — not a test regression of this job.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed runs only):

Date	Run	Conclusion
Jun 27 18:24	28297867681	✅ success (this job: skipped in the rerun)
Jun 27 17:48	28297015673	❌ cancelled (this run)
Jun 27 12:26	28289152816	✅ success
Jun 27 07:03	28281981946	✅ success
Jun 27 00:37	28273097445	✅ success

The immediately following scheduled run (28297867681, same head SHA, started the same minute) succeeded, confirming the trigger was transient infra, not a code defect.

Failure Cluster

HuggingFace Hub 429 Too Many Requests on model-metadata fetch → fast-fail cancellation cascade

Facts (verified observations)

• This job (83838610255): status: completed, conclusion: cancelled; started 17:49:17Z, cancelled 18:24:17Z. No steps recorded. Log = BlobNotFound.
• Coordinated cancel: 23 jobs all share completed_at = 2026-06-27T18:24:17Z — a single fast-fail cancel event, not independent failures.
• Run rollup: 16 success · 10 failure · 23 cancelled (49 jobs total).
• Triggering failures (the real failure jobs that fired the cascade), dominant signature is HF Hub 429:
  • stage-a-test-1-gpu-small (83838610220): httpx.HTTPStatusError: 429 Too Many Requests for meta-llama/Llama-3.1-8B-Instruct → test_basic_sanity.py + test_basic_sanity_eagle3.py exit 1 (first to fail, 18:00:46Z).
  • stage-b-1gpu-small (7) (83838610285): 429 for Qwen/Qwen2.5-VL-3B-Instruct; server exited -9; test_vlm_perf_5090.py exit 1.
  • stage-b-1gpu-small (6) (83838610271): 429 for RedHatAI/Meta-Llama-3.1-8B-FP8; test_compressed_tensors_models.py exit 1; also test_type_based_dispatcher.py TypeError: Missing required argument 'input_embeds'.
  • stage-b-1gpu-small (10) (83838610294): server exit -9 (test_srt_engine.py) + downstream ConnectionRefused on port 11000 (test_start_profile.py), test_llada2_mini.py, test_json_mode.py.
  • stage-b-1gpu-small-mi35x (83838610236): server hang, repeated py-spy dump ... Failed to get stack traces.

Hypothesised Causes (with confidence)

• [HIGH] This job's cancellation is a fast-fail cascade, not a defect in the 2-gpu-large suite.
  • Supporting: cancelled at the same millisecond as 22 other jobs; sibling failures preceded it by ~24 min; the rerun on the identical SHA succeeded.
  • Disconfirming: (none found) — no test ever executed in this job to fail on its own.
  • How to verify: re-run only this job (already effectively done — the follow-up run passed).
• [HIGH] The cascade trigger is HuggingFace Hub 429 rate-limiting during model metadata/download, hitting multiple unrelated suites simultaneously.
  • Supporting: identical 429 Too Many Requests errors across stage-a and multiple stage-b jobs for 3 different models; server -9 exits are downstream of failed model fetches.
  • Disconfirming: a few secondary errors (test_type_based_dispatcher.py TypeError, py-spy hang on mi35x) are not obviously HF-429-related and may be independent flakes worth separate triage.
  • How to verify: grep the triggering jobs for huggingface.co + 429 (done); confirm no source commit in the window touches model loading.

In-flight Fix Check

• In-flight fix: ✅ #23400 "ci: prefetch HF snapshots and skip redundant metadata calls to survive 429s" (open since 2026-04-21) — directly targets this HF-429 failure mode. Chase reviewers; do not open a duplicate.

Suggested Triage Steps

• No action needed for this specific job — it was an innocent fast-fail casualty and the next run on the same SHA passed.
• For the real trigger: land in-flight #23400 to make CI resilient to HF 429s (prefetch/cache snapshots, drop redundant metadata calls).
• Separately triage the two non-429 signatures in case they recur independently of rate-limiting: test_type_based_dispatcher.py TypeError: Missing required argument 'input_embeds' (83838610271) and the mi35x py-spy "Failed to get stack traces" hang (83838610236).

Status (factual)

• Persistence: One-off. This job was cancelled in run 28297015673 only; the 4 prior scheduled runs and the immediately following run on the same SHA were green.
• Scope: 23 jobs cancelled by one fast-fail event; root trigger (HF 429) hit ≥5 jobs across stage-a, stage-b-small, and multimodal-gen.
• Blocked work: This run's full ROCm 7.2 signal is incomplete (large/2-gpu/8-gpu stages never ran), but the next scheduled run recovered it.
• In-flight fix: ✅ #23400 (open since 2026-04-21) — awaiting merge.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (source: dev image — [CI-AITER-CHECK] reports v0.1.14rc1.dev240+g7d604afe5, "Dev/patched version detected → skipping rebuild"; matches Dockerfile default)

Failed Tests

All cases are test/server/test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[<param>]. 12 selected; 2 passed, 6 failed, 3 errored, 1 cancelled.

Test Param	Result	Error	Log
wan2_2_t2v_a14b_2gpu	FAILED	torch._inductor.exc.InductorError: AssertionError (pattern_matcher.py:314)	Log
wan2_1_t2v_14b_2gpu	FAILED	InductorError: AssertionError (pattern_matcher.py:314)	Log
wan2_1_t2v_1.3b_cfg_parallel	FAILED	InductorError: AssertionError (pattern_matcher.py:314)	Log
mova_360p_tp2	FAILED	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
wan2_1_i2v_14b_480P_2gpu	FAILED	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
wan2_1_i2v_14b_720P_2gpu	FAILED	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
flux2_modelopt_fp8_tp2_t2i	ERROR	ValueError: Could not find model … still incomplete after forced re-download	Log
ideogram4_fp8_tp2_t2i	ERROR	ValueError: … incomplete after forced re-download	Log
wan2_2_i2v_a14b_2gpu	ERROR	ValueError: … incomplete after forced re-download	Log
ltx_2.3_one_stage_ti2v	CANCELLED	Job preempted mid-startup (concurrency cancel)	Log
fsdp-inference	✅ PASSED	—	—
zimage_image_t2i_2_gpus_non_square	✅ PASSED	—	—

Failure Summary

The job 83838610250 ended with conclusion cancelled — the entire scheduled run 28297015673 (17:48 UTC) was preempted by a newer scheduled run 28297867681 with the same head SHA (created 18:24:16 UTC; "The operation was canceled" appeared at 18:24:30). Before the cancel, three independent failure clusters occurred: (A) a torch.compile/Inductor assertion on T2V Wan models, (B) incomplete HF model downloads on 3 models, and (C) a ROCm-unsupported causal Conv3D path on the I2V/mova models.

Regression Status

Indeterminate from CI history — the multimodal-gen-test-2-gpu-amd-rocm720 job was skipped in every other recent scheduled run of this workflow (28297867681, 28289152816, 28281981946, 28273097445); these diffusion cases ran only in this preempted run, so no green/red baseline for these test params exists in the queryable window (last ~8 scheduled runs, branch=main, event=schedule).

Date	Run	Job (2-gpu multimodal)	Status
Jun 27 18:24	28297867681	job	⏭️ Skipped (did not run)
Jun 27 17:48	28297015673	this job	🚫 Cancelled (6 fail / 3 error / 2 pass before preempt)
Jun 27 12:26	28289152816	job	⏭️ Skipped
Jun 27 07:03	28281981946	job	⏭️ Skipped

Failure Cluster

Three clusters in one job:

• Cluster A — "InductorError: AssertionError in post_grad decompose_triton_kernel_wrapper_functional" (T2V Wan models)
• Cluster B — "Incomplete HF model download (model_index.json/transformer/vae missing)" (infra)
• Cluster C — "causal Conv3D cat/pad fusion unsupported on ROCm" (I2V/mova models)

Facts (verified observations)

• Job conclusion = cancelled; run conclusion = cancelled; a newer same-SHA scheduled run 28297867681 was created 14s before the cancel line → concurrency preemption, not a step timeout (FACT).
• Cluster A verbatim top frames: layers/layernorm.py:534 forward_hip → forward_native (torch.compiled) → _inductor/compile_fx.py → _recursive_post_grad_passes → fx_passes/post_grad.py:1217 decompose_triton_kernel_wrapper_functional → pattern_matcher.py:314 replace_by_example → assert len(graph_with_eager_vals.graph.nodes) == len(...) → InductorError: AssertionError. Surfaces as RuntimeError: Model generation returned no output. Affects wan2_2_t2v_a14b_2gpu, wan2_1_t2v_14b_2gpu, wan2_1_t2v_1.3b_cfg_parallel (FACT).
• Cluster B: ValueError: Could not find model at <repo> and failed to download from HF Hub: Downloaded model at /sgl-data/hf-cache/hub/… is still incomplete after forced re-download for black-forest-labs/FLUX.2-dev, ideogram-ai/ideogram-4-fp8, Wan-AI/Wan2.2-I2V-A14B-Diffusers; each followed by Rank 0 scheduler is dead / Exit code: 1 (FACT).
• Cluster C: RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA, raised in [ImageVAEEncodingStage] at python/sglang/multimodal_gen/runtime/layers/parallel_conv.py:57. Affects mova_360p_tp2, wan2_1_i2v_14b_480P_2gpu, wan2_1_i2v_14b_720P_2gpu (FACT).
• In parallel_conv.py, the Triton fallback (triton.causal_conv3d_pad) is imported only under if current_platform.is_cuda(): — both before and after PR #29281; on ROCm fused_causal_conv3d_cat_pad_triton is None, so the function raises (FACT, verified via git show 5996b54b).

Hypothesised Causes (with confidence)

Cluster A — Inductor assertion (T2V Wan):

• [LOW] torch.compile/Inductor internal assertion in decompose_triton_kernel_wrapper_functional when compiling the ROCm forward_native layernorm path.
  • Supporting: identical stack top across all 3 T2V failures; the assertion is in upstream torch inductor, reached via sglang's compiled layernorm.forward_native.
  • Disconfirming: cannot tell if new — no CI baseline (job skipped in all other recent scheduled runs); could be a torch/inductor-version interaction on ROCm rather than an sglang regression.
  • How to verify: re-run test_server_2_gpu.py -k wan2_1_t2v_1.3b_cfg_parallel on MI325 ROCm 7.2 with TORCHDYNAMO_VERBOSE=1; A/B with torch.compile disabled to confirm the compile path is the trigger.

Cluster C — causal Conv3D unsupported on ROCm:

• [LOW] PR #29281 (5996b54b, Jun 26) reshaped parallel_conv.py and added the explicit raise RuntimeError("…only available on CUDA").
  • Supporting: it last modified the exact failing line (parallel_conv.py:57), one day before this run.
  • Disconfirming: the Triton fallback was CUDA-gated before #29281 too (fused_causal_conv3d_cat_pad = None on non-CUDA in the prior code), so the I2V/mova Conv3D path almost certainly never worked on ROCm — #29281 likely just converted a prior failure into a clearer error message rather than introducing the gap. Strong candidate for a pre-existing ROCm limitation / Never-passed case.
  • How to verify: check whether these I2V/mova params have ever passed on any ROCm runner; if never, treat as an unsupported-feature gap, not a regression.

Cluster B — incomplete model download:

• [FACT] Infrastructure: HF cache snapshots for 3 models are incomplete (missing model_index.json/transformer//vae/) even after forced re-download. Not a code defect.
  • How to verify: inspect the runner's /sgl-data/hf-cache/hub/models--…/snapshots/ for the 3 repos; re-prime the cache.

In-flight Fix Check

• Cluster C (causal+Conv3D+cat+pad): ❌ none directly relevant — search returned #28065, #15068 (unrelated VAE/world-opt work, not a ROCm Conv3D fallback).
• Cluster A (decompose_triton_kernel_wrapper_functional): ❌ none found (0 open PRs).
• Cluster B: ❌ none — infra/cache issue, not a PR target.

Suggested Triage Steps

• Cluster B (likely highest ROI): re-prime/repair the HF cache for black-forest-labs/FLUX.2-dev, ideogram-ai/ideogram-4-fp8, Wan-AI/Wan2.2-I2V-A14B-Diffusers on the MI325 runner, then re-run the job. These 3 errors are environmental.
• Cluster C: confirm whether I2V/mova causal-Conv3D is expected to be supported on ROCm at all. If a Triton fallback is intended, it must be importable on non-CUDA (currently gated behind is_cuda() in parallel_conv.py). Until then, consider skipping these params on AMD. Ping the author of #29281 for ROCm intent.
• Cluster A: reproduce a single T2V case on MI325 ROCm 7.2 with TORCHDYNAMO_VERBOSE=1 / TORCH_LOGS="+dynamo" to get the full inductor stack; A/B with torch.compile disabled on layernorm.forward_native to isolate whether the compiled path is the trigger.
• Process note: because this run was concurrency-cancelled and the job is skipped in other scheduled runs, trigger a dedicated re-run of just multimodal-gen-test-2-gpu-amd-rocm720 to obtain a clean, uncancelled signal and establish a baseline.

Status (factual)

• Persistence: Cannot establish — these diffusion params ran only in this single (cancelled) scheduled run; the job was skipped in all other recent scheduled runs in the window.
• Scope: 1 job, 3 distinct clusters: Cluster A = 3 T2V params, Cluster B = 3 download errors, Cluster C = 3 I2V/mova params; 2 params passed (fsdp-inference, zimage_image_t2i_2_gpus_non_square); 1 cancelled (ltx_2.3_one_stage_ti2v).
• Job outcome: conclusion cancelled (run preempted by newer same-SHA scheduled run 28297867681 via concurrency).
• In-flight fix: ❌ none found for any cluster.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe (v0.1.14rc1.dev240+g7d604afe5, source: dev/patched image preserved — AITER_COMMIT_OVERRIDE empty, so this is a regular schedule run, not an AITER Scout run)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[cosmos3_nano_t2i]	None — job cancelled mid model-download (no test assertion error)	Log

No test in this job failed. The two tests that completed before cancellation both PASSED (qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu, zimage_image_t2i_multi_lora). The job was terminated by ##[error]The operation was canceled. while the third test was still fetching the nvidia/Cosmos3-Nano snapshot (stalled at 81% of 67 files from ~18:10 until the 18:24:30 cancel).

Failure Summary

This job's conclusion is cancelled, not failure. Step 9 ("Run diffusion server tests") was cancelled at 2026-06-27 18:24:30 UTC. The job had already passed 2 of its 13 selected tests and was downloading the model for the 3rd test when every cancelled job in the run terminated simultaneously (~18:24:30). The cancellation is run-level, externally triggered, not a defect in the test or the code under test.

Regression Status

Infrastructure / cancellation — not a code regression.

The cause is a concurrency cancellation: a newer scheduled run of the same workflow on the same commit superseded this one and then passed.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date (UTC)	Run	Head SHA	Conclusion
06-27 18:24	28297867681	592f6c849b	✅ success
06-27 17:48	28297015673 (this run)	592f6c849b	⛔ cancelled
06-27 12:26	28289152816	3306233961	✅ success
06-27 07:03	28281981946	9214b9338f	✅ success
06-27 00:37	28273097445	13b5bd962a	✅ success

The superseding run 28297867681 was created at 18:24:16 on the identical SHA 592f6c849b; all 23 cancelled jobs in this run ended ~14 s later at 18:24:30. The superseding run succeeded, confirming the cosmos3_nano_t2i test path is not broken at this commit.

Failure Cluster

Run-level concurrency cancellation (newer same-SHA scheduled run superseded this one)

Facts (verified observations, no interpretation)

• Job conclusion = cancelled; step 9 conclusion = cancelled (GitHub API job 83838610258). Started 17:49:53, completed 18:24:32.
• ##[error]The operation was canceled. at log line 5258, timestamp 2026-06-27T18:24:30 (log).
• Tests qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu and zimage_image_t2i_multi_lora printed PASSED (log lines 4431, 5206) before cancellation.
• At cancel time, Fetching 67 files: 81% for nvidia/Cosmos3-Nano, last progress at 18:10:02, no further output for ~14 min (log lines 5250-5258).
• Run-wide: 49 jobs → 16 success, 10 failure, 23 cancelled. All cancelled jobs ended ~18:24:30/31/32 (synchronized → external cancel, not 23 independent timeouts).
• A second schedule run 28297867681 exists on the same SHA 592f6c849b, created 18:24:16, conclusion success.
• Sibling shard 3 (job 83838610263) failed before the cancel on a separate HF-Hub model-download infra issue: ValueError: Downloaded model at .../Tongyi-MAI--Z-Image-Turbo/... is still incomplete after forced re-download (shard3 log lines 4301-4331). This is download/infra flakiness, distinct from the concurrency cancel, and it cleared in the passing rerun.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] This job was killed by workflow concurrency cancellation when a newer scheduled run on the same commit started.
  • Supporting: superseding run 28297867681 created 18:24:16 (same SHA 592f6c849b); all 23 cancelled jobs terminated within seconds at 18:24:30; that newer run concluded success. Same back-to-back cancel pattern appears on 06-26 (runs 28256293905, 28257868547 both cancelled).
  • Disconfirming: I did not read the workflow YAML concurrency: block to confirm cancel-in-progress: true; the cancel could in principle be manual. (Low weight — the timing + same-SHA superseding-then-passing run is hard to explain otherwise.)
  • How to verify: inspect concurrency: in .github/workflows/pr-test-amd-rocm720.yml; confirm run 28297867681's triggering_actor/event and that it superseded 28297015673.
• [FACT] The cosmos3_nano_t2i test itself did not fail — there is no assertion error or stack trace; the job was simply interrupted during model download. The same test path passed in the superseding run (run-level success).

In-flight Fix Check

• In-flight fix: ❌ none needed — this is a cancellation, not a code/test defect; the same commit passes on rerun.
• Note: an open branch fix/rerun-failed-ci-cancelled-workflows (seen in fetch output, log line 753) suggests maintainers are already addressing rerun handling for cancelled workflows.
• Searched open PRs for the HF-download flake (sibling shard 3): consider keyword search is:pr is:open "incomplete after forced re-download" if that infra flake recurs — it is a separate issue from this cancellation.

Suggested Triage Steps

• Treat this job result as non-actionable noise for code health: the same SHA 592f6c849b passed in run 28297867681. No bisect warranted.
• If reducing cancelled-run noise is desired: confirm the concurrency group in pr-test-amd-rocm720.yml and whether overlapping cron schedules (17:48 + 18:24 on the same SHA) are intended — back-to-back scheduled runs are cancelling each other (also seen 06-26).
• Separately track the HF-Hub model-download flakiness (Z-Image-Turbo/Cosmos3-Nano snapshots incomplete/stalled) that hit sibling shards — pre-seeding these models into the runner cache would remove both the shard-3 failure and the shard-1 download stall.

Status (factual, no priority assignment)

• Persistence: Single occurrence for this job at this SHA; the immediately-following run on the same SHA passed. Cancelled scheduled runs also seen 06-26 (2 runs), suggesting a recurring concurrency/scheduling artifact rather than a code regression.
• Scope: 23 of 49 jobs in run 28297015673 cancelled simultaneously; this job is one of them. Code-under-test unaffected (verified by passing rerun).
• Blocked work: None — superseding run provides green signal for commit 592f6c849b.
• In-flight fix: ❌ none needed (cancellation, not a defect); maintainer branch fix/rerun-failed-ci-cancelled-workflows may already cover rerun handling.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: N/A — job was cancelled before any runner/container started; no [CI-AITER-CHECK] markers were emitted (no log produced).

Failed Tests

None. This job did not fail — it was cancelled before execution. It never acquired an 8-GPU runner (runner_name empty, steps: []), and its log blob returns BlobNotFound. There is no test-level failure to attribute to this job.

Failure Summary

The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 0) was queued at 2026-06-27T17:49:17Z but never scheduled onto a scarce linux-mi325-8gpu-sglang runner. At 2026-06-27T18:24:17Z it was cancelled — not by a test failure, but because the entire run 28297015673 was cancelled by GitHub Actions concurrency control when the next scheduled run started.

Regression Status

Infrastructure issue (run cancelled, job never executed). Not a test regression.

History context for the run, not the test (same workflow, event=schedule, branch=main, completed runs only):

Created (UTC)	Run	Conclusion	Note
2026-06-28 00:37	28306564746	queued	in-flight
2026-06-27 18:24	28297867681	✅ success	the run that cancelled ours
2026-06-27 17:48	28297015673	❌ cancelled	this run
2026-06-27 12:26	28289152816	✅ success	—
2026-06-26 18:35	28257868547	❌ cancelled	same close-spacing cancel pattern
2026-06-26 18:04	28256293905	❌ cancelled	same close-spacing cancel pattern

Failure Cluster

Scheduled run cancelled by concurrency cancel-in-progress (8-GPU job never scheduled).

Facts (verified observations)

• Job state (API): status=completed, conclusion=cancelled, started_at=2026-06-27T17:49:17Z, completed_at=2026-06-27T18:24:17Z, runner_name="", steps=[]. The job acquired no runner and ran no steps.
• Log: GET /actions/jobs/83838610260/logs → <Error><Code>BlobNotFound</Code> (no log was ever produced — consistent with a never-started job).
• Run state (API): run 28297015673 event=schedule, conclusion=cancelled, created 17:48:59Z, updated 19:06:21Z.
• Run job breakdown: 49 jobs — 16 success, 10 failure, 23 cancelled. All 7 stage-c-* (4-GPU and 8-GPU) jobs are in the cancelled set; none ran.
• Concurrency trigger: next scheduled run 28297867681 created_at=2026-06-27T18:24:16Z — within 1s of our run's cancellation (18:24:17Z).
• Concurrency config (.github/workflows/pr-test-amd-rocm720.yml:117-123): for event=schedule, group resolves to pr-test-amd-rocm720-refs/heads/main (shared by all schedule runs on main) and cancel-in-progress evaluates true. A newer schedule run therefore cancels any in-progress one.
• Job dependency (pr-test-amd-rocm720.yml:928-929): stage-c-test-large-8-gpu-amd-rocm720 needs: [check-changes] only — it is NOT downstream of the 10 stage-a/stage-b failures, so this is not a fast-fail cascade. The cancellation is purely concurrency-driven.

Hypothesised Causes

• [FACT] The run (and thus this queued job) was cancelled by GitHub Actions concurrency cancel-in-progress, triggered by the newer scheduled run 28297867681 starting at 18:24:16Z in the same group pr-test-amd-rocm720-refs/heads/main.
  • Supporting: newer run created at 18:24:16Z == this run's cancel time 18:24:17Z; concurrency block at pr-test-amd-rocm720.yml:117-123 enables cancel-in-progress for schedule; same cancel pattern seen on 06-26 runs spaced <30 min apart.
  • Disconfirming: (none found) — timing and config are unambiguous.
  • How to verify: compare created_at of consecutive schedule runs vs the completed_at of the cancelled run's jobs (already done above).
• [MEDIUM] Contributing factor: linux-mi325-8gpu-sglang runner scarcity. The job sat queued ~35 min without acquiring a runner; had a runner been available it might have started before the cancel window.
  • Supporting: runner_name="" and steps=[] after 35 min queued; all six 8-GPU stage-c shards were equally unscheduled.
  • Disconfirming: even if scheduled, the run would still have been cancelled at 18:24 mid-execution; runner scarcity only widens the exposure window, it is not the cancel trigger.
  • How to verify: inspect 8-GPU runner queue depth / availability metrics during 17:49–18:24Z.

In-flight Fix Check

• In-flight fix: ❌ none found — this is a run-orchestration/scheduling artifact, not a code defect, so an open code-fix PR is not expected. Search run: search/issues?q=repo:sgl-project/sglang+is:pr+is:open+concurrency+schedule+cancel returned no relevant CI-config PR.

Suggested Triage Steps

• No code triage required — no test failed. Treat this job as non-signal (cancelled, never executed).
• If recurrence is undesirable: investigate why two schedule runs fired ~36 min apart (17:48 and 18:24). Possible overlapping cron entries (e.g. a daily cron coinciding with the 0 */6 * * * 6h cron) cause back-to-back runs where the later cancels the earlier. Review the on.schedule cron list in pr-test-amd-rocm720.yml.
• Consider whether long-queued 8-GPU stage-c shards should be exempt from cancel-in-progress, or whether scheduled runs should use a queue (cancel-in-progress: false) so a long-running 8-GPU stage isn't discarded after 35 min of queue time.
• To get stage-c signal for this SHA, re-run stage-c on the next successful schedule run or dispatch pr-test-amd-rocm720.yml with target_stage=stage-c-test-large-8-gpu-amd-rocm720.

Status

• Persistence: Single-run cancellation. Same concurrency-cancel pattern also hit the 06-26 18:04 and 18:35 schedule runs, so the cancel-on-overlap behavior recurs whenever two schedule runs start close together.
• Scope: All 23 cancelled jobs in run 28297015673, including all 7 stage-c-* shards. The 10 genuine failures in stage-a/stage-b are separate and unrelated to this job's cancellation.
• Blocked work: stage-c (large 8-GPU) coverage produced no signal for sglang 592f6c84 in this run; covered by the subsequent successful run 28297867681.
• In-flight fix: ❌ none found — not a code defect.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: N/A — the job log blob does not exist (BlobNotFound), so no [CI-AITER-CHECK] markers are recoverable.

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failure — job was cancelled before/while running	Job (log unavailable: BlobNotFound)

This job did not fail a test. Its conclusion is cancelled, it recorded zero steps, and its log blob returns BlobNotFound. There is no failing test file or function to report.

Failure Summary

The job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 3) was cancelled at 2026-06-27T18:24:17Z after running ~35 min, not failed. The parent run 28297015673 (a schedule run, conclusion cancelled) was superseded by a newer schedule run on the same ref, which triggered the workflow's cancel-in-progress concurrency policy and terminated all still-running jobs (the long 8-GPU stage-c shards among them).

Regression Status

Infrastructure / CI-config cancellation — not a code regression.

[FACT] A newer schedule run 28297867681 was created at 2026-06-27T18:24:16Z — the same second this job was cancelled (18:24:17Z) — with the identical head_sha 592f6c849b and branch=main.
[FACT] The workflow's concurrency group for schedule events is keyed on github.ref with cancel-in-progress: true (pr-test-amd-rocm720.yml:122-123). Two schedule runs on refs/heads/main therefore share one group; the later one cancels the earlier in-progress one.
[FACT] In the superseding run, stage-c-test-large-8-gpu-amd-rocm720 is skipped (job 83843617192). Per the call-gate comment in the workflow, the 6-hourly 0 */6 * * * cron skips check-changes and every test job (runs only the two dsv4 jobs), which is why that run reports success — it ran no stage-c tests.

History of full-test schedule runs (the daily 30 17 * * * cron is the only schedule that runs stage-c; same branch, event=schedule, completed only):

Date (UTC)	Run	Type	Stage-c shard 3	Note
Jun 27 17:48	28297015673	daily full	⏹ Cancelled	superseded by 18:24 6h cron
Jun 27 18:24	28297867681	6h cron	⏭ Skipped	dsv4-only, no stage-c
Jun 27 12:26	28289152816	6h cron	⏭ Skipped	dsv4-only

The cancellation pattern recurs on prior days (e.g. 28257868547, 28256293905 on Jun 26 both cancelled at ~18:xx), consistent with the daily 17:30 full run being repeatedly cancelled by the 18:00 6-hourly cron sharing the same github.ref concurrency group.

Failure Cluster

Schedule-run cancelled by concurrency cancel-in-progress supersession (CI-config, not a test failure).

Facts (verified observations, no interpretation)

• Job 83838610261: status=completed, conclusion=cancelled, started=2026-06-27T17:49:17Z, completed=2026-06-27T18:24:17Z, steps array empty.
• Job log API returns <Error><Code>BlobNotFound</Code> — no log was persisted (consistent with a cancelled job that never finished its steps).
• Parent run 28297015673: event=schedule, conclusion=cancelled. Of 49 jobs: 16 success, 10 failure, 23 cancelled.
• Superseding run 28297867681: created 18:24:16Z, same head_sha 592f6c849b, conclusion=success, stage-c job skipped.
• Workflow concurrency: group: ...|| github.ref, cancel-in-progress: !run_all_tests && event!=workflow_call && event!=workflow_dispatch ⇒ true for schedule (pr-test-amd-rocm720.yml:117-123).
• Real (unrelated) test failures DID occur in sibling jobs of this run — e.g. stage-a-test-1-gpu-small (83838610220), several stage-b-test-1-gpu-small shards, and multimodal-gen-test-1-gpu shards — but these are different jobs and are out of scope for this stage-c shard-3 analysis.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] The job was cancelled by the workflow's cancel-in-progress concurrency policy when the next schedule run on refs/heads/main started.
  • Supporting: cancellation timestamp (18:24:17Z) coincides to the second with the superseding run's creation (18:24:16Z); same ref + same sha; concurrency config grants cancel-in-progress=true for schedule events; cancelled job has zero steps and no log.
  • Disconfirming: (none found) — there is no evidence of any test executing to failure in this job.
  • How to verify: re-run the daily full schedule run in isolation (or temporarily set cancel-in-progress: false for schedule on a branch) and confirm stage-c shard 3 completes.

In-flight Fix Check

This is a CI-orchestration/concurrency issue, not a code bug, so a test-name keyword search does not apply. Searched for an existing workflow-config fix:

• In-flight fix: ❌ none found — curl ".../search/issues?q=repo:sgl-project/sglang+is:pr+is:open+pr-test-amd-rocm720+concurrency" returned no matching open PR. A maintainer should decide whether the daily 30 17 full run and the 0 */6 6h cron should share a concurrency group.

Suggested Triage Steps

• Confirm whether the daily full schedule run (30 17 * * *) is meant to share a concurrency group with the 6-hourly dsv4-only cron (0 */6 * * *). If not, give the full run a distinct group key (e.g. include the cron/event so the 18:00 6h cron cannot cancel the in-flight 17:30 full run).
• If the daily full run must survive concurrent schedule triggers, consider gating cancel-in-progress so it does not apply to the long full-suite schedule run, or stagger the cron times so the 8-GPU stage-c shards (~35 min+) finish before the next 6h cron fires.
• This job itself needs no test triage. To get real signal for stage-c shard 3, look at the next daily full run that completes without being superseded, or trigger a workflow_dispatch run (which gets a unique concurrency group and won't be cancelled).
• Separately (out of scope here): the genuine failure jobs in this run — stage-a-test-1-gpu-small, stage-b-test-1-gpu-small shards, multimodal-gen-test-1-gpu shards — warrant their own per-job analyses.

Status (factual, no priority assignment)

• Persistence: The daily full schedule run has been cancelled at ~18:xx on multiple recent days (Jun 25–27), consistent with recurring concurrency supersession by the 18:00 6h cron.
• Scope: Affects the long-running stage-c (8-GPU) and other in-flight jobs of the daily full schedule run that are still executing when the next schedule run starts; 23 of 49 jobs in this run were cancelled.
• Blocked work: The daily ROCm 7.2 full-suite signal for stage-c (8-GPU) is lost whenever the run is superseded — no pass/fail data is produced for these shards.
• In-flight fix: ❌ none found — needs a maintainer decision on the schedule concurrency-group configuration.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:58 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 4/5)

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 2) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (source: dev image preserved — v0.1.14rc1.dev240+g7d604afe5, no override; [CI-AITER-CHECK] Dev/patched version detected → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_generate_zimage_turbo_cli.py	TestZImageTurboCLI::test_output_file_path_alias	AssertionError: 'Failed' != 'Success' : output-file-path command failed	Log
python/sglang/multimodal_gen/test/single_test_file/test_generate_zimage_turbo_cli.py	TestZImageTurboCLI::test_single_gpu	AssertionError: 'Failed' != 'Success' : Z-Image-Turbo_single_gpu command failed	Log

Failure Summary

Both tests in test_generate_zimage_turbo_cli.py failed at step 9 ("Run diffusion server tests (1-GPU)"). Each invokes sglang generate against Tongyi-MAI/Z-Image-Turbo; the CLI subprocess exited with code 1 because the model could not be loaded, so the test assertions on status == "Success" failed. Final result: 2 failed, 5 warnings in 85.30s.

Regression Status

Infrastructure issue (model artifact unavailable) — no clean baseline in queryable history.

The underlying CLI failure is a model-download error, not a code assertion mismatch in product logic. The multimodal-gen-test-1-gpu-amd-rocm720 job is path-filtered and skipped in nearly all scheduled runs; in the rest of the scheduled history it was cancelled (run-level cancellations), so there is no completed passing run of this exact test file to compare against.

Recent history of test_generate_zimage_turbo_cli.py in job multimodal-gen-test-1-gpu-amd-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job Status	Test File Status	Notes
Jun 27 17:48	28297015673	❌ Failed	❌ Failed	this run (run itself later cancelled; sibling partition 83838610262 also failed, same symptom)
Jun 27 18:24	28297867681	—	⏭️ Skipped	job skipped (path filter), same head SHA
Jun 27 12:26	28289152816	—	⏭️ Skipped	job skipped
Jun 27 07:03	28281981946	—	⏭️ Skipped	job skipped
Jun 26–27	28256293905 and earlier	❌ Cancelled	—	job cancelled at run level — not comparable

Lookback range searched: last 30 scheduled runs (Jun 25–28). No completed passing run of this test file found.

Failure Cluster

Z-Image-Turbo model download incomplete (diffusion CLI generate fails to load model)

Facts (verified observations, no interpretation)

• Verbatim CLI error (log line 4331/4418): ValueError: Could not find model at Tongyi-MAI/Z-Image-Turbo and failed to download from HF Hub: Downloaded model at /sgl-data/hf-cache/hub/models--Tongyi-MAI--Z-Image-Turbo/snapshots/f332072aa78be7aecdf3ee76d5c247082da564a6 is still incomplete after forced re-download. The model repository may be missing required components (model_index.json, transformer/, or vae/).
• Inner exception raised at python/sglang/multimodal_gen/runtime/utils/hf_diffusers_utils.py:833 (maybe_download_model), re-raised at :874; reached via composed_pipeline_base.py:175 _load_config → :345 load_modules.
• The CLI subprocess prints Rank 0 scheduler is dead then Exit code: 1; the pytest assertions (cli_generate_common.py:98 verify, test_generate_zimage_turbo_cli.py:32) then fail on status == "Success". The assertion is a downstream symptom of the subprocess exit, not a numerical/logic mismatch.
• Both failing tests fail with the identical root error. Sibling matrix partition 83838610262 in the same run also failed; partitions …258 and …266 were cancelled.
• aiter version unchanged (no override; dev image preserved) — [CI-AITER-CHECK] markers at log lines 4087–4093.
• Build step emitted only HIP compiler warnings (-Wpass-failed, "loop not unrolled", occupancy targets) — non-fatal, unrelated.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Model artifact Tongyi-MAI/Z-Image-Turbo is missing/incomplete in the runner's HF cache (model_index.json, transformer/, or vae/ absent), so the diffusion pipeline cannot build and the CLI exits 1.
  • Supporting: explicit ValueError naming the incomplete snapshot path and the missing component list; failure occurs at model-load time before any inference; identical across both tests and the sibling partition.
  • Disconfirming: (none found) — the error is self-describing and reproducible within the run; not a product code path.
  • How to verify: on the AMD MI325 runner, inspect /sgl-data/hf-cache/hub/models--Tongyi-MAI--Z-Image-Turbo/snapshots/f332072aa78be7aecdf3ee76d5c247082da564a6 for model_index.json + transformer/ + vae/; re-pull the model and re-run test_generate_zimage_turbo_cli.py.
• [LOW] Transient HF Hub network/availability failure during the forced re-download (rather than a permanently broken cache).
  • Supporting: the message says "still incomplete after forced re-download", consistent with a partial/interrupted fetch; earlier in the log a local docker registry pull also timed out (line 1353), hinting at runner network flakiness.
  • Disconfirming: a forced re-download already retried and still failed, so a one-off network blip is less likely than a persistently missing repo component.
  • How to verify: re-run the job; if it passes without cache changes, it was transient.
• [SPECULATION] sglang code regression in maybe_download_model / pipeline loading. No supporting evidence — the function correctly detected and reported an incomplete download; no diff evidence ties 592f6c849b (a mem-cache/HiCache split) to diffusion model download.

In-flight Fix Check

Searched open PRs with keywords Z-Image-Turbo / zimage — matches are all feature/optimization PRs (e.g. #23772, #21912, #29519); none addresses the model-cache/download CI failure.

• In-flight fix: ❌ none found — this is a runner model-cache/infra issue, not a code bug; no PR will fix it.

Suggested Triage Steps

• Inspect the HF cache on the failing MI325 runner: verify models--Tongyi-MAI--Z-Image-Turbo/.../f332072aa78be7aecdf3ee76d5c247082da564a6/ contains model_index.json, transformer/, and vae/; if incomplete, purge and re-pre-fetch the model on the runner image.
• Re-run the multimodal-gen-test-1-gpu-amd-rocm720 job once after cache repair to confirm the failure clears (expected, since it's not a code path).
• If the Z-Image-Turbo repo on HF is intermittently incomplete, consider pinning a known-good revision in the test/model-fetch config or pre-baking the model into the CI image to remove the network dependency.
• Confirm the sibling partition failure (83838610262) shares the same model-cache root cause (expected) rather than a distinct issue.

Status (factual, no priority assignment)

• Persistence: First observed in this run 28297015673 (Jun 27 ~17:59 UTC). No comparable completed run before it (job skipped or run cancelled), so persistence beyond this run is unknown.
• Scope: 2 test functions in 1 test file; affects ≥2 matrix partitions of multimodal-gen-test-1-gpu-amd-rocm720 in this run. Confined to the ROCm 7.2 diffusion/multimodal-gen CI lane.
• Blocked work: Z-Image-Turbo CLI generation signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found — runner model-cache/infra issue, needs runner-side model re-fetch, not a code PR.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev/patched image — Dockerfile default 7d604afe5, dev build preserved, rebuild skipped)

Failed Tests

All in python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py (summary: 5 failed, 1 passed, 4 errors in 675s):

Test File	Test Function	Error	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (server returned 500)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	Update failed: Inplace update to inference tensor outside InferenceMode is not allowed	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200)	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	Expected 200, got 500	Log
test_update_weights_from_disk.py	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	Expected 200, got 500	Log

Plus 4 setup ERRORS (distinct symptom — fixture bug) on the FLUX.2 no-offload variants (test_update_weights_from_disk_default, test_update_weights_specific_modules, test_update_weights_rejects_invalid_requests, test_corrupted_weights_rollback [FLUX.2-klein-base-4B]): AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* raised in fixture diffusion_server_no_offload at test_update_weights_from_disk.py:410 → _compute_checksum_from_disk (:220).

Failure Summary

The multimodal-gen-test-1-gpu job (shard 3) ran test_update_weights_from_disk.py on AMD MI325 / ROCm 7.2. Five update_weights_from_disk tests failed because the server returned HTTP 500 during weight reload; the captured scheduler traceback shows RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed at python/sglang/multimodal_gen/runtime/layers/linear.py:575 (param_data.copy_(loaded_weight)). Separately, four FLUX.2 no-offload variants errored at fixture setup with No weights dir for transformer (a test-fixture path issue, not the same root cause).

Regression Status

Never-passed (no green run found in queryable scheduled history).

Recent history of test_update_weights_from_disk.py in job multimodal-gen-test-1-gpu-amd-rocm720 (pr-test-amd-rocm720.yml, event=schedule, branch main, completed runs only):

Date (UTC)	Run	Job Status	Note
Jun 27 17:48	28297015673	❌ Failed	This run — first scheduled run where the job actually executed
Jun 27 18:24	28297867681	⏭️ Skipped	same head SHA, multimodal-gen job skipped (overall run "success" ≠ this test passing)
Jun 27 12:26	28289152816	⏭️ Skipped	job skipped
Jun 27 07:03	28281981946	⏭️ Skipped	job skipped
Jun 27 00:37	28273097445	⏭️ Skipped	job skipped

The multimodal-gen job is skipped in every other scheduled run in the window, so there is no passing baseline. This is the first scheduled execution of the test after the diffusion CI refactor #28762 (91f9e737, 7 commits before the run head), which modified weights_updater.py and this very test file. Lookback range: Jun 26 18:04 → Jun 28 00:37 UTC.

Failure Cluster

"Inplace update to inference tensor during update_weights_from_disk (multimodal_gen)"

Facts (verified observations)

• Scheduler traceback (verbatim): RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed.You can make a clone to get a normal tensor before doing inplace update. raised at weights_updater.py:200 (load_weights_into_model) → linear.py:533/:575 (param_data.copy_(loaded_weight)).
• The error is raised twice per request: once in _apply_weights (weights_updater.py:339) and again in _rollback (weights_updater.py:381), so rollback also fails the same way.
• Server returns HTTP 500 on POST /update_weights_from_disk; tests assert 200 → assert 500 == 200.
• The 4 FLUX.2 no-offload ERRORS are a different symptom: fixture diffusion_server_no_offload (test_update_weights_from_disk.py:410) calls _compute_checksum_from_disk(perturbed_vae_model_dir, "transformer") but the perturbed dir (/tmp/sglang_perturbed_vae_*) contains no transformer/ subdir → AssertionError: No weights dir for transformer.
• The codebase already wraps the offload consolidation path in torch.inference_mode(False) (python/sglang/multimodal_gen/runtime/managers/memory_managers/layerwise_offload.py:288); the non-offload copy path (load_weights_into_model → param_data.copy_) has no such wrapper.
• grep inference_mode in runtime/post_training/ returns nothing — the weight-update mixin does not enter an inference_mode(False) context.
• aiter ran as dev build v0.1.14rc1.dev240+g7d604afe5 ([CI-AITER-CHECK], log lines 4088-4094); matches Dockerfile default 7d604afe5, so aiter is unchanged.
• Not an amd-aiter-scout.yml run → no Baseline A/B / Failure Origin required.

Hypothesised Causes (with confidence)

• [MEDIUM] The non-offload weight-update path copies into model parameters that are inference tensors (created during model load under platform.inference_mode() = torch.inference_mode(True), see runtime/platforms/interface.py:329-336) without re-entering torch.inference_mode(False), so param_data.copy_(loaded_weight) at linear.py:575 is rejected by PyTorch.
  • Supporting: stack trace pins the exact line; the offload path at layerwise_offload.py:288 explicitly uses torch.inference_mode(False) for the same kind of in-place buffer write, showing the maintainers know this guard is required; runtime/post_training/ has zero inference_mode usage.
  • Disconfirming: the offload-enabled tests also failed (500), so a missing wrapper on the non-offload path alone cannot explain those two — the offload reload path (update_cpu_weights + remaining → load_weights_into_model) likely shares the same gap, or there is a second defect. Also cannot rule out an AMD-ROCm-specific interaction since the test has no NVIDIA baseline confirmed here.
  • How to verify: locally run test_update_weights_specific_modules[Qwen-Image]; wrap the load_weights_into_model copy in with torch.inference_mode(False): torch.no_grad(): (mirroring layerwise_offload.py:288) and re-run on AMD MI325 (rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260627).
• [FACT] The 4 FLUX.2 no-offload setup ERRORS are an independent test-fixture bug: _compute_checksum_from_disk is invoked for module transformer against a perturbed dir that only contains a perturbed vae.
  • Supporting: assertion message + fixture code path are unambiguous in the log.
  • Disconfirming: (none — this is a direct log observation, separate from the inference-tensor failure).
• [LOW] The diffusion CI refactor #28762 introduced the regression.
  • Supporting: it modified weights_updater.py (32 lines) and the test file, and enabled the test to run in scheduled CI 7 commits before the run head.
  • Disconfirming: the #28762 diff to weights_updater.py did not add/remove any inference_mode/copy_ lines (verified via git show), so it more likely exposed a pre-existing gap by enabling the test rather than introducing the bug.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for update_weights_from_disk diffusion (3 unrelated hits: #23244, #19153, #19152) and inference tensor InferenceMode (0 hits).

Suggested Triage Steps

• Reproduce test_update_weights_specific_modules[Qwen-Image] on AMD MI325 in container rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260627; confirm the 500 maps to the linear.py:575 inplace-update error.
• Test the hypothesis by wrapping the non-offload copy in load_weights_into_model (weights_updater.py:200 / linear.py:575) with torch.inference_mode(False) + torch.no_grad(), mirroring layerwise_offload.py:288; re-run all 5 failing tests including the two offload variants to confirm the offload path is covered by the same fix.
• Separately fix the FLUX.2 no-offload fixture (test_update_weights_from_disk.py:410/:220) so the perturbed-dir checksum check targets the module actually perturbed (vae), not transformer.
• If confirmed, since this never passed on AMD, decide whether to gate the test behind the fix landing or temporarily mark these update_weights_from_disk cases expected-fail on AMD until resolved.

Status

• Persistence: First and only scheduled execution (Jun 27 17:48 UTC); the job is skipped in all other scheduled runs, so no prior signal — treat as net-new failing on first run.
• Scope: 1 job (multimodal-gen-test-1-gpu-amd-rocm720, shard 3) in pr-test-amd-rocm720.yml; 5 failed + 4 errored tests, all in test_update_weights_from_disk.py. Two distinct defects (inference-tensor copy + fixture path bug).
• Blocked work: multimodal_gen update_weights_from_disk CI signal on AMD ROCm 7.2.
• In-flight fix: ❌ none found — needs a new PR or investigation.

stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 (linux-mi325-1gpu-sglang) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (source: dev image — v0.1.14rc1.dev240+g7d604afe5 detected, rebuild skipped; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/srt/test_vlm_models.py (suite stage-b-test-1-gpu-small-amd-nondeterministic)	TestVLMModels.test_vlm_mmmu_benchmark	No test failure — job cancelled mid-benchmark (not a test error)	Log

This job did not fail a test. The Run test step (step 6) was cancelled at 18:24:31 while the VLM MMMU benchmark (Qwen/Qwen2.5-VL-3B-Instruct) was still actively running.

Failure Summary

The job is conclusion: cancelled, not failure. The test_vlm_mmmu_benchmark server started successfully ("The server is fired up and ready to roll!" at 18:00:45) and the benchmark was actively processing batches (aiter GEMMs streaming continuously up to the final log line at 18:24:22) when the job was abruptly cancelled at 18:24:31 (line 6181, ##[error]The operation was canceled.). All 23 in-progress jobs in the run were cancelled in the same 18:24:17–18:24:32 window.

Regression Status

Infrastructure issue (run-level cancellation) — not a test regression.

[FACT] A newer scheduled run of the same workflow, 28297867681, was created at 2026-06-27T18:24:16Z — the exact moment every in-progress job in run 28297015673 was cancelled. The newer run shares the identical head SHA 592f6c84 and concluded success. This is a concurrency-group supersede: the later cron run cancelled the earlier still-running one.

Recent history of this job (pr-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date (UTC)	Run	Job Status	Note
Jun 27 18:24	28297867681	⏭️ Skipped	nondeterministic job not in active matrix
Jun 27 17:48	28297015673	🚫 Cancelled	this run — superseded mid-benchmark
Jun 27 12:26	28289152816	⏭️ Skipped	—
Jun 27 07:03	28281981946	⏭️ Skipped	—
Jun 27 00:37	28273097445	⏭️ Skipped	—

Note: this nondeterministic job is skipped in the surrounding scheduled runs, so there is no green/red test-level baseline to compare against. In this particular run it executed and was making progress until the external cancellation.

Failure Cluster

Run-level concurrency cancellation (superseded scheduled run) — no test failure.

Facts (verified observations, no interpretation)

• [FACT] Job conclusion is cancelled; the only failing step is Run test (step 6), conclusion cancelled. All other steps succeeded.
• [FACT] Server warmup completed normally: "The server is fired up and ready to roll!" at 18:00:45 (line ~4276).
• [FACT] A transient health-check warning at 18:01:12 ("Server couldn't get a response from detokenizer for last 20 seconds") self-recovered — GET /health_generate returned 200 OK at 18:01:23 after the mha_batch_prefill aiter kernel finished JIT-building (29.2s). This was not the cause of cancellation.
• [FACT] Benchmark was still actively running aiter GEMMs at 18:24:22, 9s before cancellation at 18:24:31 — no assertion, exception, or non-zero exit prior to cancellation.
• [FACT] Superseding run 28297867681 created at 18:24:16Z (same SHA) concluded success.
• [FACT] Every aiter GEMM in this run logged not found tuned config ... will use default config! using torch solution:0 — i.e. all GEMMs ran the untuned torch fallback path, making the benchmark slow (>23 min and still unfinished at cancellation).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Run was cancelled by the workflow's concurrency group when the next scheduled cron run superseded it.
  • Supporting: newer schedule run 28297867681 created at 18:24:16Z; all 23 in-progress jobs in 28297015673 cancelled in the 18:24:16–18:24:32 window; ##[error]The operation was canceled. with no preceding test error.
  • Disconfirming: (none found) — timing and the simultaneous cancellation of all in-flight jobs are consistent only with an external concurrency cancellation.
  • How to verify: inspect the concurrency: block in .github/workflows/pr-test-amd-rocm720.yml (expected cancel-in-progress: true on a schedule/ref key).
• [LOW] Untuned aiter GEMM fallback (torch solution:0) made the MMMU benchmark slow enough that the job was still running when the next cron fired, increasing the cancellation window.
  • Supporting: every GEMM logged the untuned fallback; benchmark ran >23 min without completing.
  • Disconfirming: this is a performance/tuning observation, not the cancellation trigger; a faster run would still have been subject to the same supersede if still in-flight.
  • How to verify: compare benchmark wall-time against a run with tuned bf16_tuned_gemm.csv configs present in /tmp/aiter_configs/.

In-flight Fix Check

No fix needed — this is a benign concurrency cancellation, not a code defect. No PR search performed (no failing test to fix).

Suggested Triage Steps

• Treat this job as non-actionable / cancelled; do not count it as a test failure in trend analysis.
• If these mid-benchmark cancellations are noisy, review the concurrency config in pr-test-amd-rocm720.yml — consider whether scheduled runs should cancel in-progress siblings, or whether the cron cadence (~6h) overlaps with long VLM-benchmark wall-times.
• Separately (perf, not correctness): investigate why aiter tuned GEMM configs (bf16_tuned_gemm.csv) are not matched at runtime so MMMU stops falling back to torch solution:0 — this would shorten the job and reduce its exposure to supersede cancellation.

Status (factual, no priority assignment)

• Persistence: single occurrence; run 28297015673 only.
• Scope: 23 jobs in this run were cancelled together at 18:24; this is run-wide, not specific to the nondeterministic job.
• Blocked work: none — superseding run 28297867681 (same SHA) passed, so CI signal for 592f6c84 is intact.
• In-flight fix: N/A — no defect to fix.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1) — failed step(s): N/A

Commit Info

• sglang: 592f6c84 — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: N/A — the target job never executed (0 steps, no log emitted, so no [CI-AITER-CHECK] markers).

Failed Tests

Test File	Test Function	Error	Log
— (no test ran)	—	Job cancelled before any step executed	Job page

This job did not fail a test — it was cancelled. The job (stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1), ID 83838610265) is recorded with conclusion: cancelled, 0 steps, and an empty runner name. Its log blob returns BlobNotFound because no runner ever picked it up — it sat queued from 17:49:17Z and was cancelled at 18:24:17Z.

Failure Summary

The target stage-c large-8-GPU job never started executing; it was cancelled as part of a run-wide cancellation (overall run 28297015673 conclusion: cancelled: 16 success, 10 failure, 23 cancelled). The cancellation was triggered by upstream stage-a / stage-b small-GPU job failures, which are dominated by HuggingFace 429 Too Many Requests model-download throttling — an infrastructure/rate-limiting condition, not a regression in this job's tests.

Regression Status

Infrastructure issue (run-wide fast-fail cancellation). Not applicable as a test-file regression — the failing test file for this job cannot be determined because the job produced no steps and no log. The 8-GPU stage-c suite was never exercised on this run.

Facts (verified observations, no interpretation)

• Target job 83838610265: conclusion=cancelled, steps=0, runner_name="", started 17:49:17Z, completed 18:24:17Z. Log endpoint returns <Error><Code>BlobNotFound</Code></Error>.
• Overall run 28297015673 conclusion=cancelled; 49 jobs total → 16 success, 10 failure, 23 cancelled.
• Upstream executed failures (the trigger), verbatim from their logs:
  • stage-a-test-1-gpu-small-amd-rocm720 (83838610220): httpx.HTTPStatusError: Client error '429 Too Many Requests' for url '.../meta-llama/Llama-3.1-8B-Instruct' → ERROR: setUpClass (TestBasicSanityEagle3) → test_basic_sanity_eagle3.py returned exit code 1.
  • stage-b-test-1-gpu-small-amd-rocm720 (…, 6) (83838610271): 429 Too Many Requests for url '.../RedHatAI/Meta-Llama-3.1-8B-FP8' → ERROR: setUpClass (TestCompressedTensorsLlama3FP8).
  • stage-b-test-1-gpu-small-amd-rocm720 (…, 7) (83838610285): 429 Too Many Requests for url '.../Qwen/Qwen2.5-VL-3B-Instruct' and Server process exited with code -9.
• A sibling stage-c job in the same run (matrix index differing) did begin executing (9 steps) before also being cancelled at 18:24:32Z, confirming the cancellation was run-level, not job-specific.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The job was cancelled by a run-wide fast-fail cascade, not by its own test logic.
  • Supporting: 0 steps + empty runner + BlobNotFound ⇒ job never ran; overall run conclusion=cancelled; 23 of 49 jobs cancelled simultaneously at ~18:24Z; the pr-test-amd-rocm720-finish gate job is in failure.
  • Disconfirming: (none found) — no evidence the 8-GPU suite itself produced any error.
• [MEDIUM] The triggering upstream failures are HuggingFace HTTP 429 rate-limiting (model-download throttling), an environmental/infra condition.
  • Supporting: three independent upstream jobs fail at setUpClass/server-startup with 429 Too Many Requests on distinct model URLs (Llama-3.1-8B, Meta-Llama-3.1-8B-FP8, Qwen2.5-VL-3B); failures are at download time, not in assertion logic.
  • Disconfirming: job 83838610285 also shows Server process exited with code -9 (killed/OOM), which is a separate symptom and may not be purely network-driven; 429 is transient and could mask a genuine code failure further along.
  • How to verify: re-run the affected stage-a/stage-b shards; if 429s clear and tests pass, confirms infra throttling.

In-flight Fix Check

• In-flight fix: ❌ none applicable — there is no defect in this job to fix. The upstream 429 throttling is an infrastructure condition (HF Hub rate limit), typically mitigated by retry/caching at the CI layer rather than an sglang PR. (No code-level regression identified to search for.)

Suggested Triage Steps

• Treat this job as cancelled, not failed — exclude it from regression accounting for this run; no 8-GPU stage-c signal exists for SHA 592f6c84.
• Investigate the real trigger upstream: re-run stage-a-test-1-gpu-small-amd-rocm720 (83838610220) and the two stage-b shards (83838610271, 83838610285); if the 429s clear on retry, the cancellation was purely an HF-throttling cascade.
• Separately check the Server process exited with code -9 in 83838610285 (Qwen2.5-VL-3B) — confirm whether it is OOM/kill on the 1-GPU runner vs a real crash, independent of the 429.
• To obtain an actual 8-GPU stage-c result, re-run the workflow once HF rate-limiting subsides (or after adding model caching), so stage-c is allowed to start rather than being fast-fail cancelled.

Status (factual, no priority assignment)

• Persistence: Single-run event on run 28297015673 (2026-06-27, schedule on main). No execution history for this job in this run to trend.
• Scope: Run-wide — 23 jobs cancelled, 10 executed failures; the cancellation of this 8-GPU stage-c job is a downstream effect of upstream stage-a/stage-b failures.
• Blocked work: No 8-GPU stage-c (linux-mi325-8gpu) test coverage was produced for this scheduled run.
• In-flight fix: ❌ none applicable — no job-level defect; upstream cause is HF 429 infra throttling.

multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2) — failed step(s): N/A

Commit Info

• sglang: 592f6c84 (592f6c849bf972370782d915c7a6528c158c2c80)
• aiter: N/A — job log is unavailable (BlobNotFound, see Facts), so no [CI-AITER-CHECK] marker could be read. This is also not an amd-aiter-scout.yml run.

Failed Tests

Test File	Test Function	Error	Log
N/A	N/A	Job cancelled, not failed — no test error	Job

This job (multimodal-gen-test-2-gpu-amd-rocm720 (linux-mi325-2gpu-sglang, 2), partition 2 = standalone single_test_file/test_disagg_server.py, invoked via multimodal_gen/test/run_suite.py --suite 2-gpu --partition-id 2 --total-partitions 3) did not produce a test failure. Its conclusion is cancelled.

Failure Summary

The job was cancelled, not failed. It started at 17:49:17Z and ran for ~35 min before being terminated at 18:24:17Z. The cancellation was triggered by the workflow's cancel-in-progress concurrency policy: a newer scheduled run of the same workflow started at 18:24:16Z and superseded this in-flight run. No test result was recorded and the job log blob does not exist.

Regression Status

Infrastructure / not a test failure (concurrency cancellation). This is not a regression — the job was a victim of cancel-in-progress, not a code defect.

Recent history of multimodal-gen-test-2-gpu-amd-rocm720 (…, 2) (same branch main, same event schedule, completed runs only):

Date	Run	Job Status	Note
Jun 27 17:48	28297015673	⏹️ Cancelled	Superseded by newer run (this job)
Jun 27 18:24	28297867681	⏭️ Skipped	check-changes gated it off; run succeeded
Jun 27 12:26	28289152816	⏭️ Skipped	Not triggered
Jun 27 07:03	28281981946	⏭️ Skipped	Not triggered

The 2-GPU multimodal-gen matrix is normally skipped (gated on main_package/sgl_kernel changes via check-changes); it was only expanded in run 28297015673. No green-or-red completed run for this exact test exists in the queryable window, because it normally does not run — its cancellation here is purely concurrency-driven.

Facts (verified observations, no interpretation)

• [FACT] Job 83838610269 conclusion = cancelled (GitHub API job endpoint); steps array is empty.
• [FACT] Job log fetch returns <Error><Code>BlobNotFound</Code> — no log retained for the cancelled job.
• [FACT] Run 28297015673 conclusion = cancelled, event = schedule, branch = main, head_sha = 592f6c84.
• [FACT] 23 of 49 jobs in the run are cancelled, all completing simultaneously at 18:24:17Z / 18:24:31Z / 18:24:32Z — the signature of a single run-wide cancellation event, not independent failures.
• [FACT] A newer scheduled run 28297867681 was created at 2026-06-27T18:24:16Z (one second before the cancellations) and completed with conclusion = success.
• [FACT] The workflow sets cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} (pr-test-amd-rocm720.yml:123), which is true for schedule events — so a newer schedule run cancels the older one.
• [FACT] The pr-test-amd-rocm720-finish gate (lines 1236-1290) only reports status (exit 1 on any failure/cancellation); it does not cancel sibling jobs.
• [FACT] 10 sibling jobs in this run genuinely failed between 18:00-18:21Z (e.g. stage-a-test-1-gpu-small, stage-b-test-1-gpu-small shards, multimodal-gen-test-1-gpu shards 2 & 3, extra-a-test-1-gpu-large) — these are separate from this cancelled job and out of scope for this per-job analysis.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] This job was cancelled by the cancel-in-progress concurrency policy when newer scheduled run 28297867681 started at 18:24:16Z.
  • Supporting: newer run created timestamp (18:24:16Z) is one second before this job's cancellation (18:24:17Z); all 23 cancelled jobs stopped at the same instant; concurrency cancel-in-progress=true for schedule events (pr-test-amd-rocm720.yml:123); the superseding run succeeded.
  • Disconfirming: (none found) — timing, the concurrency config, and the simultaneous mass cancellation all converge.
  • How to verify: already verified via the workflow runs list (two consecutive schedule runs, the older cancelled, the newer success) and the concurrency block in the workflow file.

There is no evidence of a code-level test failure in this job. The genuine failures elsewhere in the run are unrelated to this job's cancellation.

In-flight Fix Check

Not applicable — this is a concurrency cancellation, not a defect, so no fix is warranted. No PR search performed (nothing to fix).

Suggested Triage Steps

• No action needed for this job. Treat the cancelled conclusion as noise from the cancel-in-progress concurrency policy; the superseding run 28297867681 passed.
• If a green/red signal for multimodal-gen-test-2-gpu partition 2 is specifically wanted, re-dispatch pr-test-amd-rocm720.yml with the 2-GPU multimodal-gen stage targeted (it is normally skipped by check-changes gating).
• The real failures in this run (stage-a, stage-b shards, multimodal-gen-test-1-gpu shards 2 & 3, extra-a-test-1-gpu-large, finished 18:00-18:21Z) should be triaged separately under their own per-job analyses — they are not connected to this cancellation.

Status (factual, no priority assignment)

• Persistence: One-off concurrency cancellation in a single run; not a recurring condition for this test.
• Scope: 23 jobs in run 28297015673 cancelled simultaneously by the same concurrency event; the next scheduled run succeeded.
• Blocked work: None — the superseding scheduled run provided fresh CI signal and passed.
• In-flight fix: N/A (no defect).

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 — v0.1.14rc1.dev240+g7d604afe5 (source: Dev/patched image preserved; AITER_COMMIT_OVERRIDE empty, so this is not an AITER Scout run)

Failed Tests

All in sglang/multimodal_gen/test/server/test_server_1_gpu.py::TestDiffusionServerOneGpu::test_diffusion_generation[<param>]:

Test File	Test Function (param)	Error	Log
multimodal_gen/test/server/test_server_1_gpu.py	[qwen_image_t2i]	ERROR — ValueError: Could not find model at Qwen/Qwen-Image … download incomplete (missing model_index.json, transformer/, vae/)	Log
multimodal_gen/test/server/test_server_1_gpu.py	[qwen_image_t2i_cache_dit_enabled]	ERROR — same Qwen-Image download-incomplete ValueError	Log
multimodal_gen/test/server/test_server_1_gpu.py	[qwen_image_edit_2509_ti2i]	FAILED — torch._inductor.exc.InductorError: AssertionError in layernorm.py:703 forward_hip→forward_native	Log
multimodal_gen/test/server/test_server_1_gpu.py	[wan2_1_t2v_1.3b_teacache_enabled]	FAILED — torch._inductor.exc.InductorError: AssertionError → RuntimeError: Model generation returned no output	Log
multimodal_gen/test/server/test_server_1_gpu.py	[wan2_1_t2v_1.3b_frame_interp_2x]	Not concluded — job canceled mid-run (##[error]The operation was canceled.)	Log

Passed in this shard: flux_image_t2i, zimage_image_t2i, zimage_image_t2i_fp8, qwen_image_edit_2511_ti2i, joyai_image_edit_ti2i. Skipped: cosmos3_nano_t2v (AttributeError: module diffusers has no attribute Cosmos3OmniTransformer).

Failure Summary

The 1-GPU multimodal-gen diffusion server suite ran the parametrized test_diffusion_generation cases on MI325/ROCm 7.2. Two cases (qwen_image_edit_2509_ti2i, wan2_1_t2v_1.3b_teacache_enabled) failed with a torch._inductor graph-node-count AssertionError raised while compiling the native layernorm path; two qwen_image_t2i* cases errored on an incomplete HuggingFace model download; the shard was then canceled while running wan2_1_t2v_1.3b_frame_interp_2x. The overall job conclusion is cancelled.

Regression Status

Known recurring (job-level) — needs deeper history to confirm at test-function level.

Recent history of job multimodal-gen-test-1-gpu-amd-rocm720 on pr-test-amd-rocm720.yml (same branch main, event=schedule, completed runs only; skipped runs excluded):

Date	Run	Job conclusion (when non-skipped)	Notes
Jun 27	28297015673	cancelled (shard 0) / failure (sibling shards)	this run
Jun 26	28256293905	cancelled (all shards)	log truncated/empty
Jun 25	28190970158	cancelled (all shards)	log truncated/empty
Jun 24	28119463964	cancelled (all shards)	log truncated/empty
Jun 23	28046873327	cancelled (all shards)	log truncated/empty

The job is skipped on most scheduled runs (conditional trigger); when it does run it has been cancelled/failure every day Jun 23–27. The re-run 28297867681 at the same head SHA had multimodal-gen skipped, so it does not exonerate this run. I could not confirm test-function-level recurrence because the prior cancelled-job logs return empty (cancelled jobs truncate their logs). Lookback window: Jun 23–28, 2026.

Failure Cluster

Three distinct clusters in this shard:

1. torch.compile InductorError (graph node-count AssertionError) on ROCm — diffusion denoising layernorm path
2. HuggingFace model download incomplete (Qwen-Image) — infra/cache
3. Job cancellation (fail-fast / timeout) — frame_interp_2x cut off

Facts (verified observations)

• [FACT] qwen_image_edit_2509_ti2i and wan2_1_t2v_1.3b_teacache_enabled both fail with torch._inductor.exc.InductorError: AssertionError whose final frame is assert len(graph_with_eager_vals.graph.nodes) == len(...). Stack originates in python/sglang/multimodal_gen/runtime/layers/layernorm.py:703 (forward_hip → forward_native), reached from dits/qwen_image.py:1161 (txt_norm1) via denoising.py _predict_noise.
• [FACT] Immediately preceding each InductorError, the log repeats ImportError: cannot import name 'specialize_impl' from 'triton.runtime.jit' (/sgl-workspace/triton-custom/python/triton/runtime/jit.py) from torch/_higher_order_ops/triton_kernel_wrap.py:948.
• [FACT] qwen_image_t2i / qwen_image_t2i_cache_dit_enabled error with ValueError: … Downloaded model at …models--Qwen--Qwen-Image… is still incomplete after forced re-download. The model repository may be missing required components (model_index.json, transformer/, or vae/).
• [FACT] Job ends with ##[error]The operation was canceled. at 18:24:30 UTC while wan2_1_t2v_1.3b_frame_interp_2x was running; that case's warmup logged causal Conv3D cat/pad fusion is only available on CUDA.
• [FACT] aiter version is the Dockerfile-default dev image (v0.1.14rc1.dev240+g7d604afe5), preserved (no rebuild, no override) per [CI-AITER-CHECK] markers.
• [FACT] flux, zimage (×2), qwen_image_edit_2511, joyai cases PASSED in the same shard — the failure is case-specific, not a whole-server bringup failure.

Hypothesised Causes (with confidence + disconfirming evidence)

• [MEDIUM] torch.compile/Inductor incompatibility with the container's triton-custom build causes the layernorm forward_native compile to fail for the Qwen-Image-Edit-2509 and Wan-teacache denoising graphs.
  • Supporting: specialize_impl ImportError from triton-custom/.../jit.py fires repeatedly right before each InductorError; both InductorErrors share the identical layernorm.py:703 compile site.
  • Disconfirming: other diffusion models (flux, zimage, qwen_edit_2511, joyai) pass through the same server/layernorm path without an InductorError, so the trigger is model/graph-shape specific, not a blanket Inductor breakage; the ImportError is logged as a warning (assuming every input is mutated) and may be a benign red herring rather than the compile-abort cause.
  • How to verify: re-run only [qwen_image_edit_2509_ti2i] with TORCHDYNAMO_VERBOSE=1/TORCH_LOGS=+dynamo in the ROCm 7.2 container to capture the real assertion; A/B with enable_torch_compile=false to confirm the compile path is the trigger.
• [FACT] qwen_image_t2i* ERRORs are an HF model-fetch/cache problem (missing model_index.json/transformer//vae/), independent of sglang code — infra, not a code regression.
  • Disconfirming: none — the error is explicitly a download-completeness check.
• [LOW] Job cancellation is fail-fast cascade: sibling shards reported failure and shard 0 was cancelled mid-test.
  • Supporting: 4 shards in this run; 2 failure, 2 cancelled; cancellation message is generic.
  • Disconfirming: could alternatively be a job/step timeout — not distinguishable from this log alone.

In-flight Fix Check

• In-flight fix: ❌ none found directly targeting the diffusion InductorError or the Qwen-Image download error. Searched open PRs for specialize_impl, InductorError diffusion, Qwen-Image download incomplete, causal Conv3D. The only loosely-related open PR is #21839 ("[AMD] ROCm Dockerfile update 720 base image"), which touches the ROCm/triton image and may affect the triton-custom specialize_impl mismatch — verify, do not assume.

Suggested Triage Steps

• Reproduce [qwen_image_edit_2509_ti2i] alone in container rocm/sgl-dev:v0.5.14-rocm720-mi30x-20260627 with TORCHDYNAMO_VERBOSE=1 to capture the suppressed assertion text behind InductorError: AssertionError.
• A/B the InductorError cases with torch.compile disabled (enable_torch_compile=false) to confirm the compile path is the trigger; if so, the fix direction is the layernorm forward_native compile path on ROCm.
• Check whether the triton-custom build in the image exposes specialize_impl; track #21839 as a candidate image fix for the triton/inductor mismatch.
• Treat the qwen_image_t2i* download ERRORs as infra: validate/repopulate the Qwen/Qwen-Image HF cache on the MI325 runners; these are not a code regression.
• Investigate the recurring job-level cancellation separately (fail-fast vs job timeout) — inspect sibling shards 83838610262 / 83838610263 for the first failing shard that may be triggering fast-fail.

Status

• Persistence: multimodal-gen-test-1-gpu-amd-rocm720 has been cancelled/failure on every non-skipped scheduled run Jun 23–27, 2026; skipped on most other scheduled runs. Test-function-level recurrence unconfirmed (prior cancelled-job logs are empty).
• Scope: This run's 1-GPU multimodal-gen shards (shard 0 cancelled, siblings failure). Confirmed failing params: qwen_image_edit_2509_ti2i, wan2_1_t2v_1.3b_teacache_enabled (InductorError); qwen_image_t2i, qwen_image_t2i_cache_dit_enabled (HF download).
• Blocked work: AMD ROCm 7.2 diffusion (multimodal-gen) CI signal for Qwen-Image-Edit-2509 and Wan2.1 teacache paths.
• In-flight fix: ❌ none found targeting these failures; #21839 is a candidate ROCm image PR worth checking for the triton/inductor mismatch.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:58 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 5/5)

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12) — failed step(s): Run test

Commit Info

• sglang: 592f6c84 — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: 7d604afe5 (source: dev image — Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/spec/dflash/test_dflash.py	TestDFlashServerBase.setUpClass	Exception: Server process exited with code -9 (root cause: HfHubHTTPError: 429 Too Many Requests)	Log

Failure Summary

The setUpClass of TestDFlashServerBase failed while launching the DFlash speculative-decoding server for meta-llama/Llama-3.1-8B-Instruct. The local HF snapshot had no weight files, so the model loader attempted to download weights from the Hugging Face Hub and received an HTTP 429 Too Many Requests (api rate limit, 3000 req/5min quota exhausted). The scheduler child crashed, popen_launch_server reported the server exited with code -9, and the test errored. The job was subsequently cancelled at 18:24:30 UTC while a later test (test_deterministic.py) was still running.

Regression Status

Infrastructure issue (transient HF rate limit) — not a code regression.

Recent history of test/registered/spec/dflash/test_dflash.py in pr-test-amd-rocm720.yml (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Note
Jun 27 18:24	28297867681	✅ Passed	Re-run at identical head SHA 592f6c849b — workflow succeeded
Jun 27 17:48	28297015673	❌ Failed	This run (HF 429)
Jun 27 12:26	28289152816	✅ Passed	—
Jun 27 07:03	28281981946	✅ Passed	—

The immediate re-run at the same sglang SHA passed, conclusively ruling out a code regression in 592f6c849b.

Failure Cluster

HF Hub 429 Too Many Requests rate limit during model weight download (server setUpClass crash).

Facts (verified observations)

• httpx.HTTPStatusError: Client error '429 Too Many Requests' for url 'https://huggingface.co/api/models/meta-llama/Llama-3.1-8B-Instruct' (log line ~4161).
• 429 Too Many Requests: you have reached your 'api' rate limit. Retry after 54 seconds (0/3000 requests remaining in current 300s window) — current: team plan (log ~4194).
• Trigger: Local HF snapshot at .../models--meta-llama--Llama-3.1-8B-Instruct/... has no files matching ['*.safetensors','*.bin','*.pt']; will attempt download. (log ~4154) → fell through to a live HF Hub fs.ls call that got rate-limited.
• Stack trace path: model_runner.load_model → loader.load_model → _prepare_weights → download_weights_from_hf → hf_file_system.ls → repo_info (sglang model_loader/weight_utils.py:568).
• Server exit reported as code -9 by popen_launch_server (test_utils.py:1027); Ran 0 tests in 36.035s … FAILED (errors=1).
• Same SHA re-run 28297867681 succeeded.
• aiter unchanged vs Dockerfile default (dev build 7d604afe5 preserved).

Hypothesised Causes (with confidence)

• [FACT] Transient Hugging Face Hub API rate limiting (HTTP 429) during weight download for meta-llama/Llama-3.1-8B-Instruct.
  • Supporting: explicit 429 ... you hit the quota of 3000 api requests per 5 minutes period in the log; the local cache lacked weight files forcing a live download; immediate re-run at the same SHA passed.
  • Disconfirming: (none found) — the error is unambiguous and reproducibly absent on retry.
  • How to verify: already verified — re-run 28297867681 at identical SHA is green.
• [LOW] Stale/incomplete HF snapshot cache on the linux-mi325-1gpu-sglang runner (snapshot dir present but missing weight files), which forced the online fetch that hit the rate limit.
  • Supporting: has no files matching ['*.safetensors','*.bin','*.pt']; will attempt download indicates the cache was not fully populated.
  • Disconfirming: other models in the same suite loaded fine; cannot confirm cache state across runners from this log alone.
  • How to verify: inspect /sgl-data/hf-cache/hub/models--meta-llama--Llama-3.1-8B-Instruct on the runner; pre-seed weights.

In-flight Fix Check

• In-flight fix: ❌ none found — searched repo:sgl-project/sglang is:pr is:open 429 rate limit huggingface (0 results). No code fix is warranted for a transient infra rate limit.

Suggested Triage Steps

• No code action needed — confirm closure by noting the same-SHA re-run 28297867681 passed.
• If HF 429s recur, consider pre-seeding meta-llama/Llama-3.1-8B-Instruct weights into the runner HF cache (/sgl-data/hf-cache) so DFlash setUpClass does not hit the live Hub, and/or set HF_HUB_OFFLINE=1 for models with cached weights.
• Investigate why the snapshot dir existed but lacked weight files (partial prior download) to avoid future online fallbacks during rate-limit windows.

Status

• Persistence: Single occurrence in run 28297015673; the very next scheduled run at the same SHA passed.
• Scope: This job/test only (stage-b-test-1-gpu-small-amd-rocm720, partition 12); failure is environmental (HF Hub), not sglang/aiter code.
• In-flight fix: ❌ none found — not needed (transient infrastructure rate limit).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 6) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe — v0.1.14rc1.dev240+g7d604afe5 (source: dev/patched image preserved; not relevant to either failure below)

Failed Tests

Test File	Test Function	Error	Log
test/registered/utils/test_type_based_dispatcher.py	test_type_dispatcher_e2e_performance	TypeError: Missing required argument 'input_embeds'	Log
test/registered/models/test_compressed_tensors_models.py	TestCompressedTensorsLlama3FP8.setUpClass	Exception: Server process exited with code -9 (root: HfHubHTTPError 429)	Log

Failure Summary

The job (stage-b-test-1-gpu-small-amd-rocm720 shard 6) failed at step "Run test" with two independent failures. (1) test_type_dispatcher_e2e_performance raised a TypeError constructing TokenizedGenerateReqInput without the now-required input_embeds keyword (test/code mismatch). (2) TestCompressedTensorsLlama3FP8.setUpClass failed because the server process exited with code -9 after the model-load path hit a HuggingFace 429 Too Many Requests rate limit while downloading RedHatAI/Meta-Llama-3.1-8B-FP8 weights (external/infra).

Regression Status

• test_type_based_dispatcher.py: New regression (test not updated after struct field change) — Never-passed in queryable scheduled history, see note below.
• test_compressed_tensors_models.py: Infrastructure issue (transient HF rate limit), not a code regression.

Recent history of stage-b-test-1-gpu-small-amd-rocm720 (same branch main, event=schedule, completed runs only):

Date	Run	Job Status	Note
Jun 27 18:24	28297867681	⏭️ stage-b-small skipped	same head SHA 592f6c84; test not executed
Jun 27 12:26	28289152816	⏭️ stage-b-small skipped	test not executed
Jun 27 07:03	28281981946	⏭️ stage-b-small skipped	test not executed
Jun 27 00:37	28273097445	⏭️ stage-b-small skipped	test not executed

Note: stage-b-test-1-gpu-small was skipped in every recent scheduled run, so test_type_based_dispatcher.py has no green baseline in the queryable window (lookback ~24 h / last 8 scheduled runs). The current run is the only one where the shard actually executed this test; it failed deterministically.

Failure Cluster

• test_type_based_dispatcher: "TypeError: missing required input_embeds arg in TokenizedGenerateReqInput construction"
• test_compressed_tensors_models: "Server exit -9 from HuggingFace 429 rate limit during weight download"

Facts (verified observations)

• TokenizedGenerateReqInput is defined at python/sglang/srt/managers/io_struct.py:771 as class TokenizedGenerateReqInput(BaseReq, kw_only=True); input_embeds: Optional[List[List[float]]] (line 776) has no default → it is a required keyword argument. [FACT]
• The test at test/registered/utils/test_type_based_dispatcher.py:124-134 constructs TokenizedGenerateReqInput(...) and omits input_embeds (and token_type_ids) → TypeError: Missing required argument 'input_embeds'. [FACT]
• Commit be193013 (#28688, "Convert IPC dataclasses to msgspec.Struct", merged 2026-06-26 19:04 UTC) converted @dataclass class TokenizedGenerateReqInput(BaseReq) → msgspec.Struct(... kw_only=True); it is an ancestor of the failing SHA. [FACT]
• The compressed-tensors failure: scheduler child raised huggingface_hub.errors.HfHubHTTPError ... 429 Too Many Requests ... you have reached your 'api' rate limit at weight_utils.py:568 (download_weights_from_hf), then Received sigquit from a child process, server exited -9. [FACT — job log lines 12451-12524]
• aiter was unchanged (dev image preserved); NVIDIA-independent — both failures are AMD-run-agnostic in nature (one is pure Python type error, one is network rate limit). [FACT]

Hypothesised Causes

• [HIGH] The test_type_dispatcher_e2e_performance failure is a test/struct contract mismatch: input_embeds is a required field of TokenizedGenerateReqInput and the test omits it.
  • Supporting: error message names the exact missing arg; struct def confirms no default; open fix PR #29376 edits only the test (+6/-2) to add the missing fields, titled "Fix test_type_based_dispatcher after TokenizedGenerateReqInput field changes".
  • Disconfirming: input_embeds also had no default in the prior @dataclass form, so the test would strictly have failed before be193013 too — meaning the precise commit that "broke" it cannot be pinned to the msgspec conversion alone; the test simply has not run (stage-b-small skipped) in recent scheduled history, so no green baseline exists to anchor a regression point.
  • How to verify: merge/apply #29376 and re-run test/registered/utils/test_type_based_dispatcher.py on AMD MI325.
• [FACT] The TestCompressedTensorsLlama3FP8 failure is caused by a transient HuggingFace API rate limit (429), not sglang/aiter code.
  • Supporting: log shows 429 ... 0/3000 requests remaining in current 300s window for RedHatAI/Meta-Llama-3.1-8B-FP8; server killed via sigquit → exit -9; same step also logged 429 for the model tree listing.
  • Disconfirming: (none — this is directly observable as an external HTTP 429).
  • How to verify: re-run the shard; if it passes, confirms transient rate limit. Consider pre-caching the model or HF_HUB_OFFLINE for CI.

In-flight Fix Check

• test_type_based_dispatcher: ✅ #29376 (open since 2026-06-26, mergeable_state: blocked) — adds the required input_embeds/token_type_ids to the test. Chase reviewers; do NOT open a duplicate.
• test_compressed_tensors_models (HF 429): ❌ none found — transient infra; no code fix needed.

Suggested Triage Steps

• For test_type_based_dispatcher: unblock and merge in-flight fix #29376, then re-run shard 6 to confirm green.
• For test_compressed_tensors_models: re-run the shard (likely passes once HF rate limit clears). If it recurs in CI, consider caching RedHatAI/Meta-Llama-3.1-8B-FP8 in the runner image or setting an authenticated HF token / higher rate-limit tier for CI weight downloads.
• Investigate why stage-b-test-1-gpu-small is skipped in most scheduled runs but executed here — the test had no green baseline because the shard rarely runs, masking the broken test until now.

Status

• Persistence: test_type_based_dispatcher fails deterministically whenever the shard runs; no green run in the queryable scheduled window (shard skipped in all 4 prior runs checked). The HF-429 failure is transient (first/only observed occurrence in this run).
• Scope: 1 job, shard 6 of stage-b-test-1-gpu-small-amd-rocm720. Both failures are independent; the input_embeds mismatch would affect any backend running this test (not AMD-specific).
• In-flight fix: ✅ #29376 for the test mismatch; none needed for the HF-429 infra failure.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 7) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: v0.1.14rc1.dev240+g7d604afe5 (source: dev image — Dev/patched version detected → skipping rebuild; Dockerfile default was 7d604afe)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_vlm_perf_5090.py	TestVLMPerf5090.test_vlm_offline_throughput	Exception: Server process exited with code -9 (server died after httpx.HTTPStatusError: 429 Too Many Requests from huggingface.co)	Log

Failure Summary

The single failing test is test_vlm_offline_throughput in test/registered/perf/test_vlm_perf_5090.py (9/10 tests in the shard passed). The SGLang server failed to launch because the local HF snapshot for Qwen/Qwen2.5-VL-3B-Instruct had no weight files (has no files matching ['*.safetensors', '*.bin', '*.pt']; will attempt download), forcing an online download that hit a HuggingFace 429 rate limit (you hit the quota of 3000 api requests per 5 minutes). The scheduler aborted via sigquit and the server process exited with code -9 on both attempts, failing the test.

Regression Status

Infrastructure issue (transient) — not a code regression.

Recent history of test_vlm_perf_5090.py (test_vlm_offline_throughput) in job stage-b-test-1-gpu-small-amd-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Conclusion	Test File Status	Note
Jun 27 18:24	28297867681	✅ success	✅ Passed	Same head_sha 592f6c849b as failing run
Jun 27 17:48	28297015673	cancelled	❌ Failed	This run (HF 429)
Jun 27 12:26	28289152816	✅ success	✅ Passed	—
Jun 27 07:03	28281981946	✅ success	✅ Passed	—

The decisive evidence: the identical commit 592f6c849b passed in run 28297867681 ~35 minutes later. A code regression cannot pass and fail on the same SHA — this is a transient HF rate-limit/cache problem.

Failure Cluster

HuggingFace 429 rate limit → server launch failure (exit -9) during VLM weight download

Facts (verified observations, no interpretation)

• Log line 4162: Local HF snapshot at /sgl-data/hf-cache/hub/models--Qwen--Qwen2.5-VL-3B-Instruct/... has no files matching ['*.safetensors', '*.bin', '*.pt']; will attempt download. — the cache was incomplete, forcing an online fetch.
• Log lines 4168, 4224-4227: httpx.HTTPStatusError: Client error '429 Too Many Requests' and you hit the quota of 3000 api requests per 5 minutes period.
• Log line 4174-4205: the 429 was raised inside download_weights_from_hf → fs.ls(...) during qwen2_5_vl.py:795 load_weights at scheduler init (production model-load path, not test logic).
• Log lines 4229-4244: Received sigquit from a child process → Server process exited with code -9 on the first attempt; retry also failed → retry() exceed maximum number of retries → FAILED (errors=1).
• Log line 4411 / 6267: ✗ FAILED: test_vlm_perf_5090.py returned exit code 1; final shard Test Summary: 9/10 passed; job exit code 255.
• Same head_sha 592f6c849b passed in the next scheduled run (verified via API conclusions above).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [FACT] HuggingFace API 429 rate limit during weight download caused the server launch to fail.
  • Supporting: verbatim 429 Too Many Requests ... you hit the quota of 3000 api requests per 5 minutes (log 4224-4227); the download was only attempted because the local snapshot lacked weight files (log 4162).
  • Disconfirming: (none found) — the same SHA passing 35 min later is consistent with, not contrary to, a transient rate limit.
• [LOW] Incomplete/corrupt local HF cache (/sgl-data/hf-cache) on this runner forced the online fetch that then hit the limit.
  • Supporting: log 4162 shows the snapshot directory existed but had no weight files; a complete cache would have avoided the HF API call entirely.
  • Disconfirming: cannot confirm whether the cache is runner-specific or shared; not verifiable from this log alone.

This failure is environmental; no sglang or aiter commit is implicated. (Not a scout run, so no aiter A/B attribution required.)

In-flight Fix Check

• In-flight fix: ✅ #23400 — "ci: prefetch HF snapshots and skip redundant metadata calls to survive 429s". Directly targets this exact failure mode. Chase reviewers; do NOT open a duplicate.
• Search ran: is:pr is:open 429 huggingface (2 results; #23400 is the relevant one).

Suggested Triage Steps

• Treat as transient infra; a re-run of run 28297015673 already effectively succeeded on the same SHA (28297867681) — no code action needed.
• Verify in-flight PR #23400 lands to prefetch HF snapshots and cut redundant metadata calls, which would prevent the 429 path.
• Investigate why the local snapshot at /sgl-data/hf-cache/hub/models--Qwen--Qwen2.5-VL-3B-Instruct had no weight files on this runner (incomplete pre-seed) — a complete cache makes the test independent of HF availability.

Status

• Persistence: single occurrence; same commit passed ~35 min later in run 28297867681. Not persistent.
• Scope: 1 test (test_vlm_offline_throughput) in 1 job shard; 9/10 of the shard passed.
• Blocked work: none — transient HF rate limit, run already re-passed.
• In-flight fix: ✅ #23400 (prefetch HF snapshots to survive 429s).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: 592f6c84 — [minimax-m3] Split 2/4: mem-cache / HiCache / sparse KV pool (#28713)
• aiter: 7d604afe5 (source: dev image preserved — log: Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild; AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/moe/test_torch_compile_moe.py	N/A (cancelled during server warmup / CUDA graph capture, before any assertion)	Job cancelled: ##[error]The operation was canceled.	Log

This partition (2/14) enabled exactly one test file: test/registered/moe/test_torch_compile_moe.py (model Qwen/Qwen1.5-MoE-A2.7B, --enable-torch-compile).

Failure Summary

The job did not fail on a test assertion — it was cancelled at 18:24:30 UTC while still capturing CUDA graph batches (progress shown at 51/52). The run reached that point only after the scheduler spent ~24 minutes blocked downloading the model weights from HuggingFace. No pytest failure or numerical mismatch occurred.

Regression Status

Infrastructure issue (cancellation), not a code regression for this test file.

• [FACT] The scheduler (PID 3634) was stuck in huggingface_hub snapshot_download → thread_map → … → xet_get (file_download.py:566) from ~17:58:50 onward (py-spy stack at log). Triggered the soft watchdog (soft_watchdog_timeout=300, soft=True so non-fatal) repeatedly at 18:03/18:08/18:13/18:18/18:23 UTC.
• [FACT] Local HF snapshot was incomplete: Local HF snapshot … has no files matching ['*.safetensors']; will attempt download (log), forcing the slow network download.
• [FACT] Download eventually completed (~18:23), model loaded, and CUDA-graph capture began progressing (Capturing batches … 51/52), i.e. the test was healthy and running when it was cut off.
• [HIGH] Cancellation was via GitHub Actions concurrency cancel-in-progress: the re-run 28297867681 of the same head SHA 592f6c84 was created at 18:24:16, and this run was cancelled 14 s later at 18:24:30. The per-file timeout (--timeout-per-file 1800 = 30 min) was not reached (test ran ~26.5 min).
• Note: prior scheduled runs of pr-test-amd-rocm720.yml (28289152816, 28281981946, 28273097445) skipped the stage-b partitions (only ran the extra suite), so there is no directly comparable green run of this exact partition job in the queryable window — but those runs concluded success.

Failure Cluster

HuggingFace weight-download hang (xet_get) during server warmup, followed by GHA concurrency cancellation

Facts (verified observations)

• Test command: run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd --auto-partition-id 1 --auto-partition-size 14 --timeout-per-file 1800 --continue-on-error (log).
• py-spy: active worker thread ThreadPoolExecutor-1_0 in xet_get (file_download.py:566); main scheduler thread waiting on the download future. sglang serve main process blocked in _wait_for_scheduler_ready (engine.py:1376).
• Final log line: ##[error]The operation was canceled. at 18:24:30 (log).
• The repeated [rank0]:W…triton_kernel_wrap.py:948 … ValueError: Incorrect number of arguments passed to kernel … expected [… 'USE_PDL'] lines are warnings (identify_mutated_tensors falls back to "assuming every input is mutated"); torch.compile continued (autotune + batch capture proceeded). These are NOT the cause of the cancellation.
• aiter image: v0.1.14rc1.dev240+g7d604afe5, no override.

Hypothesised Causes (with confidence)

• [HIGH] Job cancelled by GHA concurrency cancel-in-progress when the same-SHA re-run 28297867681 started 14 s earlier.
  • Supporting: re-run created 18:24:16, cancel at 18:24:30, same head_sha 592f6c84; test was still actively capturing batches (no error/timeout).
  • Disconfirming: (none found) — timeline is tight and consistent.
  • How to verify: inspect concurrency: block in .github/workflows/pr-test-amd-rocm720.yml; check the trigger source of run 28297867681.
• [HIGH] The ~24-min HuggingFace xet_get download stall is the real productivity/health problem — it consumed almost the entire window so the test never finished even before being cancelled.
  • Supporting: py-spy shows download in progress through 5 watchdog cycles; incomplete local snapshot forced a fresh download.
  • Disconfirming: download did eventually succeed (transient network/xet-backend slowness, not a hard failure).
  • How to verify: confirm the model is pre-baked/cached on the rocm720 runners; check HF xet backend reliability; consider HF_HUB_DISABLE_XET=1 or pre-seeding Qwen/Qwen1.5-MoE-A2.7B weights.

In-flight Fix Check

• In-flight fix: ❌ none found for the xet/download hang or this test. Searches: is:pr is:open xet download → only unrelated MoE-loading PRs (#29178, #28905, #28584); is:pr is:open test_torch_compile_moe → only #28636 (a DO-NOT-MERGE CI sandbox).

Suggested Triage Steps

• Treat this as infra/cancellation, not a test regression — no code bisect warranted for test_torch_compile_moe.py.
• Verify the runner image pre-bakes Qwen/Qwen1.5-MoE-A2.7B weights, or pre-seed the HF cache, to avoid the cold xet download path.
• Investigate HF xet backend reliability on the rocm720 MI325 runners; consider setting HF_HUB_DISABLE_XET=1 for CI if xet stalls recur.
• Confirm the concurrency policy is intended (re-runs cancelling in-flight same-SHA runs); if these scheduled stage-b partitions matter, ensure they are not being clobbered by overlapping triggers.
• Separately, file/track the USE_PDL triton-kernel-wrap warning (shared-experts fused kernel signature mismatch under torch.compile) — benign here but worth confirming it doesn't silently disable a fast path.

Status

• Persistence: Single cancelled run of this exact partition job; same SHA 592f6c84 was immediately re-run. No comparable green/red history for this partition in the queryable scheduled window (prior scheduled runs skipped stage-b).
• Scope: 1 job, 1 test file (test_torch_compile_moe.py), pr-test-amd-rocm720.yml only.
• Blocked work: None confirmed — the cancellation appears to be a concurrency artifact; the underlying download stall risks slow/flaky CI but did not produce a test failure.
• In-flight fix: ❌ none found — infra investigation (cache pre-seed / xet backend) rather than a code PR.

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 9) — failed step(s): Run test

Commit Info

• sglang: 592f6c849bf972370782d915c7a6528c158c2c80
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (v0.1.14rc1.dev240+g7d604afe5; source: dev/patched image version preserved — [CI-AITER-CHECK] reports "Dev/patched version detected … → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
—	—	No test failed in this job. Step 6 "Run test" was cancelled mid-execution.	Log line

This job (partition 9 of 14) ran 10 test files. Files 0–5 completed and passed (test_multi_tokenizer.py, test_lora_eviction.py, test_reasoning.py, test_anthropic_tool_use.py, test_pytorch_sampling_backend.py, test_radix_cache_hit.py). The runner emitted ##[error]The operation was canceled. at 18:24:30 just after Begin (6/10) for test_request_length_validation.py.

Failure Summary

This job did not fail on a test — it was cancelled (conclusion: cancelled, step 6 cancelled) while running, due to a workflow-level fail-fast cascade. The overall run 28297015673 concluded cancelled with 16 jobs success, 10 failures, and 23 cancelled. The 10 sibling failures (e.g. stage-b partitions 6, 7, 10) tripped fail-fast, which cancelled the remaining in-flight jobs including this one.

Regression Status

Flaky / transient (infrastructure-orchestration) — not a regression in partition 9.

Recent pr-test-amd-rocm720.yml scheduled runs on main (completed only):

Date	Run	Head SHA	Run Conclusion	Partition 9
Jun 27 18:24	28297867681	592f6c849b	✅ success	✅ Passed (re-run, same SHA)
Jun 27 17:48	28297015673	592f6c849b	❌ cancelled	⚠️ Cancelled (this run)
Jun 27 12:26	28289152816	3306233961	✅ success	✅ Passed
Jun 27 07:03	28281981946	9214b9338f	✅ success	✅ Passed
Jun 27 00:37	28273097445	13b5bd962a	✅ success	✅ Passed

The immediate re-run at the identical head SHA passed, which is conclusive evidence that neither the cancellation nor the underlying sibling failures are a code regression at this commit.

Failure Cluster

Fail-fast cancellation cascade (job cancelled mid-run, no test failure of its own).

Facts (verified observations)

• [FACT] Job 83838610292 conclusion: cancelled; step 6 "Run test" cancelled, all other steps success.
• [FACT] Log line 9007: ##[error]The operation was canceled. at 18:24:30, immediately after Begin (6/10) (line 8990).
• [FACT] Test files 0–5 completed with passing results (e.g. line 8920 OK, line 8979 Overall cache_hit_rate=0.8502) before cancellation.
• [FACT] Run 28297015673 overall cancelled: 16 success / 10 failure / 23 cancelled out of 49 jobs.
• [FACT] Sibling failure signatures: partition 6 Server process exited with code -9 + TypeError: Missing required argument 'input_embeds'; partition 7 test_vlm_perf_5090.py server -9 / retry exhaustion; partition 10 test_srt_engine.py exit -9, plus test_llada2_mini.py, test_json_mode.py, test_start_profile.py.
• [FACT] Re-run 28297867681 at same head SHA 592f6c849b concluded success.
• [FACT] AITER dev version v0.1.14rc1.dev240+g7d604afe5 preserved (no rebuild).

Hypothesised Causes (with confidence)

• [FACT] This job was cancelled by the workflow's fail-fast policy after sibling jobs failed; it had no test failure of its own.
  • Supporting: step-6 cancellation timestamp aligns with sibling job failures; run conclusion cancelled; this job's executed tests all passed.
  • Disconfirming: (none — this is directly observable).
• [LOW] The triggering sibling failures (server exited with code -9 = SIGKILL, likely OOM/GPU memory) are transient infra/flakes rather than a code regression.
  • Supporting: full re-run at the identical SHA passed; multiple -9 server-death signatures across unrelated tests (perf, engine, compressed-tensors) point to host/GPU resource pressure, not a single code path.
  • Disconfirming: the TypeError: Missing required argument 'input_embeds' in partition 6 looks code-shaped rather than resource-shaped — but it did not recur on the passing re-run, so it too cleared. (Sibling root causes are out of scope for this partition-9 analysis.)
  • How to verify: already verified — re-run 28297867681 green at same SHA.

In-flight Fix Check

Not applicable — this job has no failure to fix (cancelled, and the SHA passes on re-run). No PR search warranted for partition 9. The sibling -9/input_embeds symptoms, if they recur, would warrant separate tracking.

Suggested Triage Steps

• No action needed for this job/partition: the cancellation was a fail-fast side effect and the same commit passed on re-run 28297867681.
• If sibling Server process exited with code -9 failures recur across runs, investigate them independently (GPU memory pressure / runner health on linux-mi325-1gpu-sglang); they are the true trigger, not partition 9.
• Optionally consider whether fail-fast should cancel long-running sibling partitions, since it discards otherwise-passing signal (partition 9 was healthy through file 5).

Status

• Persistence: One-off. Cancelled in run 28297015673; same SHA passed on re-run and 3 prior scheduled runs are green for this partition.
• Scope: This job had zero test failures; it was 1 of 23 jobs cancelled by fail-fast in a single run.
• Blocked work: None — re-run cleared the run.
• In-flight fix: N/A (no failure attributable to this job).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10) — failed step(s): Run test

Commit Info

• sglang: 592f6c849bf972370782d915c7a6528c158c2c80
• aiter: 7d604afe5fa7efba63c0dce323b95d9daf2db112 (source: Dockerfile default; dev image v0.1.14rc1.dev240+g7d604afe5 detected and preserved, no override). This is not a scout run, so no Baseline A/B check applies.

Failed Tests

Test File	Test Function	Error	Log
test/registered/dllm/test_llada2_mini.py	setUpClass (server launch)	HfHubHTTPError: 429 Too Many Requests on inclusionAI/LLaDA2.0-mini	Log
test/registered/core/test_srt_engine.py	engine init (exit code -9 / SIGKILL)	HfHubHTTPError: 429 Too Many Requests on meta-llama/Llama-3.2-1B-Instruct	Log
test/registered/openai_server/features/test_json_mode.py	setUpClass (line 113)	HfHubHTTPError: 429 Too Many Requests on meta-llama/Llama-3.2-1B-Instruct	Log
test/registered/profiling/test_start_profile.py	test_start_profile_2	Scheduler watchdog timeout (300s) → server crash → ConnectionError: Connection refused	Log

Failure Summary

The shard ran 9 test files; 5 passed and 4 failed. Three of the four failures (test_llada2_mini, test_srt_engine, test_json_mode) crashed during server/engine startup because the HuggingFace Hub API returned 429 Too Many Requests while resolving model repos. The fourth (test_start_profile) failed independently: the scheduler hit a 300 s watchdog timeout while stuck in a CUDA graph replay, killing the server so the test's HTTP request was refused. The overall run was later cancelled (schedule overlap), but this job had already run to failure.

Regression Status

Flaky / Infrastructure — not a code regression.

The three 429 failures are environmental: the log states you hit the quota of 3000 api requests per 5 minutes period (current: team), caused by many parallel CI jobs hitting the HF API simultaneously. The profiling watchdog hang is an intermittent GPU/replay stall.

History for this exact shard (stage-b-test-1-gpu-small-amd-rocm720 (… , 10), branch main, event=schedule, completed runs only) is sparse because the stage-b matrix only expands when change-detection triggers it — it is skipped in most scheduled runs:

Date	Run	Shard-10 Status	Note
Jun 27 17:48	28297015673	❌ Failed	this job (3× HF 429 + 1× watchdog hang)
Jun 26 18:04	28256293905	⏹ Cancelled	shard expanded but run cancelled — not a usable baseline
Jun 27 (12:26 / 07:03 / 00:37)	28289152816 etc.	⏭ Skipped	stage-b small not expanded (run conclusion success)

No clean green run is queryable at this exact shard granularity in the window, but test_srt_engine.py and test_json_mode.py are core tests that pass routinely; the 429 symptom is unrelated to any sglang code path.

Failure Cluster

• HuggingFace Hub 429 rate-limit during server startup — test_llada2_mini, test_srt_engine, test_json_mode (3 tests).
• Scheduler watchdog timeout during CUDA graph replay — test_start_profile (1 test).

Facts (verified observations)

• httpx.HTTPStatusError: Client error '429 Too Many Requests' raised inside run_scheduler_process → ModelRunner.load_model → download_weights_from_hf → hf_file_system.ls for three distinct tests (log lines 4170, 4286, 4405).
• HF message verbatim: 429 Too Many Requests: you have reached your 'api' rate limit … you hit the quota of 3000 api requests per 5 minutes period … (current: team).
• test_srt_engine.py returned exit code -9 (SIGKILL) after the 429 SIGQUIT cascade; test_llada2_mini.py and test_json_mode.py returned exit code 1.
• test_start_profile: [18:13:52] Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=False); py-spy dump shows the scheduler MainThread blocked in pthread_mutex_lock (libamdhip64.so.7.2.70200) under at::cuda::CUDAGraph::replay ← full_cuda_graph_backend.py:129 ← decode_cuda_graph_runner.py:991 ← model_runner.py:3048. Then SIGQUIT received … one child failed, server exits, test gets Connection refused on 127.0.0.1:11000/start_profile and fails after retries.
• aiter unchanged from Dockerfile default; [CI-AITER-CHECK] shows Dev/patched version detected … → skipping rebuild.

Hypothesised Causes

• [FACT] The three startup failures are caused by HuggingFace API 429 rate limiting, not sglang/aiter code.
  • Supporting: explicit 429 Too Many Requests + quota message in the log for all three; failures occur in HF download/resolve calls before any model code runs.
  • Disconfirming: (none) — the error is self-evident in the log.
  • How to verify: re-run the shard; with HF quota recovered (or full offline cache) these tests pass.
• [LOW] The test_start_profile watchdog hang is a transient GPU/CUDA-graph-replay stall on the MI325 runner.
  • Supporting: stack stuck in HIP runtime pthread_mutex_lock under CUDAGraph::replay; no Python-level exception; isolated to one test.
  • Disconfirming: only one occurrence at this shard in the window — cannot distinguish a real replay deadlock from a one-off runner/GPU stall without a repeat. Unrelated to the 429 cluster.
  • How to verify: re-run test/registered/profiling/test_start_profile.py on an MI325 runner; if it recurs, capture a coredump with CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1 and inspect the replayed decode graph.

In-flight Fix Check

• 429 rate limit: In-flight fix: ❌ none found — this is an HF account-quota / CI-concurrency infra issue, not fixable by a source PR (keyword search huggingface rate limit returned only unrelated PRs). Mitigation lives in CI (offline model cache / staggering), not code.
• Watchdog/CUDA-graph hang: In-flight fix: ❌ none found — single occurrence, no signature to match yet.

Suggested Triage Steps

• Re-run the shard once HF quota resets; expect the three 429 tests to pass — confirms infra root cause.
• For the 429 cluster: pre-warm / fully cache meta-llama/Llama-3.2-1B-Instruct and inclusionAI/LLaDA2.0-mini in the CI image (or force HF_HUB_OFFLINE=1 before repo resolution) so startup never hits the HF API; consider staggering parallel job starts to stay under the 3000 req / 5 min team quota.
• For test_start_profile: re-run in isolation on MI325; if the replay hang recurs, bisect decode CUDA-graph paths (full_cuda_graph_backend.py, decode_cuda_graph_runner.py) and capture a coredump. Treat as a separate flake from the 429 cluster.

Status

• Persistence: Single observed failing run at this shard; the immediately prior expansion (28256293905) was cancelled, so no multi-run trend. Both symptoms are intermittent.
• Scope: One job/shard (stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 10)). The 429 quota issue can affect any concurrently-running AMD/NVIDIA CI job hitting the HF API.
• Blocked work: None code-side; failures are environmental. Run overall conclusion was cancelled.
• In-flight fix: ❌ none for either cluster — both are infra/flake, not code regressions.

stage-c-test-4-gpu-amd-rocm720 — failed step(s): N/A

Commit Info

• sglang: 592f6c84
• aiter: N/A (the target job never ran; no [CI-AITER-CHECK] markers were emitted)

Failed Tests

The target job stage-c-test-4-gpu-amd-rocm720 (job 83840854947) did not run any tests. It was cancelled at 18:24:32Z with started_at == completed_at and zero executed steps. Per the workflow (pr-test-amd-rocm720.yml:873) it needs: [check-changes, stage-b-test-1-gpu-small-amd-rocm720, stage-b-test-2-gpu-large-amd-rocm720]; an upstream stage-b failure fast-failed the whole run before stage-c could start.

The actual upstream failures that executed (root of the cascade):

Job	Test File	Test Function	Error	Log
stage-a-test-1-gpu-small	test/registered/core/test_basic_sanity_eagle3.py, test_basic_sanity.py	(suite)	httpx.HTTPStatusError: 429 Too Many Requests for huggingface.co/.../Llama-3.1-8B-Instruct	Log
stage-b-test-1-gpu-small (7)	test/registered/perf/test_vlm_perf_5090.py	(suite)	429 Too Many Requests for Qwen/Qwen2.5-VL-3B-Instruct → retry() exceed maximum number of retries	Log
extra-a-test-1-gpu-large	test_streaming_session_extra.py, test_fp8kv_triton.py	(suite)	429 Too Many Requests / OSError: We couldn't connect to 'https://huggingface.co'	Log
multimodal-gen-test-1-gpu (2)	test_generate_zimage_turbo_cli.py	test_single_gpu, test_output_file_path_alias	ValueError: Downloaded model ... is still incomplete after forced re-download (Z-Image-Turbo)	Log

Failure Summary

The job under analysis (stage-c-test-4-gpu-amd-rocm720) was cancelled, not failed — it executed no steps and ran no tests. The entire PR Test ROCm 7.2 (AMD) scheduled run was cancelled by a fast-fail cascade after multiple upstream jobs failed. Those upstream failures are overwhelmingly HuggingFace 429 Too Many Requests rate-limiting during model metadata/weight downloads, plus one incomplete model download — i.e. infrastructure/network issues, not a code defect in this commit.

Regression Status

Infrastructure issue (not a code regression). The cascade cancellation is environmental.

Recent history of stage-c-test-4-gpu-amd-rocm720 in workflow pr-test-amd-rocm720.yml (same branch main, event=schedule, completed runs only):

Date	Run	Job Status	Notes
Jun 27 18:24	28297867681	⏭️ Skipped	Same head_sha 592f6c849b; re-triggered run succeeded (stage-b/c skipped by check-changes)
Jun 27 17:48	28297015673	🚫 Cancelled	This run — fast-fail cascade from stage-b
Jun 27 12:26	28289152816	⏭️ Skipped	gated off by check-changes
Jun 27 07:03	28281981946	⏭️ Skipped	gated off by check-changes
Jun 27 00:37	28273097445	⏭️ Skipped	gated off by check-changes

This job is almost always skipped in scheduled runs (gated by check-changes). There is no recent run where it actually executed and failed, so there is no test-level regression to attribute. Notably, the immediately re-triggered run on the identical commit completed success.

Failure Cluster

Fast-fail cascade cancellation (no code executed) — root: HuggingFace 429 rate-limiting on model download

Facts (verified observations)

• The target job recorded conclusion: cancelled, started_at == completed_at == 2026-06-27T18:24:32Z, and an empty steps array — it ran nothing (verified via job API).
• pr-test-amd-rocm720.yml:873: stage-c-test-4-gpu-amd-rocm720 depends on the two stage-b jobs; stage-b shards 6/7/10 (83838610271, 83838610285, 83838610294) recorded failure before the cancellation.
• Across the four executed failing jobs sampled, the recurring error is httpx.HTTPStatusError: Client error '429 Too Many Requests' against huggingface.co (e.g. Llama-3.1-8B-Instruct, Qwen2.5-VL-3B-Instruct, Meta-Llama-3-8B-Instruct-FP8-KV).
• Each job log also shows an early local docker-registry pull timeout: Error response from daemon: Get "https://10.245.143.50:5000/v2/": context deadline exceeded (recovered via fallback; not the failure cause).
• The same head_sha 592f6c849b produced a success run 28297867681 ~36 min later — strong disconfirmation of any code-level cause.

Hypothesised Causes (with confidence)

• [FACT] The target job's "failure" is a fast-fail cascade cancellation — it executed no code. Verified from empty steps + needs: graph.
• [HIGH] The root trigger is HuggingFace 429 rate-limiting causing model-download failures across stage-a/stage-b/extra-a jobs.
  • Supporting: identical 429 Too Many Requests error string in 3 independent jobs against 4 different model repos; one job hit OSError: couldn't connect to huggingface.co.
  • Disconfirming: one multimodal job (83838610263) failed on an incomplete Z-Image-Turbo download (ValueError), a slightly different symptom — but still a download/infra failure, not code.
  • How to verify: re-run the run; HF 429s are transient and should clear (the same-SHA re-trigger already passed/skipped).
• [LOW] Commit 592f6c84 (HEAD) caused a real test break.
  • Disconfirming: same SHA succeeded on re-run; failing tests fail at model-download time, before any commit-touched code path executes.

In-flight Fix Check

• In-flight fix: ❌ not applicable — this is environmental (HF rate-limiting / runner network), not a code bug requiring a PR. Searched open PRs for huggingface 429 keyword; no relevant code fix exists because none is warranted.

Suggested Triage Steps

• Re-run the scheduled run — HF 429 is transient; the same-SHA re-trigger 28297867681 already cleared.
• Consider routing AMD-CI model loads through a local HF mirror / HF_HUB_OFFLINE cache to remove the dependency on huggingface.co rate limits during scheduled bursts (multiple parallel jobs hammering HF triggered the 429s).
• Investigate the local docker registry timeout (10.245.143.50:5000) seen at job start, as a separate runner-network reliability item.
• For the Z-Image-Turbo incomplete-download case, verify the model cache integrity on the multimodal runner.
• No code action against 592f6c84 is warranted.

Status

• Persistence: One cancelled run; not recurring at the test level (job is normally skipped, and the same-SHA re-run did not reproduce).
• Scope: Whole pr-test-amd-rocm720.yml run cancelled; the target stage-c-4-gpu job and ~25 sibling jobs were cancelled by the cascade. ~5 jobs recorded genuine failure, all download/infra-related.
• Blocked work: stage-c (4-GPU) ROCm 7.2 signal was not produced for this run.
• In-flight fix: ❌ not applicable — environmental, resolved by re-run.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 00:58 UTC)

[amd-bot]

pr-test-amd.yml — 35 failure(s)

Run started (UTC): 2026-06-27 02:28 UTC
Last scanned (UTC): 2026-06-28 02:40 UTC
Commits: sglang da802dd

Summary

Counts: 35 failures · 11 clusters · 5 carrying over (≥3 days) · 3 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Conv3D cat/pad CUDA-only on ROCm (Wan/MOVA VAE)	multimodal-gen-test-2-gpu-amd (2gpu,0)	test/server/test_server_2_gpu.py	test_diffusion_generation[wan*]	16 jobs; since Jun 26 (~2d)	[HIGH] #29281 5996b54b broke None-guarded ROCm fallback
2	TokenizedGenerateReqInput missing input_embeds/token_type_ids	stage-b-test-1-gpu-small-amd (shard 6)	test/registered/utils/test_type_based_dispatcher.py	test_type_dispatcher_e2e_performance	5 jobs; never-passed ≥3d	[HIGH] test omits no-default fields of io_struct.py:776,779
3	Diffusion server fixture timeout → port-5555 --strict-ports cascade	multimodal-gen-test-1-gpu-amd (1gpu,0)	test/server/test_server_1_gpu.py	test_diffusion_generation (setup)	5 jobs; recurring (infra)	[HIGH] HF model-download stall + teardown leaves port 5555/GPU mem held
4	ModelOpt FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED	multimodal-gen-test-2-gpu-amd (2gpu,1)	test/server/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8*]	4 jobs (co-located w/ R1)	[MEDIUM] _scaled_mm fp8 path unsupported on ROCm
5	update_weights_from_disk HTTP 500 (inplace inference-tensor) + fixture missing transformer dir	multimodal-gen-test-1-gpu-amd (1gpu,3)	test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::*	3 jobs; since Jun 27	[MEDIUM] introduced already-red by #28762
6	GPU mem access fault during DeepSeek-R1 MXFP4 MTP prefill	stage-c-test-large-8-gpu-amd-mi35x (8,0)	test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py	TestDeepseekR1MXFP4MTP.test_a_gsm8k	1 job; new today	[no candidate ≥ MEDIUM] — 3-commit window all CUDA-gated/no-op on AMD
7	DeepSeek-V3.2 gsm8k HSA out-of-resources abort	stage-c-test-large-8-gpu-amd (8gpu,1)	test/registered/amd/test_deepseek_v32_basic.py	TestDeepseekV32TP.test_a_gsm8k	1 job; new today (1/6)	[no candidate ≥ MEDIUM] — likely flake/node OOM
8	Qwen3-Coder-Next decode hang → watchdog → conn refused	stage-c-test-large-8-gpu-amd-mi35x (8,1)	test/registered/amd/test_qwen3_coder_next_8gpu.py	TestQwen3CoderNext.test_bs_1_speed	1 job; new today	[no candidate ≥ MEDIUM] — GPU stream stall mid-decode
9	Kimi-K2.5-MXFP4 startup hang during MoE load (exit -9)	stage-c-test-large-8-gpu-amd-mi35x (8,0)	test/registered/amd/test_kimi_k25_mxfp4_bcg_mi35x.py	test_bcg_gsm8k	1 job; flaky 1/6	[MEDIUM] multithread-load HIP mutex contention
10	HF cache file-lock timeout during MOE load	stage-b-test-1-gpu-small-amd-mi35x	test/registered/quant/test_quark_mxfp4.py	TestOnlineQuantizationMemoryLoadMOE.setUpClass	1 job; new (infra)	[MEDIUM] HF .lock contention (Qwen3-30B-A3B)
11	MORI build failure — corrupt libabsl_base.so	multimodal-gen-test-1-gpu-amd (1gpu,1)	N/A (build)	N/A	1 job; transient 1/4	[HIGH] corrupt system Abseil lib (invalid ELF)

Cluster: Conv3D cat/pad CUDA-only on ROCm (Wan/MOVA VAE decode)

• Affected (16 jobs): multimodal-gen 1-/2-GPU shards across all 4 runs — e.g. 2gpu,0 r28297966351, 2gpu,1, 2gpu,0 r28289294173, 1gpu,0, 1gpu,1, 1gpu,2 r28282108241, 2gpu,0 r28273478888, 2gpu,0 r28306914695. All test_diffusion_generation[wan*/mova*].
• Shared facts: identical top frame parallel_conv.py:57 fused_causal_conv3d_cat_pad → RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA; only Wan/MOVA params (use wanvae.py) fail; aiter unchanged. Pass→fail boundary at 781537b61d→5996b54b.
• Top hypothesis: [HIGH] #29281 made fused_causal_conv3d_cat_pad a module-level fn (never None), so on ROCm (x.is_cuda==True) _can_fuse_* routes into it and raises instead of the prior torch.cat+F.pad fallback. Disconfirming: none found.
• In-flight fix: ❌ none found.
• Suggested triage: revert 5996b54b on a branch and re-run Wan params on MI325 ROCm; or gate fused dispatch on current_platform.is_cuda().

Cluster: TokenizedGenerateReqInput missing required field

• Affected (5 jobs): stage-b-1gpu-small shard 6 every run — r28297966351, r28289294173, r28282108241, r28273478888, r28306914695.
• Shared facts: TypeError: Missing required argument 'input_embeds' at test_type_based_dispatcher.py:124; never-passed back to Jun 25; #28688 only changed error wording (dataclass→msgspec), bug predates it.
• Top hypothesis: [HIGH] test never updated when input_embeds/token_type_ids became no-default fields. Disconfirming: none.
• In-flight fix: ✅ #29376 (open since 2026-06-26) — chase reviewers, do NOT duplicate.

Cluster: Diffusion port-5555 --strict-ports cascade

• Affected (5 jobs): 1gpu,0 r28297966351, 1gpu,1, 1gpu,0 r28306914695, 1gpu,1, 2gpu,1.
• Shared facts: first test hits TimeoutError: Server not ready within 1200.0s during incomplete HF download (Qwen-Image-Edit / Cosmos3-Nano); remaining tests Scheduler port 5555 is unavailable + GPU memory not fully cleared (7.76 GB).
• Top hypothesis: [HIGH] model-download race + teardown not reaping scheduler child. Disconfirming: none.
• In-flight fix: ❌ none. Triage: pre-warm HF cache; harden fixture teardown (kill_process_tree).

Cluster: ModelOpt FP8 GEMM HIPBLAS_STATUS_NOT_SUPPORTED

• Affected (4 jobs, co-located with Conv3D): 2gpu,1 r28297966351, 2gpu,1 r28289294173, 2gpu,1 r28282108241, 2gpu,1 r28273478888. Only flux2_modelopt_fp8*.
• Top hypothesis: [MEDIUM] fp8 _scaled_mm GEMM unsupported on ROCm/hipBLAS. In-flight fix: ❌ none.

Cluster: update_weights_from_disk HTTP 500 + fixture bug

• Affected (3 jobs): 1gpu,3 r28297966351, 1gpu,3 r28289294173, 1gpu,3 r28306914695.
• Shared facts: 5× HTTP 500 Inplace update to inference tensor outside InferenceMode; 4× setup error No weights dir for transformer (FLUX). Test introduced already-red by #28762.
• Top hypothesis: [MEDIUM] #28762 exposed/introduced inference-tensor write path + a fixture clone bug. Disconfirming: #28762 did not modify the linear.py:575 crash site (may be pre-existing product behavior exposed by the new test). In-flight fix: ❌ none.

Cluster: stage-c large-model GPU crash/hang family (4 singletons, new today)

• DeepSeek-R1 MXFP4 MTP mem fault 8,0: Memory access fault node-7/8 during MTP prefill; non-MTP sibling passed (0.947). Window 592f6c84..da802ddc = 3 commits, all CUDA-gated/no-op on AMD → No hypothesis ≥ MEDIUM. Triage: re-run once (flake test), else bisect with CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1.
• DeepSeek-V3.2 HSA OOM 8gpu,1: SIGABRT at ~99% gsm8k; passed prior 5 runs (1/6). Likely flake/node OOM.
• Qwen3-Coder-Next decode hang 8,1: stream stall mid-decode → watchdog SIGQUIT; passed prior 3 runs.
• Kimi-K2.5-MXFP4 startup hang 8,0: all loader threads parked in pthread_mutex_lock/libamdhip64 → exit -9; 1/6 flaky.
• In-flight fix: ❌ none for any.

Cluster: HF cache file-lock timeout (Job 10) & MORI build failure (Job 11)

• HF lock stage-b-mi35x: two procs blocked on .lock for Qwen3-30B-A3B until 1200s hard-timeout; passed prior 4 runs (infra). ❌ no fix.
• MORI 1gpu,1: libabsl_base.so.20210324: invalid ELF header (exit 127) during gtest discovery; 1/4 transient container corruption. ❌ no fix.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Merge in-flight #29376, re-run stage-b shard 6	Cluster 2 (5 jobs)	Fix exists, ≥3-day blocker, zero-investigation
2	Revert/forward-fix 5996b54b (#29281); re-run Wan diffusion	Cluster 1 (16 jobs)	Largest cluster, HIGH-confidence single commit
3	Pre-warm HF cache + harden fixture teardown	Cluster 3 (5 jobs)	Infra; unblocks diffusion server signal
4	Review #28762 weights_updater.py inference-mode path; fix fixture	Cluster 5 (3 jobs)	Newly-enabled, never-green on AMD
5	Re-run stage-c new-today failures once; bisect only if reproducible	Clusters 6–9	Likely flakes; cheap disconfirmation before investing

---

Job	Failed Steps	Started
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-27T18:42
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-27T20:15
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-06-27T18:36
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-27T20:16
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-06-27T19:26
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-06-27T18:28
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-27T14:32
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-27T18:53
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-27T13:21
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-06-27T14:03
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-27T13:05
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-06-27T14:04
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-27T13:18
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-27T08:24
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-27T09:00
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-27T08:15
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Install dependencies	2026-06-27T10:29
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-27T09:27
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-27T07:09
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-27T03:55
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-27T08:21
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-27T02:39
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-27T02:28
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 2)	Run diffusion server tests (1-GPU)	2026-06-27T02:41
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-06-27T02:38
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-27T03:39
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3)	Run diffusion server tests (1-GPU)	2026-06-28T00:55
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0)	Run diffusion server tests (2-GPU)	2026-06-28T00:55
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 1)	Run test	2026-06-28T00:54
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6)	Run test	2026-06-28T01:11
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 1)	Run test	2026-06-28T00:55
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-06-28T00:55
multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1)	Run diffusion server tests (1-GPU)	2026-06-28T00:55
multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1)	Run diffusion server tests (2-GPU)	2026-06-28T01:15
stage-c-test-large-8-gpu-amd-mi35x (linux-mi35x-gpu-8, 0)	Run test	2026-06-28T00:54

Per-Job Analysis

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (source: dev image preserved — Dev/patched version detected: v0.1.14rc1.dev240+g7d604afe5 → skipping rebuild, per [CI-AITER-CHECK] markers)

Failed Tests

All 7 errors are in the same file/class, all at setup (the server fixture never became ready), so no test body executed.

Test File	Test Function	Error	Log
sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[qwen_image_edit_2511_ti2i]	TimeoutError: Server not ready within 1200.0s (model still downloading)	Log
…/test_server_1_gpu.py	…[joyai_image_edit_ti2i]	RuntimeError: Server exited early (code 1) → Scheduler port 5555 is unavailable and --strict-ports is enabled	Log
…/test_server_1_gpu.py	…[cosmos3_nano_t2v]	same port-5555 cascade	Log
…/test_server_1_gpu.py	…[wan2_1_t2v_1.3b_teacache_enabled]	same port-5555 cascade	Log
…/test_server_1_gpu.py	…[wan2_1_t2v_1.3b_frame_interp_2x]	same port-5555 cascade	Log
…/test_server_1_gpu.py	…[wan2_1_t2v_1.3b_frame_interp_2x_upscaling_4x]	same port-5555 cascade	Log
…/test_server_1_gpu.py	…[fast_hunyuan_video]	same port-5555 cascade	Log

The job retried (attempt 2/7) and produced the identical 7 errors (521.93s); session 1 took 2538.43s (0:42:18).

Failure Summary

The 1-GPU diffusion server test job (shard 0) errored on all 7 selected test_diffusion_generation parametrizations. The first test (qwen_image_edit_2511_ti2i) hit TimeoutError: Server not ready within 1200.0s because the model snapshot for Qwen/Qwen-Image-Edit-2511 was incomplete in the HF cache and was still downloading (Fetching 35 files: 46%|… 16/35) when the 1200 s deadline expired. The remaining 6 tests then failed at setup with RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled / Server exited early (code 1), with the fixture warning ROCm GPU memory not fully cleared after 60s (used: 7.76 GB) — i.e. a lingering process from the timed-out first server held port 5555 and GPU memory, cascading into every subsequent server start.

Regression Status

Known recurring failure (test file), but new failure shape today.

Recent history of test_server_1_gpu.py (TestDiffusionServerOneGpu::test_diffusion_generation) in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27 18:28	28297966351	shard 0	❌ Failed	all 7 (setup ERROR)	Server not ready 1200s + port-5555 cascade
Jun 27 12:32	28289294173	shard 0	❌ Failed	3× wan2_1_* (real FAILED; 9 passed)	server started, video tests failed
Jun 27 07:09	28282108241	shard 0	❌ Failed	(shard 0 failed)	—
Jun 27 00:51	28273478888	shard 0	❌ Failed	(shard 0 failed)	—
Jun 26 18:40	28258143566	shard 0	❌ Failed	(shard 0 failed)	—

Shard 0 of this job has failed in every completed scheduled run in the queryable window (≥ Jun 26 18:40), so test_server_1_gpu.py is persistently red on this shard. However, the prior run (28289294173) had only 3 genuine test failures (the wan2_1 video variants — server started, 9 passed), whereas today all 7 errored at setup because the server never started. Today's specific symptom (download timeout → port cascade) is therefore a new/distinct failure mode layered on top of an already-red test file.

Failure Cluster

Diffusion server fixture setup failure: HF model-download timeout on first test, then port-5555/GPU-memory cleanup cascade on the rest.

Facts (verified observations)

• First test error verbatim: TimeoutError: Server not ready within 1200.0s. raised at test_server_utils.py:492 via _wait_for_ready (test_server_common.py:191 → test_server_utils.py:449).
• At timeout the server log tail showed an in-progress HF download: Model found in cache but incomplete, will download from HF Hub → Downloading model snapshot … for Qwen/Qwen-Image-Edit-2511 (attempt 1/5) → Fetching 35 files: 46%|… 16/35 [00:11<00:15, 1.20it/s].
• Tests 2–7 error verbatim: RuntimeError: Server exited early (code 1). with server stderr RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled. raised at server_args.py:748 (_require_port ← _adjust_network_ports server_args.py:776).
• Fixture cleanup warning before each cascade error: ROCm GPU memory not fully cleared after 60s (used: 7.76 GB). Proceeding anyway - this may cause OOM. (test_server_utils.py:360).
• _wait_for_ready deadline is self.wait_deadline (1200 s); on early process exit it raises immediately, on deadline it raises TimeoutError (test_server_utils.py:470-492).
• Retry attempt 2 reproduced the identical 7 setup errors in 521 s; the GPU memory from session 1 was never freed.
• FFmpeg / libavutil.so.* OSError lines in the log are non-fatal — they precede PASSED results elsewhere in the same job and are unrelated to these errors.
• AITER ran as a preserved dev build v0.1.14rc1.dev240+g7d604afe5; no override. Head commit 592f6c849b is a [minimax-m3] mem-cache split, unrelated to multimodal_gen.

Hypothesised Causes (with confidence)

• [HIGH] First failure is an environment/infrastructure issue: the Qwen/Qwen-Image-Edit-2511 model was incomplete in the runner cache and the HF download did not finish within the 1200 s wait_deadline.
  • Supporting: log shows Model found in cache but incomplete, will download and the download was at 16/35 files (46%, ~1.2 it/s) when TimeoutError fired; the error is a timeout, not a code exception in server init.
  • Disconfirming: none found — the stack trace bottoms out in the test harness wait loop, not in sglang server code.
  • How to verify: pre-warm / verify the HF cache for Qwen/Qwen-Image-Edit-2511 on the runner, or raise wait_deadline for first-download cases; re-run shard 0.
• [HIGH] Failures 2–7 are a cascade from failure 1: the timed-out first server process was not fully torn down, leaving port 5555 bound and ~7.76 GB GPU memory in use, so every later server start aborts via the --strict-ports check.
  • Supporting: every cascade error pairs Scheduler port 5555 is unavailable with the preceding GPU memory not fully cleared (used: 7.76 GB) warning; retry session reproduced it from the very first test because memory stayed occupied.
  • Disconfirming: none — the port/GPU-not-cleared evidence is direct in the log.
  • How to verify: confirm the fixture's teardown/kill_process_tree actually reaps the scheduler child before the next test; check whether the 60 s GPU-clear wait is sufficient.
• [LOW] Pre-existing instability of the wan2_1 video variants (the 3 tests that genuinely FAILED in the prior run) may also be contributing to test-file redness over time.
  • Supporting: prior run 28289294173 FAILED exactly the 3 wan2_1_* variants while the rest passed.
  • Disconfirming: in today's run those tests never executed (errored at setup), so this is orthogonal to today's root cause.
  • How to verify: once the server-startup/port issue is fixed, re-run and check whether wan2_1_* still fails on its own.

This is not attributable to the head commit or to aiter; treat it as an infra/test-harness issue.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for strict-ports / test_server_1_gpu / diffusion port (GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+strict-ports+OR+test_server_1_gpu+OR+diffusion+port, 505 broad matches, none addressing the model-download timeout or port-5555 teardown cascade).

Suggested Triage Steps

• Pre-stage/verify the HF cache for Qwen/Qwen-Image-Edit-2511 (35 files) on the MI325 1-GPU runner so the first test does not race the 1200 s wait_deadline; alternatively allow a longer deadline on first-download.
• Inspect the fixture teardown path (kill_process_tree, the 60 s GPU-memory-clear loop at test_server_utils.py:~340-365) — ensure the scheduler child is reaped and port 5555 released before the next manager.start(), so a single slow/failed test does not cascade into the rest of the shard.
• Re-run shard 0 in isolation after the cache is warmed; if the 6 cascade errors disappear, confirm whether the wan2_1_* video variants (seen FAILED in the prior run) are a separate standing issue to track.
• Since session 2's retry inherited the stuck GPU state, consider a hard GPU/process reset between pytest retry attempts on ROCm.

Status

• Persistence: test_server_1_gpu.py shard 0 has been red in every completed scheduled run since at least Jun 26 18:40 UTC; today's download-timeout + port-cascade shape is new vs the prior run's 3 genuine wan2_1 test failures.
• Scope: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 0) in pr-test-amd.yml; both pytest attempts failed identically. Other 1-GPU shards (shard 2, shard 3) pass in recent runs.
• Blocked work: 1-GPU diffusion server CI signal on AMD for these 7 model parametrizations.
• In-flight fix: ❌ none found — needs a new infra/harness fix (cache warm-up + robust teardown).

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev/patched image preserved — matches Dockerfile default, no override)

Failed Tests

All in test/server/test_server_2_gpu.py (TestDiffusionServerTwoGpu::test_diffusion_generation), parametrized by model. Summary: 5 failed, 7 passed, 1 skipped, 12 deselected.

Test File	Test Function (param)	Error	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_2_t2v_a14b_teacache_2gpu]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_2_t2v_a14b_lora_2gpu]	same	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_t2v_1_3b_cache_dit_sp_only_2gpu]	same	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[mova_360p_ring1_uly2]	same	Log
sglang/multimodal_gen/test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_i2v_14b_lora_2gpu]	same	Log

Failure Summary

Five Wan-family diffusion server tests failed during VAE decode. Each request aborts with RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA, raised from parallel_conv.py:57 in the VAE decode path (wanvae.py → causal_conv3d_cat_pad). All five are Wan/MOVA models (which use wanvae.py); the 7 passing tests (flux, ltx, qwen_image, zimage, joy_echo) do not hit this code path.

Regression Status

New regression — first appeared 2026-06-26, started exactly when #29281 merged.

Recent history of test/server/test_server_2_gpu.py in job multimodal-gen-test-2-gpu-amd (…, 0) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Head SHA	Test File Status	Error
Jun 27 18:28	28297966351	592f6c84	❌ Failed	causal Conv3D on CUDA-only
Jun 27 12:32	28289294173	3306233961	❌ Failed	same
Jun 26 07:23	28223639512	5996b54b	❌ Failed (first)	same
Jun 26 00:54	28210138009	781537b61d	✅ Passed	—

Last passing: Jun 26 00:54 UTC (781537b61d). First failing: Jun 26 07:23 UTC (5996b54b) — the run whose head IS #29281. Persistent on every completed scheduled run since.

Failure Cluster

ROCm RuntimeError "causal Conv3D cat/pad fusion is only available on CUDA" in Wan VAE decode

Facts (verified observations)

• Stack trace (verbatim, run log line ~20:29:15): wanvae.py:1616 decode → wanvae.py:1331 forward → parallel_conv.py:641 forward → parallel_conv.py:140 causal_conv3d_cat_pad → parallel_conv.py:57 fused_causal_conv3d_cat_pad → RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA.
• Platform: log shows Automatically detected platform rocm and attention_backend: aiter — this is an AMD/ROCm runner.
• All 5 failing params are Wan/MOVA models (use wanvae.py); 7 passing are non-Wan (flux/ltx/qwen/zimage/joy_echo).
• aiter version unchanged (Dockerfile default dev image, no override) — not an aiter variable.
• Test-file-level A/B: shard 0 of this job was ✅ in run 28210138009 (pre-commit), ❌ in 28223639512 (commit-landing run, 203 occurrences of the error string).

Hypothesised Causes

• [HIGH] Commit 5996b54b (#29281, "Add diffusion causal Conv3D cat-pad CUDA fast path for Cosmos3") broke the ROCm fallback in parallel_conv.py.
  • Supporting: The diff added the exact raise RuntimeError("causal Conv3D cat/pad fusion is only available on CUDA") (parallel_conv.py:57), which is the verbatim failing error and the top stack frame. Before the commit, on non-CUDA fused_causal_conv3d_cat_pad was set to None, so the dispatch guard _can_fuse_causal_conv3d_cat_pad (line 110: if … or fused_causal_conv3d_cat_pad is None: return False) short-circuited to the safe torch.cat+F.pad fallback. After the commit, fused_causal_conv3d_cat_pad is a module-level function (never None), so on ROCm — where tensors report x.is_cuda == True — _can_fuse_... now returns True and routes into the function, which finds both _cuda and _triton kernels None and raises. Temporal: shard 0 passed in the run immediately before the commit and failed in the run that includes it, on the same test file with the same error.
  • Disconfirming evidence: (none found) — the code change, error string, stack frame, and pass→fail boundary all converge on this commit.
  • How to verify: git revert 5996b54b on a branch and re-run multimodal-gen-test-2-gpu-amd (Wan params) on an MI325 ROCm runner; or locally confirm that on ROCm _can_fuse_causal_conv3d_cat_pad no longer returns True (e.g. guard the fused dispatch on current_platform.is_cuda() rather than x.is_cuda).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for causal conv3d (4 unrelated VAE PRs) and conv3d ROCm (0 results). No existing fix; needs a new PR or investigation.

Suggested Triage Steps

• Revert 5996b54b (#29281) on a branch and re-run the Wan params of multimodal-gen-test-2-gpu-amd to confirm it restores green.
• If confirming a forward fix instead of revert: in parallel_conv.py, gate the fused-path dispatch on current_platform.is_cuda() (not x.is_cuda, which is True on ROCm), or restore a ROCm fallback inside fused_causal_conv3d_cat_pad so it returns the torch.cat+F.pad path instead of raising.
• Ping the #29281 author (Xiaoyu Zhang) — the CUDA fast-path PR did not preserve the prior None-guarded ROCm behavior.

Status

• Persistence: Failing since 2026-06-26 07:23 UTC (#29281 landing), every completed scheduled run since (~1.5 days, 4+ runs).
• Scope: multimodal-gen-test-2-gpu-amd shard 0 (and other shards show related failures); all Wan/MOVA diffusion server tests on ROCm. NVIDIA unaffected (CUDA path has the kernels).
• Blocked work: Wan-family 2-GPU diffusion serving CI signal on AMD.
• In-flight fix: ❌ none found — needs new PR.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5f (source: Dockerfile default — AITER_COMMIT_OVERRIDE empty; image had dev build v0.1.14rc1.dev240+g7d604afe5)

Failed Tests

All 11 failures are pytest ERROR at setup in test/server/test_server_1_gpu.py (TestDiffusionServerOneGpu::test_diffusion_generation). The server fixture lives in python/sglang/multimodal_gen/test/server/test_server_common.py.

Test File	Test Function (param)	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	test_diffusion_generation[cosmos3_nano_t2i]	TimeoutError: Server not ready within 1200.0s (root)	Log
same	test_diffusion_generation[qwen_image_edit_ti2i]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[qwen_image_layered_i2i]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[wan2_1_t2v_1.3b]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[wan2_1_t2v_1.3b_upscaling_4x]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[wan2_1_t2v_1_3b_lora_1gpu]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[wan2_2_ti2v_5b]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[fastwan2_2_ti2v_5b]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[sana_wm_ti2v]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[ltx_2_3_hq_pipeline]	RuntimeError: Scheduler port 5555 is unavailable	Log
same	test_diffusion_generation[lingbot_world_realtime_plastic_beach]	RuntimeError: Scheduler port 5555 is unavailable	Log

(qwen_image_t2i_cache_dit_scm_config_diffusers_1gpu and zimage_image_t2i_multi_lora ran first and PASSED.)

Failure Summary

The diffusion server suite started fine (2 models passed), then the 3rd case cosmos3_nano_t2i hung for the full 1200s readiness window while downloading nvidia/Cosmos3-Nano from HF Hub (model "found in cache but incomplete", download stalled at 81% — 54/67 files) and raised TimeoutError. After that timeout, all 10 remaining cases failed at setup with RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled. This is one root failure plus a 10-test cascade caused by the leaked scheduler holding the fixed port 5555.

Regression Status

Known recurring failure (latest 2 completed scheduled runs).

Recent history of test_server_1_gpu.py cosmos3/port-5555 cascade in job multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
Jun 27 18:28	28297966351	partition 1	❌ Failed	cosmos3_nano_t2i (+10 cascade)	TimeoutError → port 5555 cascade
Jun 27 12:32	28289294173	partition 1	❌ Failed	2× server timeout → port 5555 cascade (12 errors)	same pattern
Jun 27 07:09	28282108241	partition 1	⚪ N/A	—	Build failed (mori/libabsl ELF error); tests never ran

First observed (this cascade): run 28289294173 (Jun 27 12:32 UTC). The Jun 27 07:09 run failed earlier at the build stage so the diffusion tests didn't execute — no clean passing baseline for these test files exists in the queried 12h window.

Failure Cluster

"Diffusion server scheduler port 5555 leak after readiness timeout (--strict-ports cascade)"

Facts (verified observations)

• cosmos3_nano_t2i server bound the scheduler at tcp://127.0.0.1:5555 at 18:53:59, began Downloading model snapshot from HF Hub for nvidia/Cosmos3-Nano (attempt 1/5) at 18:54:01 ("Model found in cache but incomplete"), stalled at Fetching 67 files: 81% and produced no further progress until the test ERRORed at 19:13:40.
• Setup error verbatim: E TimeoutError: Server not ready within 1200.0s. raised at test/server/test_server_utils.py:492 in _wait_for_ready.
• Every subsequent case (qwen_image_edit_ti2i → lingbot) raised verbatim: RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled. Either use a different port or disable --strict-ports. at python/sglang/multimodal_gen/runtime/server_args.py:748 (_require_port), called from _adjust_network_ports (line 776) during ServerArgs.__post_init__.
• 42 Scheduler port 5555 is unavailable occurrences in this run's log; scheduler_port: 5555 is hardcoded/shared across every server launch in the partition (confirmed in each server_args dump).
• Server fixture code path: ServerManager.start() (test_server_utils.py:369-451) calls _wait_for_ready at line 449 and only constructs ServerContext at line 451 after it returns. The fixture in test_server_common.py:190-204 wraps ctx = manager.start() in try/except (RuntimeError, TimeoutError); that except branch only pytest.skips for diffusers-version messages, otherwise raise — with no cleanup() / kill_process_tree call. ctx.cleanup() only runs in the later try/finally at line 220, which is unreachable when start() raised.
• The OSError: Could not load this library ... libtorio_ffmpeg{4,5,6}.so / RuntimeError: FFmpeg extension is not available. tracebacks are benign — they also print in the passing cases.

Hypothesised Causes (with confidence)

• [HIGH] The port-5555 cascade is caused by the server-test fixture failing to kill the spawned sglang serve process tree when ServerManager.start() raises TimeoutError/RuntimeError. The leaked scheduler child keeps the hardcoded port 5555, so every later --strict-ports launch in the partition fails at _require_port.
  • Supporting: start() returns ServerContext only after _wait_for_ready succeeds (test_server_utils.py:449-451); the fixture except path (test_server_common.py:191-204) re-raises without cleanup; port 5555 stays bound for all 10 subsequent cases; scheduler_port=5555 is shared.
  • Disconfirming: (none found) — the timeline and code path are consistent; the first 2 cases that completed (and were torn down via the normal finally) did not leave port 5555 bound.
  • How to verify: add kill_process_tree on the start-failure path (or bind scheduler to an ephemeral/per-test port), rerun the partition with an intentionally slow first model.
• [MEDIUM] The trigger (cosmos3 readiness timeout) is an HF model-download/infra flake, not an sglang code regression: the nvidia/Cosmos3-Nano snapshot was incomplete in the runner cache and the re-download stalled at 81% for ~19.5 min, exhausting the 1200s deadline.
  • Supporting: log shows Model found in cache but incomplete, will download from HF Hub then a stalled Fetching 67 files: 81%; no Python exception from sglang model code; prior run (12:32) hit the same timeout→cascade on this partition, while the 07:09 run failed at an unrelated build step.
  • Disconfirming: only the latest 2 runs are observable here; a deeper history might show a model/config change that enlarged the download. Not investigated beyond 12h.
  • How to verify: check HF cache seeding for nvidia/Cosmos3-Nano on the MI325 runners; rerun and confirm whether the download completes within deadline.
• [LOW] aiter involvement — not implicated; AITER_COMMIT_OVERRIDE is empty (Dockerfile default) and the failure is a port/download issue with no kernel stack frames.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for strict-ports scheduler port, diffusion server port, cosmos3 download (https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<kw>). The 1 strict-ports hit (#24227) is an unrelated WanVideo TeaCache off-by-one. Needs new PR or investigation.

Suggested Triage Steps

• Make the server-test fixture cleanup robust: in test_server_common.py:191-204, on the except (RuntimeError, TimeoutError) path call manager's process kill (kill_process_tree) / a finally that tears down any spawned process before raise, so a single readiness timeout cannot leak the scheduler.
• Alternatively/additionally, stop sharing a fixed scheduler_port=5555 across cases in a partition — allocate a free/ephemeral scheduler port per server launch (or disable --strict-ports for the scheduler port in tests) so a leftover process can't cascade.
• Address the trigger: pre-seed / verify the nvidia/Cosmos3-Nano HF snapshot on MI325 runners (cache was "incomplete"); consider a download-completeness check or shorter download timeout that fails fast instead of consuming the full 1200s readiness window.
• To confirm the cascade mechanism: rerun partition 1 and check whether, after the first timeout, a leftover sglang process holds 5555 (ss -ltnp | grep 5555).

Status

• Persistence: cosmos3 timeout → port-5555 cascade present in the latest 2 completed scheduled runs (since Jun 27 12:32 UTC). The Jun 27 07:09 run failed earlier at the build stage, so no green baseline for these test files in the 12h window.
• Scope: multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 1) in pr-test-amd.yml; 1 root timeout + 10 cascaded ERRORs in this run. 2 cases in the partition still passed.
• Blocked work: AMD 1-GPU diffusion server CI signal for ~10 diffusion models in this partition.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:40 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/7)

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 1) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 592f6c849b
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — Dockerfile default matched as dev build, rebuild skipped per [CI-AITER-CHECK] markers)

Failed Tests

All in sglang/multimodal_gen/test/server/test_server_2_gpu.py, class TestDiffusionServerTwoGpu, method test_diffusion_generation[<scenario>]. 8 failed, 4 passed, 13 deselected in 2319.79s.

Test File	Test Function (scenario)	Error	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_2_i2v_a14b_2gpu]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_2_t2v_a14b_2gpu]	same	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_t2v_14b_2gpu]	same	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_t2v_1.3b_cfg_parallel]	same	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[mova_360p_tp2]	same	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_i2v_14b_480P_2gpu]	same	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[wan2_1_i2v_14b_720P_2gpu]	same	Log
test/server/test_server_2_gpu.py	test_diffusion_generation[flux2_modelopt_fp8_tp2_t2i]	openai.InternalServerError: Internal Server Error	Log

Failure Summary

8 of 12 diffusion server scenarios failed in this 2-GPU AMD job. Seven video scenarios (all wan2_* + mova_360p_tp2) failed during VAE decode with RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA. The eighth (flux2_modelopt_fp8_tp2_t2i) failed separately with an openai.InternalServerError from the diffusion server. These are two distinct failure clusters with different histories.

Regression Status

Cluster 1 (causal Conv3D, 7 video tests): New regression, first failing run 28223639512 (head 5996b54bd3).
Cluster 2 (flux2 FP8): Pre-existing recurring, fails independently of this commit.

Recent history of test_server_2_gpu.py scenarios in job multimodal-gen-test-2-gpu-amd (..., 1) (branch main, event=schedule, completed runs only):

Date (UTC)	Run	Head	causal Conv3D tests	flux2 FP8 test
Jun 27	28297966351	592f6c849b	❌ 7 failed	❌ Failed
Jun 26	28223639512	5996b54bd3	❌ failed (first)	❌ Failed
Jun 26	28210138009	781537b61d	✅ Passed (10 passed, 1 skip)	❌ Failed

For cluster 1: last passing run 28210138009 (head 781537b61d, Jun 26 00:54 UTC, 0 occurrences of the error); first failing run 28223639512 whose head IS the introducing commit (Jun 26 07:23 UTC, 145 occurrences). For cluster 2: the flux2 test already failed in the last cluster-1-passing run, so it predates this commit.

Failure Cluster

• Cluster 1: "RuntimeError: causal Conv3D cat/pad fusion only available on CUDA (AMD VAE decode)"
• Cluster 2: "Diffusion server InternalServerError on flux2 modelopt-fp8 (AMD)"

Facts (verified observations)

• = 8 failed, 4 passed, 13 deselected, 7 warnings in 2319.79s (this run's pytest summary).
• 7 video scenarios each fail with: Model generation returned no output. Error from scheduler: ... causal Conv3D cat/pad fusion is only available on CUDA (verbatim from log).
• The error string is raised at python/sglang/multimodal_gen/runtime/layers/parallel_conv.py:57.
• That file was last modified by #29281 (5996b54bd3, merged Jun 26 07:06 UTC).
• Pre-commit run 28210138009 (head 781537b61d): 0 occurrences of the error; the 7 video scenarios passed (10 passed, 1 skipped); only flux2 failed.
• First-failing run 28223639512, head = the introducing commit 5996b54bd3: 145 occurrences, 6 failed.
• flux2 fails via openai.InternalServerError: Internal Server Error (client-side); the underlying server error is not in this client log.

Hypothesised Causes (with confidence + disconfirming evidence)

• [HIGH] Cluster 1 is caused by #29281 (5996b54bd3), which broke the AMD fallback path in parallel_conv.py.

  • Supporting: (a) the commit adds the exact RuntimeError string and the fused_causal_conv3d_cat_pad function that always raises on non-CUDA; (b) it changed the non-CUDA symbol from fused_causal_conv3d_cat_pad = None to a function object — defeating the guard if cache_x is None or fused_causal_conv3d_cat_pad is None: return False at parallel_conv.py:110. The only other CUDA guard, not x.is_cuda at line 112, is ineffective on ROCm where tensor.is_cuda returns True for HIP tensors, so _can_fuse_causal_conv3d_cat_pad now returns True on AMD and reaches the raising call at line 140; (c) timing: the first failing run's head_sha IS this commit, and the immediately prior run (0 occurrences) passed these tests.
  • Disconfirming: (none found) — the A/B at the commit boundary is clean (0 → 145 occurrences across the exact SHA).
  • How to verify: git revert 5996b54b on a branch and re-run test_server_2_gpu.py video scenarios on AMD MI325; or confirm torch.Tensor.is_cuda returns True under ROCm in the CI image.

• [MEDIUM] Cluster 2 (flux2 FP8) is an AMD FP8 GEMM issue, independent of cluster 1.

  • Supporting: the test already failed in the pre-commit run 28210138009; open PR #28889 title explicitly targets "flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED".
  • Disconfirming: the client-side InternalServerError in this job log does not directly show HIPBLAS_STATUS_NOT_SUPPORTED (server-side error not captured here), so the link to #28889 is by title/scenario match, not log confirmation.
  • How to verify: inspect the diffusion server stderr for the flux2 run; confirm the HIPBLAS FP8 error, then validate against PR #28889.

In-flight Fix Check

• Cluster 1 (causal Conv3D): In-flight fix: ❌ none found — searched open PRs for causal conv3d and conv3d rocm; no PR addresses the AMD/ROCm fallback regression. Needs new PR or revert of #29281.
• Cluster 2 (flux2 FP8): In-flight fix: ✅ #28889 "[AMD] Fix FP8 e4m3fn GEMM on ROCm: dequant->bf16 fallback in apply_fp8_linear (fixes flux2 modelopt-fp8 HIPBLAS_STATUS_NOT_SUPPORTED)" — likely fix; chase reviewers, do NOT open a duplicate.

Suggested Triage Steps

• Cluster 1: Revert 5996b54b (#29281) on a branch and re-run the wan2_* / mova_360p_tp2 scenarios on AMD MI325 to confirm green.
• Cluster 1 (fix direction, hypothetical): restore an AMD-safe path in parallel_conv.py — e.g. make _can_fuse_causal_conv3d_cat_pad short-circuit on non-CUDA platforms (gate on current_platform.is_cuda() rather than relying on x.is_cuda, which is True on ROCm), or provide a Triton/eager fallback instead of raising at line 57.
• Cluster 2: Verify open PR #28889 resolves flux2_modelopt_fp8_tp2_t2i; capture the diffusion server stderr to confirm the HIPBLAS FP8 root cause before merge.

Status

• Persistence: Cluster 1 failing since first appearance Jun 26 07:23 UTC (28223639512), every completed scheduled run since (Jun 26 → Jun 27). Cluster 2 (flux2) failing at least since Jun 26 00:54 UTC, predates the cluster-1 regression.
• Scope: 7 video-generation scenarios in test_server_2_gpu.py on the 2-GPU AMD MI325 job (cluster 1); 1 FP8 scenario (cluster 2). Cluster 1 likely also affects sibling AMD diffusion jobs that exercise Wan/Mova VAE decode.
• Blocked work: Wan2.1/Wan2.2 + Mova video generation CI signal on AMD.
• In-flight fix: Cluster 1 ❌ none found (revert #29281 or add AMD fallback). Cluster 2 ✅ #28889.

multimodal-gen-test-1-gpu-amd (linux-mi325-1gpu-sglang, 3) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: 592f6c84
• aiter: N/A — no [CI-AITER-CHECK] markers in this log; AITER_COMMIT_OVERRIDE= empty (Dockerfile default, regular schedule run — not a scout job)

Failed Tests

Job ran 10 items (test_update_weights_from_disk.py): 5 failed, 1 passed, 4 errors. Two distinct root causes.

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/single_test_file/test_update_weights_from_disk.py	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[Qwen-Image]	AssertionError: Update failed: {'success': False, 'message': 'Inplace update to inference tensor outside InferenceMode is not allowed.'} — HTTP 500 (line 454)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[Qwen-Image]	assert 500 == 200 (line 432) — server returned 500 from /update_weights_from_disk	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[Qwen-Image]	assert (500 == 200) (line 535)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[FLUX.2-klein-base-4B]	AssertionError: Expected 200, got 500 (line 638)	Log
same	TestUpdateWeightsFromDiskWithOffload::test_update_weights_with_offload_enabled[Qwen-Image]	AssertionError: Expected 200, got 500 (line 638)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_from_disk_default[FLUX.2-klein-base-4B]	ERROR at setup — AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_* (fixture line 410 → 220)	Log
same	TestUpdateWeightsFromDisk::test_update_weights_specific_modules[FLUX.2-klein-base-4B]	ERROR at setup — same as above	Log
same	TestUpdateWeightsFromDisk::test_update_weights_rejects_invalid_requests[FLUX.2-klein-base-4B]	ERROR at setup — same as above	Log
same	TestUpdateWeightsFromDisk::test_corrupted_weights_rollback[FLUX.2-klein-base-4B]	ERROR at setup — same as above	Log

Failure Summary

The standalone test file test_update_weights_from_disk.py (1-gpu shard 3) reports 5 failed / 1 passed / 4 errors. The 5 failures are HTTP 500 responses from the /update_weights_from_disk server endpoint, all carrying the message Inplace update to inference tensor outside InferenceMode is not allowed. The 4 errors are pytest setup errors in the diffusion_server_no_offload fixture for the FLUX model, where a checksum helper asserts on a transformer subdirectory that does not exist in the perturbed-VAE temp clone.

Regression Status

New regression — test newly enabled/refactored on AMD CI by #28762, failing on its first and every subsequent scheduled run.

History of test_update_weights_from_disk.py in multimodal-gen-test-1-gpu-amd (branch main, event=schedule, completed runs only — checked at test-file level across all 1-gpu shards because the CI refactor reshuffled shard assignments):

Date (UTC)	Run	head_sha	Test File Status	Notes
06-27 18:28	28297966351	592f6c84	❌ 5 failed/1 passed/4 errors	this run (shard 3)
06-27 12:32	28289294173	3306233961	❌ 5 failed/1 passed/4 errors	first run after #28762; identical signature (job)
06-27 07:09	28282108241	9214b9338f	➖ Not present	test did not run on AMD before #28762 (merged ~11:22 UTC 06-27)

First observed failure: run 28289294173 (06-27 12:32 UTC). No green run of this test on AMD exists in the queryable window — it was introduced to AMD CI already-failing by #28762 (lookback: last 8 scheduled runs, 06-26 00:54 → 06-27 18:28 UTC).

Failure Cluster

• Cluster A: update_weights_from_disk endpoint returns 500 — "Inplace update to inference tensor outside InferenceMode" (5 failures)
• Cluster B: Test fixture bug — perturbed-VAE clone missing 'transformer' dir (4 setup errors, FLUX only)

Facts (verified observations)

• Server-side crash (Cluster A), verbatim from log line 5935-5957:

  File ".../runtime/post_training/weights_updater.py", line 200, in load_weights_into_model
    weight_loader(param, loaded_weight.to(param.dtype))
  File ".../runtime/layers/linear.py", line 575, in weight_loader
    param_data.copy_(loaded_weight)
  RuntimeError: Inplace update to inference tensor outside InferenceMode is not allowed.
  

  The first attempt also raised AssertionError: param_data.shape == loaded_weight.shape (linear.py:411), which triggered _rollback in weights_updater.py:381; the rollback then hit the inference-tensor copy_ error. POST /update_weights_from_disk → 500 Internal Server Error (log line ~5975).
• Fixture setup error (Cluster B), verbatim from ERRORS section (log line 12902-12908):

  test_update_weights_from_disk.py:410: in diffusion_server_no_offload
    ) != _compute_checksum_from_disk(perturbed_vae_model_dir, module), (
  test_update_weights_from_disk.py:220: in _compute_checksum_from_disk
    assert os.path.exists(weights_dir)
  E AssertionError: No weights dir for transformer in /tmp/sglang_perturbed_vae_2qsjlcyz
  

  The perturbed clone dir (/tmp/sglang_perturbed_vae_*) contains only the perturbed vae module, not transformer, so the per-module checksum loop fails. Only the FLUX non-offload fixture hit this; the Qwen non-offload fixtures proceeded to the request and returned 500 (Cluster A).
• [CI-AITER-CHECK] markers: none. AITER_COMMIT_OVERRIDE empty → Dockerfile-default aiter; not a scout run.
• #28762 (91f9e737, "[diffusion] CI: refactor CI", merged 06-27 ~11:22 UTC) modified weights_updater.py (32 lines) and rewrote test_update_weights_from_disk.py (95 lines).
• linear.py was last modified by efbe67d2 (06-25, Gemma4 memory recipe) — the param_data.copy_() crash site itself is unchanged by #28762.
• Reproducibility: identical 5-failed/1-passed/4-error signature in both completed post-#28762 scheduled runs (12:32 and 18:28 UTC 06-27).

Hypothesised Causes (with confidence)

• [MEDIUM] Cluster A: the diffusion weight-update path (weights_updater.py → linear.py:575 param_data.copy_()) performs an in-place copy into model parameters that are PyTorch inference tensors (loaded under inference_mode), which PyTorch forbids outside InferenceMode.

  • Supporting: stack trace lands exactly on param_data.copy_(loaded_weight) (linear.py:575) and the error string is PyTorch's verbatim inference-tensor guard; all 5 failures are the same 500 message; endpoint returns 500. weights_updater.py was touched by #28762, and the test only began exercising this path on AMD after #28762.
  • Disconfirming: linear.py:575 itself was not changed by #28762, and the same code runs on NVIDIA — this run cannot prove the path passes on NVIDIA, so an AMD/ROCm-specific load-mode interaction cannot be ruled out. Because the test never ran on AMD before #28762, there is no green AMD baseline to confirm this is a regression vs a never-worked-on-AMD path.
  • How to verify: on an AMD MI325 container, POST /update_weights_from_disk against Qwen-Image and check whether wrapping the copy_ in with torch.inference_mode(): (or cloning the param to a normal tensor) clears the 500; compare against an NVIDIA run of the same test.

• [FACT] Cluster B: the diffusion_server_no_offload fixture checksums every module (including transformer) from the perturbed temp clone, but the clone only contains the perturbed vae, so os.path.exists(.../transformer) is False.

  • Supporting: the assertion message names /tmp/sglang_perturbed_vae_*/transformer and the fixture loops modules at test_update_weights_from_disk.py:410; this is a test-code logic issue, directly observable.
  • Disconfirming: (none — the assertion is deterministic and in test code).
  • How to verify: inspect _clone_model_with_modified_module / the perturbed-dir construction around line 410 and confirm only vae is copied into the clone.

• [LOW] Both clusters are entirely attributable to #28762 (the only commit touching weights_updater.py + the test in the window, and the test did not run before it).

  • Supporting: temporal correlation is exact — test absent at 07:09 UTC, failing at 12:32 UTC (first run after merge).
  • Disconfirming: #28762 did not modify the linear.py:575 crash site for Cluster A; the underlying inference-tensor issue may be pre-existing product behavior merely exposed (not introduced) by the test refactor.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with update_weights_from_disk+inference+tensor, Inplace+update+inference+tensor, and weights_updater+inference; no matching diffusion weight-update fix (results were unrelated MoE/quantization PRs). Needs a new PR or investigation.

Suggested Triage Steps

• Cluster A: Reproduce POST /update_weights_from_disk for Qwen-Image on AMD MI325; inspect weights_updater.py:200 / linear.py:575 and test whether wrapping the in-place copy_ in torch.inference_mode() (or param_data = param_data.clone() to a normal tensor) resolves the 500. Confirm whether the same test passes on NVIDIA to determine if this is AMD-specific or a general product bug.
• Cluster A: Review the weights_updater.py changes in #28762 (91f9e737) (git show 91f9e737 -- python/sglang/multimodal_gen/runtime/post_training/weights_updater.py) to see if the refactor altered the inference-mode context around weight loading.
• Cluster B: Fix the test fixture at test_update_weights_from_disk.py:410 so the per-module checksum only iterates modules actually present in the perturbed clone (or clone all required modules into the temp dir).
• If the maintainers consider this test not yet ready for AMD signal, consider gating/skipping it on AMD until both the fixture bug and the inference-tensor path are resolved, since it has never produced a green run on AMD.

Status

• Persistence: Failing since 06-27 12:32 UTC (every completed scheduled run since #28762 merged — 2/2 runs).
• Scope: 1 test file (test_update_weights_from_disk.py), shard 3 of multimodal-gen-test-1-gpu-amd in pr-test-amd.yml; 9 of 10 items affected (5 fail + 4 setup-error). Two independent causes (server endpoint 500 + test fixture bug).
• Blocked work: AMD CI signal for diffusion update_weights_from_disk (post-training weight reload) on 1-GPU.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — AITER_COMMIT_OVERRIDE empty, dev build detected so no rebuild). Not relevant to this failure.

Failed Tests

Test File	Test Function	Error	Log
test/registered/quant/test_quark_mxfp4.py	TestOnlineQuantizationMemoryLoadMOE.setUpClass (server launch for Qwen/Qwen3-30B-A3B-Instruct-2507)	Timeout on acquiring lock on HF cache .lock → whole file ✗ TIMEOUT after 1200s	Log

Failure Summary

The test runner aborted test/registered/quant/test_quark_mxfp4.py after the 1200 s hard timeout (est_time was 106 s). The TestOnlineQuantizationMemoryLoadDense cases (Qwen/Qwen3-8B) ran and passed; the run then entered TestOnlineQuantizationMemoryLoadMOE.setUpClass, which launched a server for Qwen/Qwen3-30B-A3B-Instruct-2507. Two server processes blocked on a HuggingFace cache file lock (/sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/aeb13307…dae4.lock) from 18:40:11 until the timeout at 18:57:09, never acquiring it, so the MOE server never loaded weights or became healthy.

Regression Status

New regression at the job level / infrastructure-flavored failure. The same job and the same test file passed in the immediately preceding scheduled runs.

Recent history of test/registered/quant/test_quark_mxfp4.py in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
Jun 27 18:28	28297966351	job	❌ Failed	TestOnlineQuantizationMemoryLoadMOE.setUpClass	Timeout on acquiring lock → TIMEOUT 1200s
Jun 27 12:32	28289294173	job	✅ Passed (elapsed=543s)	—	—
Jun 27 07:09	28282108241	job	✅ Passed	—	—
Jun 27 00:51	28273478888	job	✅ Passed	—	—
Jun 26 18:40	28258143566	job	✅ Passed	—	—

First observed failure: 2026-06-27 18:28 UTC run. Last known passing: 2026-06-27 12:32 UTC run (same test, 543 s).

Failure Cluster

HuggingFace cache file-lock timeout during MOE model load (test hard-timeout)

Facts (verified observations, no interpretation)

• Verbatim from log: [2026-06-27 18:57:02] Timeout on acquiring lock 140245344447472 on /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/... and Timeout on acquiring lock 135049363198384 on ... (two distinct processes).
• ✗ TIMEOUT: /sglang-checkout/test/registered/quant/test_quark_mxfp4.py after 1200 seconds (log line 87492); {"file": ".../test_quark_mxfp4.py", "passed": false, "elapsed": 1200}.
• 40,469 not acquired … waiting 0.05 seconds lines for the same model lock, spanning 18:40:11 → 18:57:09.
• The Dense (Qwen3-8B) portion completed normally in this same run: Peak GPU memory before loading weights: 5.778 GiB (18:37:40), Online quark_mxfp4 quantization: quantized 144 layers (18:38:00), Server …/health is ready (18:39:39). The MOE server (Qwen3-30B) never reached weight load.
• In the prior passing run 28289294173, the MOE server for the same model loaded fine (quantized 144 layers of types: ['experts', 'o_proj', 'qkv_proj'] at 12:49:08) and the file passed in 543 s — already ~5× over its 106 s estimate.
• Commit window pass..fail is only 6 commits (3306233961..592f6c84): #28713, #29507, #29186, #27563, #29106, #29342. None touch HF-cache, filelock, or Qwen3-MoE loading.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Shared HF cache lock contention / stale lock on the /sgl-data/hf-cache mount — a concurrent or previously-crashed process held (or left a stale) .lock for the Qwen3-30B-A3B-Instruct-2507 weight shard, so this run's two server processes timed out acquiring it.
  • Supporting: Timeout on acquiring lock printed for two distinct lock owners on the same shared-mount .lock; the model loaded fine 6 h earlier on the same cache; 6-commit window contains no related code change; /sgl-data is a shared CI cache mount, a known source of cross-run lock contention.
  • Disconfirming: the log does not name the lock-holding PID, so we cannot prove another process held it (could in principle be a hung downloader inside this same run with HF_HUB_OFFLINE=0).
  • How to verify: re-run the job; inspect /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/ on the linux-mi35x-gpu-1 runner for stale .lock files; check whether another CI job sharing the cache ran concurrently at 18:40 UTC.
• [LOW] Slow/stalled HF download of the 30B MOE weights pushing past the 1200 s cap.
  • Supporting: HF_HUB_OFFLINE=0; the file already ran 543 s in the prior pass, leaving little headroom; the 30B model is large.
  • Disconfirming: the log shows lock acquisition timeouts, not download-progress stalls — weights download never started because the lock was never obtained.
  • How to verify: confirm the weight shards are already present in the shared cache (they were used 6 h prior), which would rule out a fresh download.
• No hypothesis reaches HIGH; the 6 in-window commits are unrelated, so a code regression is unlikely.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for hf-cache lock, filelock CI, and quark_mxfp4; results (#28605, #22451, #29178, #29328, #28291) are unrelated to the HF-cache lock timeout.

Suggested Triage Steps

• Re-run the job (28297966351, job 83841118252); if it passes, this is confirmed transient HF-cache lock contention rather than a code regression.
• On the linux-mi35x-gpu-1 runner, check /sgl-data/hf-cache/hub/.locks/models--Qwen--Qwen3-30B-A3B-Instruct-2507/ for stale .lock files and clear them; check for concurrent CI jobs sharing /sgl-data at the failure time.
• Given the test already runs ~543 s against a 106 s estimate, consider (separately) pre-warming the Qwen3-30B-A3B weights into the cache or raising the per-file timeout, so transient lock waits don't blow the 1200 s cap.

Status

• Persistence: 1 completed run (2026-06-27 18:28 UTC). The previous 4 consecutive scheduled runs of this job passed, including the same test file passing 543 s at 12:32 UTC.
• Scope: this job / this test file only (stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)); the MOE Qwen/Qwen3-30B-A3B-Instruct-2507 server load specifically. Dense Qwen3-8B cases passed in the same run.
• Blocked work: blocks the MXFP4 online-quantization MOE memory/accuracy signal on AMD MI35x.
• In-flight fix: ❌ none found — likely transient infra (HF cache lock); re-run to confirm before opening a PR.

multimodal-gen-test-2-gpu-amd (linux-mi325-2gpu-sglang, 0) — failed step(s): Run diffusion server tests (2-GPU)

Commit Info

• sglang: 33062339
• aiter: N/A — AITER_COMMIT_OVERRIDE is empty in this run; no [CI-AITER-CHECK] markers relevant (Dockerfile default, not implicated)

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[wan2_2_t2v_a14b_teacache_2gpu]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[wan2_2_t2v_a14b_lora_2gpu]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[wan2_1_t2v_1_3b_cache_dit_sp_only_2gpu]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[mova_360p_ring1_uly2]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log
python/sglang/multimodal_gen/test/server/test_server_2_gpu.py	TestDiffusionServerTwoGpu::test_diffusion_generation[wan2_1_i2v_14b_lora_2gpu]	RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA	Log

Failure Summary

5 of 13 diffusion 2-GPU server tests failed (5 failed, 7 passed, 1 skipped). Every failure is the same scheduler-side RuntimeError: causal Conv3D cat/pad fusion is only available on CUDA, raised during the VAE decode stage on the AMD MI325 (ROCm) runner. The error surfaces to the test as Model generation returned no output. Error from scheduler: .... The error count appears 200+ times in the log because it fires on every decode chunk / warmup request.

Regression Status

New regression — first appeared in the run carrying commit 5996b54b (#29281).

Recent history of test_server_2_gpu.py (test_diffusion_generation[wan*/mova*]) in job multimodal-gen-test-2-gpu-amd (branch=main, event=schedule, completed runs only). Note: tests are sharded across 3 partitions by run, so the table tracks the conv3d failure signature across all partitions of each run, not the single partition-0 job:

Date (UTC)	Run	Conv3D failures in run	Test File Status
Jun 27 12:32	28289294173 (this run)	yes (203 in part 0)	❌ Failed
Jun 26 07:23	28223639512	yes (203 in part 0)	❌ Failed (first failing)
Jun 26 00:54	28210138009	0 across all 3 partitions	✅ Passed (last green)

• First observed failure: run 28223639512, head 5996b54bd3 (Jun 26 07:23 UTC).
• Last passing: run 28210138009, head 781537b61d (Jun 26 00:54 UTC), 0 conv3d errors across all partitions.
• Regression window: 781537b61d..5996b54bd3 — contains exactly one commit touching parallel_conv.py: 5996b54b.

Failure Cluster

RuntimeError "causal Conv3D cat/pad fusion is only available on CUDA" during VAE decode on ROCm

Facts (verified observations)

• Verbatim error + stack (log lines 4727–4757):

  File ".../runtime/models/vaes/wanvae.py", line 1331, in forward
    x = self.conv_in(x, _feat_cache[idx])
  File ".../runtime/layers/parallel_conv.py", line 641, in forward
    x = causal_conv3d_cat_pad(x, cache_x, padding)
  File ".../runtime/layers/parallel_conv.py", line 140, in causal_conv3d_cat_pad
    return fused_causal_conv3d_cat_pad(x, cache_x, padding)
  File ".../runtime/layers/parallel_conv.py", line 57, in fused_causal_conv3d_cat_pad
    raise RuntimeError("causal Conv3D cat/pad fusion is only available on CUDA")
  

• Reproducibility: 5/5 failing tests share the identical error and stack-trace top frames. 203 occurrences in this job's log; also 203 in the first-failing run's partition-0 log (job 83610402410).
• A/B by run: pre-commit run 28210138009 had 0 conv3d errors across all 3 partitions; first run with 5996b54b had 203.
• Code paths in stack trace verified by reading source: parallel_conv.py:57, :140, wanvae.py:1331.
• This is pr-test-amd.yml (event=schedule), not an aiter-scout run — no aiter override; Failure Origin / Baseline A/B section omitted.

Hypothesised Causes (with confidence)

• [HIGH] Commit 5996b54b (#29281) broke the ROCm fallback path in parallel_conv.py.
  • Supporting:
    • The commit changed the non-CUDA branch (parallel_conv.py:28-31) from fused_causal_conv3d_cat_pad = None to defining fused_causal_conv3d_cat_pad as an always-present wrapper function that raises RuntimeError("causal Conv3D cat/pad fusion is only available on CUDA") when both CUDA and Triton kernels are None (the ROCm case) — parallel_conv.py:37-58.
    • The gate _can_fuse_causal_conv3d_cat_pad (parallel_conv.py:110) still short-circuits on fused_causal_conv3d_cat_pad is None, but that symbol is now never None, so the only remaining platform guard is x.is_cuda at parallel_conv.py:112. On ROCm, PyTorch reports tensor.is_cuda == True, so the guard passes, _can_fuse... returns True, and causal_conv3d_cat_pad (:139-140) calls the fused wrapper, which raises. Before the commit, fused_causal_conv3d_cat_pad was None on ROCm → _can_fuse... returned False → it fell back to the torch.cat+F.pad path (:141-144).
    • Exact temporal match: 0 conv3d errors in the last run before the commit, 203 in the first run with it; it is the only commit in the window touching this file.
  • Disconfirming: (none found) — the regression window contains exactly one commit touching parallel_conv.py, and the raised string is literally introduced by this commit's diff.
  • How to verify: git revert 5996b54b on a branch (or guard the fast path with current_platform.is_cuda() instead of x.is_cuda) and re-run test_server_2_gpu.py::TestDiffusionServerTwoGpu::test_diffusion_generation[wan2_1_t2v_1_3b_cache_dit_sp_only_2gpu] on AMD MI325.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for causal conv3d (4 hits, none related) and parallel_conv / conv3d rocm (no ROCm conv3d-fallback fix). Needs a new PR or to ping the #29281 author.

Suggested Triage Steps

• Confirm the platform-guard hypothesis: the x.is_cuda check at parallel_conv.py:112 is true on ROCm; gate the fused path on current_platform.is_cuda() (or restore a fused_causal_conv3d_cat_pad is None fast-exit for non-CUDA) so ROCm falls back to torch.cat+F.pad.
• Revert 5996b54b on a branch and re-run the 5 failing wan*/mova* cases on AMD MI325 to confirm green, as an A/B check.
• Ping the #29281 author — the fix is small (a platform guard) and the PR is CUDA-only by design ("CUDA fast path for Cosmos3"); the ROCm fallback regression was likely unintended.

Status

• Persistence: Failing on every completed scheduled run since Jun 26 07:23 UTC (run 28223639512) — ~1.2 days, every run since.
• Scope: multimodal-gen-test-2-gpu-amd on pr-test-amd.yml; 5 diffusion VAE-decode tests (wan2_1/wan2_2/mova families). Likely also affects other ROCm diffusion VAE-decode tests that hit causal_conv3d_cat_pad with a non-null cache.
• Blocked work: AMD diffusion 2-GPU VAE-decode CI signal for wan/mova models.
• In-flight fix: ❌ none found in open PRs.

stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 6) — failed step(s): Run test

Commit Info

• sglang: 592f6c84
• aiter: 7d604afe5 (v0.1.14rc1.dev240+g7d604afe5, source: dev image preserved — Dockerfile default matched, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/utils/test_type_based_dispatcher.py	test_type_dispatcher_e2e_performance	TypeError: Missing required argument 'input_embeds'	Log

All other 9 test files in this shard passed (Test Summary: 9/10 passed).

Failure Summary

The unit test test_type_dispatcher_e2e_performance constructs a TokenizedGenerateReqInput at test/registered/utils/test_type_based_dispatcher.py:124 without supplying the input_embeds (and token_type_ids) fields. Those fields are declared on the struct with no default value, so construction raises TypeError and the test errors out before any dispatch logic runs. This is a test/source mismatch in unit-test code, not a GPU/kernel/runtime failure.

Regression Status

Never-passed (no green run found in queryable window). The test fails in every completed scheduled pr-test-amd.yml run back to the earliest one queryable (2026-06-25 00:53 UTC). Note: #28688 (merged 2026-06-26 19:04) converted these IPC classes from @dataclass to msgspec.Struct, which only changed the error wording — the test was already failing for the identical root cause before that PR (dataclass variant: missing 2 required positional arguments: 'input_embeds' and 'token_type_ids').

Recent history of test_type_based_dispatcher.py (test_type_dispatcher_e2e_performance), same branch (main), same event (schedule), completed runs only:

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
06-27 18:28	run	shard 6	❌ Failed	test_type_dispatcher_e2e_performance	Missing required argument 'input_embeds' (msgspec)
06-27 00:51	run	shard 6	❌ Failed	same	Missing required argument 'input_embeds' (msgspec)
06-26 18:40	run	shard 6	❌ Failed	same	missing 2 required positional arguments (dataclass)
06-26 12:48	run	shard 6	❌ Failed	same	dataclass variant
06-26 00:54	run	shard 6	❌ Failed	same	dataclass variant
06-25 18:48	run	shard 6	❌ Failed	same	dataclass variant
06-25 07:17	run	shard 0	❌ Failed	same	dataclass variant
06-25 00:53	run	shard 0	❌ Failed	same	dataclass variant

Lookback window: 2026-06-25 00:53 → 2026-06-27 18:28 UTC. No passing run for this test file found. (Shard index varies between runs because the dispatcher re-partitions files; the test still fails wherever it lands.)

Failure Cluster

TypeError: TokenizedGenerateReqInput missing required field (input_embeds/token_type_ids) in unit test

Facts (verified observations)

• Verbatim error (this run, log ~line 13256): TypeError: Missing required argument 'input_embeds', raised at test/registered/utils/test_type_based_dispatcher.py:124 inside TokenizedGenerateReqInput(...).
• The test call at test_type_based_dispatcher.py:124-134 passes input_text, input_ids, mm_inputs, sampling_params, return_logprob, logprob_start_len, top_logprobs_num, token_ids_logprob, stream — it does not pass input_embeds or token_type_ids.
• In python/sglang/srt/managers/io_struct.py:771-779, class TokenizedGenerateReqInput(BaseReq, kw_only=True) declares input_embeds (line 776) and token_type_ids (line 779) with no default, so both are required.
• #28688 (be1930133, merged 2026-06-26 19:04) only changed @dataclass → msgspec.Struct(..., kw_only=True). Pre-merge run 83726425964 shows the dataclass-form error for the same test → the bug predates #28688.
• aiter is unchanged across passing/failing comparison (no green run exists); failure is pure Python TypeError at import/construction, no kernel involvement. NVIDIA-vs-AMD is irrelevant — this is platform-independent unit-test code.

Hypothesised Causes

• [HIGH] The test was not updated when input_embeds / token_type_ids became required (no-default) fields of TokenizedGenerateReqInput. The test constructs the struct omitting both.
  • Supporting: stack trace points exactly at the construction site (test_type_based_dispatcher.py:124); the two named missing args (input_embeds, token_type_ids) are precisely the no-default fields declared at io_struct.py:776,779; failure reproduces identically across 8 consecutive completed runs and in both the dataclass and msgspec forms.
  • Disconfirming: (none found) — error is deterministic and self-explanatory; no flakiness, no green run.
  • How to verify: add input_embeds=None, token_type_ids=None to the three TokenizedGenerateReqInput(...) constructions in the test and re-run locally (python3 test/registered/utils/test_type_based_dispatcher.py); this is exactly what in-flight PR #29376 does.

In-flight Fix Check

• In-flight fix: ✅ #29376 "Fix test_type_based_dispatcher after TokenizedGenerateReqInput field changes" (open since 2026-06-26) — directly targets this test. Chase reviewers; do not open a duplicate. (Search: repo:sgl-project/sglang is:pr is:open test_type_based_dispatcher.)
• A branch amd_fix_test_type_based_dispatcher also appears being fetched in the CI runs, consistent with an active fix.

Suggested Triage Steps

• Verify #29376 adds the missing input_embeds / token_type_ids arguments to all three TokenizedGenerateReqInput(...) constructions in test/registered/utils/test_type_based_dispatcher.py, then merge to restore the shard-b 1-GPU-small AMD signal.
• After merge, re-run pr-test-amd.yml (schedule) and confirm test_type_based_dispatcher.py reports passed: true.
• If #29376 has stalled, ping its reviewers — this single test has blocked the stage-b-test-1-gpu-small-amd shard on every scheduled run for ≥3 days.

Status

• Persistence: Failing every completed scheduled run since at least 2026-06-25 00:53 UTC (≥3 days; no green run in queryable window).
• Scope: One test file / one function on the stage-b-test-1-gpu-small-amd shard of pr-test-amd.yml. Platform-independent (unit-test construction error), so any workflow running this file is affected.
• In-flight fix: ✅ #29376 (open since 2026-06-26) — awaiting merge.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-06-28 02:40 UTC)