Fix AMD 8-GPU-feature CI#3896
Conversation
yuankaichen-amd
requested review from
fegin,
tianyu-l,
wconstab and
wwwjn
as code owners
meta-cla
Bot
added
the
CLA Signed
|
The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:
Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows. |
Compiled ROCm flex attention (torch.compile(flex_attention)) on gfx950 is run-to-run unstable without explicit kernel_options and amplifies tiny (~1e-6) bf16 activation drift between FSDP8 all-gather and HSDP(4,2) replicate paths, causing step-1 loss mismatches that fail the exact --assert-equal parity check. Run flex attention eagerly (uncompiled) in --debug.deterministic mode on ROCm, which matches varlen FA parity exactly. CUDA keeps the compiled (max_autotune disabled) path unchanged. Refresh the ROCm llama3 reference loss file to match the eager-flex curve. Co-authored-by: Cursor <cursoragent@cursor.com>
The previous reference was generated while another job contended for the GPUs, introducing ~1e-6 run-to-run numerical drift. Regenerated from a fresh FSDP8 eager-flex run on an uncontended node; two back-to-back runs now produce bitwise-identical losses, so the exact --assert-equal reference check reproduces reliably. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@yuankaichen-amd can you explain why you rocm is not deterministic |
|
@drisspg we find that the computation of dq and dk in the backward kernel in compiled mode is non-deterministic due to the numerical difference caused by dynamic reduction order. I don't know how CUDA passes it. But the compiled mode uses FlashAttention which algorithmically does not guarantee determinism. The eager mode is the dense attention implementation which is algorithmically deterministic. |
3 participants
Compiled ROCm flex attention (torch.compile(flex_attention)) on gfx950 is run-to-run unstable without explicit kernel_options and amplifies tiny (~1e-6) bf16 activation drift between FSDP8 all-gather and HSDP(4,2) replicate paths, causing step-1 loss mismatches that fail the exact --assert-equal parity check. Run flex attention eagerly (uncompiled) in --debug.deterministic mode on ROCm, which matches varlen FA parity exactly. CUDA keeps the compiled (max_autotune disabled) path unchanged.
Refresh the ROCm llama3 reference loss file to match the eager-flex curve.