[ci][regression] 2025-12-23 ragged_attention regression AMD 350X

[xuzhao9]

Signal:

Before Dec 23, 2025 12:55 AM: (GOOD)
Triton Commit Time: 20251223005523
Triton Commit Hash: 96e075c65e2d92800598a130d5d2d1ab2f9cdf74
Latency: 0.241 ms
TFLOPS: 208.37 TFLOPS

After Dec 23, 2025 7:35 PM: (BAD)
Triton Commit Time: 20251223193547
Triton Commit Hash: d3a4c78cb231c69f652a8fbfe6a087908881206f
Latency: 0.268 ms
TFLOPS: 188.75 TFLOPS

[scxiao]

it seems like the perf regression is related to the commit: triton-lang/triton@b54cb94.
Can you set an env var TRITON_HIP_USE_ASYNC_COPY=0 to disable the async copy? then the perf numbers will come back?

[xuzhao9]

Kicked off abtest at https://github.com/meta-pytorch/tritonbench/actions/runs/21013479350

Side a: 606eebc1d464a16f716579c50c8f43328cbe97a8
Side b: b54cb947faf1d49058cc4417698080000d0add1c

[xuzhao9]

@scxiao BTW, we also noticed higher shared memory usage in our CI: https://github.com/meta-pytorch/tritonbench/actions/runs/20963766951/job/60248167621?pr=780 Is this also related to async copy?

[scxiao]

Yes, it is. You can check the description of PR: triton-lang/triton@b54cb94. turn on async copy will help our performance in long run, let me do some investigation to see if we can get the perf back even with async_copy on.

[xuzhao9]

@scxiao Yes, both perf and functionality bisection points to triton-lang/triton@b54cb947faf1d49058cc441 :

https://github.com/meta-pytorch/tritonbench/actions/runs/21088260188/job/60655638954

https://github.com/meta-pytorch/tritonbench/actions/runs/21089783484/job/60659354021

[antiagainst]

Async copy is one major feature from mi350 and we expect it to unlock more performance. But it does mean we need to adjust compilation configurations etc. Can we do that to see if you can get perf back too?

[xuzhao9]

Hi @antiagainst, by compilation configurations, do you mean the autotuning configs?

[antiagainst]

Yes, like tile size, number of buffers, and so on.

[scxiao]

I just have a try, with the tuning parameter wave_per_eu added, we can get the perf back, please check the chanes in the PR: meta-recsys/generative-recommenders#451. @xuzhao9 , could you please integrate the changes and have a try?

[scxiao]

Yes, it is. You can check the description of PR: triton-lang/triton@b54cb94. turn on async copy will help our performance in long run, let me do some investigation to see if we can get the perf back even with async_copy on.

Yes, turning on async_copy will need one more copy of shared memory compared to off. Let me take a look at the test to see.

[xuzhao9]

Hi @scxiao, can you help take a look at the test failures at https://github.com/meta-pytorch/tritonbench/actions/runs/20984344872/job/60315412074 ?

The following tests will fail due to insufficient shared memory with async copy:

• test_gpu_tritonbench_addmm
• test_gpu_tritonbench_fp8_attention
• test_gpu_tritonbench_fp8_gemm
• test_gpu_tritonbench_gemm
• test_gpu_tritonbench_grouped_gemm
• test_gpu_tritonbench_swiglu

[scxiao]

Hi @xuzhao9 , I tried the first test above, but got the following error even with async_copy off. I think it could be due to the torch version. I saw the CI build uses torch 2.11, do you know where I can get that version? Thanks

TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=256, BLOCK_N=128, EVEN_K=True, GROUP_M=4, USE_FAST_ACCUM=False, kpack=2, matrix_instr_nonkdim=16, waves_per_eu=2, num_stages=2, num_warps=4)
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0] Traceback (most recent call last):
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     result = self.fn(*self.args, **self.kwargs)
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py", line 2739, in precompile_with_captured_stdout
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     choice.precompile()
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/select_algorithm.py", line 1994, in precompile
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     self.bmreq.precompile()
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/autotune_process.py", line 659, in precompile
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     getattr(mod, self.kernel_name).precompile()
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 451, in precompile
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     self._make_launchers()
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 614, in _make_launchers
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     launchers.append(result.make_launcher())
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]                      ^^^^^^^^^^^^^^^^^^^^^^
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]   File "/opt/venv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 1757, in make_launcher
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]     (binary.metadata.num_ctas, *binary.metadata.cluster_dims)
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        E0120 15:25:42.244000 19214 torch/_inductor/select_algorithm.py:2820] [0/0] AttributeError: 'KernelMetadata' object has no attribute 'cluster_dims'

[xuzhao9]

Hi @scxiao, can you try with the latest torch nightly release?