Skip to content

Detect Hopper for FlashAttention 3 by compute capability, not GPU name (H200/H800/H20 silently fell back to FA2) - #13

Merged
frankjiang merged 1 commit into
amap-cvlab:mainfrom
stayinalive181:fa3-hopper-by-capability
Sep 14, 2026
Merged

frankjiang merged 1 commit into
amap-cvlab:mainfrom
stayinalive181:fa3-hopper-by-capability

Conversation

@stayinalive181

Copy link
Copy Markdown
Contributor

Problem

wan/modules/attention.py enables FlashAttention 3 only when the GPU name contains h100 or hopper:

device_name = torch.cuda.get_device_name(0).lower()
return "h100" in device_name or "hopper" in device_name

torch.cuda.get_device_name() reports product names such as NVIDIA H200, NVIDIA H800 and NVIDIA H20. None of them match, so on those Hopper parts FLASH_ATTN_3_AVAILABLE is False and every attention call silently takes the FlashAttention 2 path, even with flash_attn_interface installed. Nothing is logged, so the slowdown is easy to miss.

Change

Detect Hopper by compute capability, torch.cuda.get_device_capability(0)[0] == 9, which is what the flashattn-hopper build targets (sm_90). H100 behaviour is unchanged; H200/H800/H20 now use FA3 when it is installed; Blackwell (major 10 and 12) still returns False, as the original comment intends. The docstring says why.

Verification

On an H100 80GB SXM the flag evaluates to True before and after the change (same kernels, same output). The device-name strings above are what torch.cuda.get_device_name() returns for those parts, which is what makes the old check fail there. The same name check exists in other Wan-derived repositories; we carry the identical fix in our DreamX-World deployment.

…name

is_hopper_gpu() enabled FlashAttention 3 only when the device name contained
"h100" or "hopper". torch.cuda.get_device_name() reports "NVIDIA H200",
"NVIDIA H800" and "NVIDIA H20" for the other Hopper parts, so on those GPUs
FLASH_ATTN_3_AVAILABLE was False and every attention call silently took the
FlashAttention 2 path even with flash_attn_interface installed. Check the
compute capability (major 9 == sm_90, what flashattn-hopper targets) instead.
H100 behaviour is unchanged; Blackwell still returns False.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@frankjiang
frankjiang merged commit d314f6b into amap-cvlab:main Sep 14, 2026
@frankjiang

Copy link
Copy Markdown
Member

@stayinalive181 Thank you for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants