Detect Hopper for FlashAttention 3 by compute capability, not GPU name (H200/H800/H20 silently fell back to FA2) - #13
Merged
frankjiang merged 1 commit intoSep 14, 2026
Conversation
…name is_hopper_gpu() enabled FlashAttention 3 only when the device name contained "h100" or "hopper". torch.cuda.get_device_name() reports "NVIDIA H200", "NVIDIA H800" and "NVIDIA H20" for the other Hopper parts, so on those GPUs FLASH_ATTN_3_AVAILABLE was False and every attention call silently took the FlashAttention 2 path even with flash_attn_interface installed. Check the compute capability (major 9 == sm_90, what flashattn-hopper targets) instead. H100 behaviour is unchanged; Blackwell still returns False. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
AIprogrammer
approved these changes
Sep 11, 2026
Member
|
@stayinalive181 Thank you for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
wan/modules/attention.pyenables FlashAttention 3 only when the GPU name containsh100orhopper:torch.cuda.get_device_name()reports product names such asNVIDIA H200,NVIDIA H800andNVIDIA H20. None of them match, so on those Hopper partsFLASH_ATTN_3_AVAILABLEisFalseand every attention call silently takes the FlashAttention 2 path, even withflash_attn_interfaceinstalled. Nothing is logged, so the slowdown is easy to miss.Change
Detect Hopper by compute capability,
torch.cuda.get_device_capability(0)[0] == 9, which is what the flashattn-hopper build targets (sm_90). H100 behaviour is unchanged; H200/H800/H20 now use FA3 when it is installed; Blackwell (major 10 and 12) still returnsFalse, as the original comment intends. The docstring says why.Verification
On an H100 80GB SXM the flag evaluates to
Truebefore and after the change (same kernels, same output). The device-name strings above are whattorch.cuda.get_device_name()returns for those parts, which is what makes the old check fail there. The same name check exists in other Wan-derived repositories; we carry the identical fix in our DreamX-World deployment.