RuntimeError: FlashAttention only supports Ampere GPUs or newer

[HuChundong]

please support old gpus
i have 4 2080ti 22Gb, but can not run this model

[ConneRa1]

me too,so sad about it

[shirubei]

same

[ConneRa1]

我尝试把所有用到flash_attention的地方都换成了attention,可以运行了，但爆显存了
I tried to replace all the places where flash_attention is used with attention, and it can run now, but the graphics memory is exploding
i use this commond python generate.py --task t2v-1.3B --size 832*480 --ckpt_dir ./Wan2.1-T2V-1.3B --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
File "D:\毕设\Wan2.1\wan\modules\model.py", line 146, in forward
x = attention(
File "D:\毕设\Wan2.1\wan\modules\attention.py", line 176, in attention
out = torch.nn.functional.scaled_dot_product_attention(
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 47.98 GiB. GPU 0 has a total capacity of 22.00 GiB of which 12.29 GiB is free.
I will try a few more times

[HuChundong]

我尝试把所有用到flash_attention的地方都换成了attention,可以运行了，但爆显存了 I tried to replace all the places where flash_attention is used with attention, and it can run now, but the graphics memory is exploding i use this commond python generate.py --task t2v-1.3B --size 832*480 --ckpt_dir ./Wan2.1-T2V-1.3B --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage." File "D:\毕设\Wan2.1\wan\modules\model.py", line 146, in forward x = attention( File "D:\毕设\Wan2.1\wan\modules\attention.py", line 176, in attention out = torch.nn.functional.scaled_dot_product_attention( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 47.98 GiB. GPU 0 has a total capacity of 22.00 GiB of which 12.29 GiB is free. I will try a few more times

有fork的代码吗？我有4张2080ti，可以测试多卡推理

[ConneRa1]

ok我待会fork一下

[ConneRa1]

已经提交到我fork的库上了，可以帮忙试一下，不知道为什么替换完以后，显存需求后涨这么多，我再看看代码

[ConneRa1]

测了下单卡刚好能跑40帧，泪目

[Assassinxc]

测了下单卡刚好能跑40帧，泪目

什么卡？

[ConneRa1]

测了下单卡刚好能跑40帧，泪目

什么卡？

2080ti22g

[Assassinxc]

测了下单卡刚好能跑40帧，泪目

什么卡？

2080ti22g

我也是2080ti22g单卡，提示RuntimeError: FlashAttention only supports Ampere GPUs or newer. 不支持turing架构的显卡，是怎么解决的？

[ConneRa1]

测了下单卡刚好能跑40帧，泪目

什么卡？

2080ti22g

我也是2080ti22g单卡，提示RuntimeError: FlashAttention only supports Ampere GPUs or newer. 不支持turing架构的显卡，是怎么解决的？

我将修改后的代码提交到我fork的库里了，你可以在我的仓库里找到，clone下来跑，但要记得在Python命令行多加个frame num的参数指定40帧，默认是81帧会爆显存

[Assassinxc]

测了下单卡刚好能跑40帧，泪目

什么卡？

2080ti22g

我也是2080ti22g单卡，提示RuntimeError: FlashAttention only supports Ampere GPUs or newer. 不支持turing架构的显卡，是怎么解决的？

我将修改后的代码提交到我fork的库里了，你可以在我的仓库里找到，clone下来跑，但要记得在Python命令行多加个frame num的参数指定40帧，默认是81帧会爆显存

明白了

[tgrhuy]

测了下单卡刚好能跑40帧，泪目

什么卡？

2080ti22g

我也是2080ti22g单卡，提示RuntimeError: FlashAttention only supports Ampere GPUs or newer. 不支持turing架构的显卡，是怎么解决的？

我将修改后的代码提交到我fork的库里了，你可以在我的仓库里找到，clone下来跑，但要记得在Python命令行多加个frame num的参数指定40帧，默认是81帧会爆显存

我拉取了您的代码，然后运行依然报flashAttention的错，是为啥，我的GPU是V100 下面是我执行的结果：
root@bc80774bc139:~/Wan2.1# python generate.py --task t2v-1.3B --size 832480 --ckpt_dir ./Wan2.1-T2V-1.3B --offload_model True --t5_cpu --sample_shift 8 --sample_guide_scale 6 --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage."
[2025-02-27 23:55:55,833] INFO: Generation job args: Namespace(task='t2v-1.3B', size='832480', frame_num=81, ckpt_dir='./Wan2.1-T2V-1.3B', offload_model=True, ulysses_size=1, ring_size=1, t5_fsdp=False, t5_cpu=True, dit_fsdp=False, save_file=None, prompt='Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.', use_prompt_extend=False, prompt_extend_method='local_qwen', prompt_extend_model=None, prompt_extend_target_lang='ch', base_seed=7015891214762318096, image=None, sample_solver='unipc', sample_steps=50, sample_shift=8.0, sample_guide_scale=6.0)
[2025-02-27 23:55:55,833] INFO: Generation model config: {'name': 'Config: Wan T2V 1.3B', 't5_model': 'umt5_xxl', 't5_dtype': torch.bfloat16, 'text_len': 512, 'param_dtype': torch.bfloat16, 'num_train_timesteps': 1000, 'sample_fps': 16, 'sample_neg_prompt': '色调艳丽，过曝，静态，细节模糊不清，字幕，风格，作品，画作，画面，静止，整体发灰，最差质量，低质量，JPEG压缩残留，丑陋的，残缺的，多余的手指，画得不好的手部，画得不好的脸部，畸形的，毁容的，形态畸形的肢体，手指融合，静止不动的画面，杂乱的背景，三条腿，背景人很多，倒着走', 't5_checkpoint': 'models_t5_umt5-xxl-enc-bf16.pth', 't5_tokenizer': 'google/umt5-xxl', 'vae_checkpoint': 'Wan2.1_VAE.pth', 'vae_stride': (4, 8, 8), 'patch_size': (1, 2, 2), 'dim': 1536, 'ffn_dim': 8960, 'freq_dim': 256, 'num_heads': 12, 'num_layers': 30, 'window_size': (-1, -1), 'qk_norm': True, 'cross_attn_norm': True, 'eps': 1e-06}
[2025-02-27 23:55:55,833] INFO: Input prompt: Two anthropomorphic cats in comfy boxing gear and bright gloves fight intensely on a spotlighted stage.
[2025-02-27 23:55:55,834] INFO: Creating WanT2V pipeline.
[2025-02-27 23:57:03,262] INFO: loading ./Wan2.1-T2V-1.3B/models_t5_umt5-xxl-enc-bf16.pth
[2025-02-27 23:57:11,578] INFO: loading ./Wan2.1-T2V-1.3B/Wan2.1_VAE.pth
[2025-02-27 23:57:12,038] INFO: Creating WanModel from ./Wan2.1-T2V-1.3B
[2025-02-27 23:57:14,148] INFO: Generating video ...
0%| | 0/50 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/root/Wan2.1/generate.py", line 411, in
generate(args)
File "/root/Wan2.1/generate.py", line 313, in generate
video = wan_t2v.generate(
^^^^^^^^^^^^^^^^^
File "/root/Wan2.1/wan/text2video.py", line 236, in generate
noise_pred_cond = self.model(
^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Wan2.1/wan/modules/model.py", line 564, in forward
x = block(x, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Wan2.1/wan/modules/model.py", line 298, in forward
y = self.self_attn(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/Wan2.1/wan/modules/model.py", line 146, in forward
x = attention(
^^^^^^^^^^
File "/root/Wan2.1/wan/modules/attention.py", line 149, in attention
return flash_attention(
^^^^^^^^^^^^^^^^
File "/root/Wan2.1/wan/modules/attention.py", line 113, in flash_attention
x = flash_attn.flash_attn_varlen_func(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flash_attn/flash_attn_interface.py", line 1448, in flash_attn_varlen_func
return FlashAttnVarlenFunc.apply(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flash_attn/flash_attn_interface.py", line 930, in forward
out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 1123, in call
return self._op(*args, **(kwargs or {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_library/autograd.py", line 113, in autograd_impl
result = forward_no_grad(*args, Metadata(keyset, keyword_only_args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_library/autograd.py", line 40, in forward_no_grad
result = op.redispatch(keyset & _C._after_autograd_keyset, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_ops.py", line 728, in redispatch
return self._handle.redispatch_boxed(keyset, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py", line 305, in backend_impl
result = self._backend_fns[device_type](*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_compile.py", line 32, in inner
return disable_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/torch/_library/custom_ops.py", line 337, in wrapped_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flash_attn/flash_attn_interface.py", line 170, in _flash_attn_varlen_forward
out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.varlen_fwd(
^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: FlashAttention only supports Ampere GPUs or newer.

[ConneRa1]

哦噢有地方忘改了，在wan/module/attention.py里要把FLASH_ATTN_2_AVAILABLE，FLASH_ATTN_3_AVAILABLE都置为false才会启用普通的attention，明天下午我更新一下

[tgrhuy]

哦噢有地方忘改了，在wan/module/attention.py里要把FLASH_ATTN_2_AVAILABLE，FLASH_ATTN_3_AVAILABLE都置为false才会启用普通的attention，明天下午我更新一下

嗯嗯，谢谢，我刚刚发现了，已经修改了。但32G显存还是不太行 ，直接OOM