Skip to content

序列并行报错,版本不兼容 #7561

@Batmana

Description

@Batmana

Describe the bug
版本兼容问题
[rank1]: File "/opt/conda/envs/megatron/lib/python3.10/site-packages/swift/trainers/sequence_parallel/ulysses.py", line 185, in init_sequence_parallel
[rank1]: from transformers.modeling_flash_attention_utils import is_flash_attn_available
[rank1]: ImportError: cannot import name 'is_flash_attn_available' from 'transformers.modeling_flash_attention_utils'

Your hardware and system info
H800,序列并行时,开启flash_attention报错。
transfomers版本:4.50.0
flash_attn: 2.7.4.post1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions