Skip to content

[GLM4MoE] Set attention_softmax_in_fp32 and bf16 defaults in GLMMoEMo…#4314

Open
zhanghonggeng wants to merge 1 commit intoPaddlePaddle:developfrom
zhanghonggeng:eager1
Open

[GLM4MoE] Set attention_softmax_in_fp32 and bf16 defaults in GLMMoEMo…#4314
zhanghonggeng wants to merge 1 commit intoPaddlePaddle:developfrom
zhanghonggeng:eager1

Conversation

@zhanghonggeng
Copy link
Copy Markdown
Contributor

@zhanghonggeng zhanghonggeng commented Apr 17, 2026 •
edited
Loading

…delProvider

Before submitting

  • Lint code. If there are lint issues, please format the code first.
# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install

# Process previous code files separately
pre-commit run --file XXXX.py
  • Add test cases into tests folder. If there are codecov issues, please add tests cases first.

PR types

PR changes

Description

attention_softmax_in_fp32 和 bf16 参数默认设置为 True。

@paddle-bot
Copy link
Copy Markdown

paddle-bot Bot commented Apr 17, 2026

Thanks for your contribution!

From00
From00 reviewed Apr 20, 2026
View reviewed changes
Comment thread paddleformers/transformers/glm4_moe/modeling.py
moe_grouped_gemm: bool = False

attention_softmax_in_fp32: bool = True
Copy link
Copy Markdown
Collaborator

@From00 From00 Apr 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个是不是可以默认启用就行，不需要通过开关控制？

From00
From00 approved these changes Apr 20, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@From00 From00 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhanghonggeng
Copy link
Copy Markdown
Contributor Author

zhanghonggeng commented Apr 20, 2026

/re-run all-failed

3 similar comments
@zhanghonggeng
Copy link
Copy Markdown
Contributor Author

zhanghonggeng commented Apr 20, 2026

/re-run all-failed

@zhanghonggeng
Copy link
Copy Markdown
Contributor Author

zhanghonggeng commented Apr 20, 2026

/re-run all-failed

@zhanghonggeng
Copy link
Copy Markdown
Contributor Author

zhanghonggeng commented Apr 20, 2026

/re-run all-failed

lugimzzz
lugimzzz approved these changes Apr 21, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@lugimzzz lugimzzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhanghonggeng
Copy link
Copy Markdown
Contributor Author

zhanghonggeng commented Apr 22, 2026

/re-run all-failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@From00 From00 From00 approved these changes

@lugimzzz lugimzzz lugimzzz approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhanghonggeng @From00 @lugimzzz