Skip to content

[Qwen 3] promote_dtype_for_stats flag to optionally casting RMSNorm to FP32#1825

Open
JiriesKaileh wants to merge 5 commits intomainfrom
jiries/qwen3-rms-dtype-promotion
Open

[Qwen 3] promote_dtype_for_stats flag to optionally casting RMSNorm to FP32#1825
JiriesKaileh wants to merge 5 commits intomainfrom
jiries/qwen3-rms-dtype-promotion

Conversation

@JiriesKaileh
Copy link
Collaborator

Description

Setting promote_dtype_for_stats=False keeps RMSNorm at BF16.
Setting promote_dtype_for_stats=True casts RMSNorm to FP32.

Tested perf and accuracy differences in this bug

Signed-off-by: JiriesKaileh <jiries@google.com>
@github-actions
Copy link

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

  • why is this change being made,
  • the problem being solved and any relevant context,
  • why this is a good solution,
  • some information about the specific implementation,
  • shortcomings of the solution and possible future improvements.

If the change fixes a Github issue, please include a link, e.g.,:
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure:

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have made or will make corresponding changes to any relevant documentation.

@jrplatin
Copy link
Collaborator

jrplatin commented Mar 2, 2026

Also, please include the 1k/4k and MMLU results here (with repro commands) in the Testing seciton, thanks!

Signed-off-by: JiriesKaileh <jiries@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants