Qwen3 235B A22B MXFP8 NaN grad norm #2096
Description
Nan grad norm is observed in the following case:
- GB200 MXFP8 Qwen3 235B A22B with config
PP8 VP3 EP32 CG(attn, moe_router, moe_preprocess) - GB300 MXFP8 Qwen3 235B A22B with config
PP4 VP12 EP32 MBS2 CG(attn, moe_router, moe_preprocess)
@dingqingy-nv observed it in 26.02 RC1 container, and @sanandaraj5597 observed it in 25.11 container too.