Repository navigation
[megatron, doc] feat: MXFP4 MoE recipe and Qwen3-30B-A3B example #8426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
1293f71
8186e63
1d346c0
ee0cb28
5bbc1e4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| # MXFP4 MoE QAT on Blackwell | ||
|
|
||
| Last updated: 11/06/2026 | ||
|
|
||
| MXFP4 is the OCP microscaling FP4 format: E2M1 elements with one E8M0 scale per 32-element block. In verl it covers | ||
| the routed experts of MoE models. The rollout serves MXFP4 expert weights, quantized on the trainer side, and | ||
| DeepGEMM quantizes the expert activations to FP8 at run time. Transformer Engine has no MXFP4 training, so training | ||
| stays in BF16; with QAT, the routed experts are fake-quantized during training: their weights exactly as the | ||
| exported rollout weights, their inputs to MXFP8, close to the rollout's FP8 activations. | ||
|
|
||
| The rollout quantization interface shared by all formats, the kernel defaults and the format details are described | ||
| in [Low-Precision Rollout and Quantization-Aware Training](qat.md). For MXFP8 end to end, see | ||
| [MXFP8 RL on Blackwell](mxfp8.md). | ||
|
|
||
| ## Requirements | ||
|
|
||
| - Blackwell GPUs for the rollout: the MXFP4-weight × FP8-activation experts are served by vLLM's DeepGEMM FP4 MoE | ||
| kernel (SM100/SM120). Hopper has no such kernel and is outside this recipe; see [qat.md](qat.md). | ||
| - Training: the Megatron-Bridge model path; QAT uses NVIDIA ModelOpt. | ||
| - Rollout: vLLM (the version verl locks). | ||
|
|
||
| ## Configuration | ||
|
|
||
| ```yaml | ||
| # MXFP4 rollout experts; dense layers stay in BF16 | ||
| actor_rollout_ref.rollout.quantization: | ||
| linear: null | ||
| moe: MXFP4 | ||
|
|
||
| # optional: MXFP4 QAT on the routed experts | ||
| actor_rollout_ref.actor.megatron.qat: | ||
| enable: True | ||
| mode: mxfp4 # mxfp4_mxfp8 also trains the dense layers with MXFP8 QAT (vLLM: linear: MXFP8) | ||
| ``` | ||
|
|
||
| Notes: | ||
|
|
||
| - `lm_head`, the token embedding, MoE routers and every dense layer stay in BF16 on the rollout and in training. | ||
| - ModelOpt installs the fake quantizers; they run verl's TileLang MXFP4 kernel, the one the trainer-side exporter | ||
| uses without dequantization, so the QAT forward and the exported rollout weights agree bit for bit (see | ||
| [qat.md](qat.md)). The experts' inputs are fake-quantized to MXFP8 (1x32 blocks), while the rollout's DeepGEMM | ||
| kernel quantizes them to FP8 in 1x128 groups with E8M0 scales. | ||
| - QAT quantizes the Transformer Engine modules of the layer spec that Megatron-Bridge builds for the model, the | ||
| grouped expert linears included, so tensor and expert parallelism can be combined. All expert weights of a grouped | ||
| linear are fake-quantized in one kernel launch. | ||
| - QAT and Transformer Engine FP8 training are mutually exclusive; the engine refuses the combination. | ||
|
|
||
| ## Example: Qwen3-30B-A3B on DAPO-Math-17k | ||
|
|
||
| [`examples/mxfp4/run_qwen3_30b_a3b_megatron.sh`](https://github.com/verl-project/verl/blob/main/examples/mxfp4/run_qwen3_30b_a3b_megatron.sh) | ||
| is the MXFP4 counterpart of the MXFP8 Qwen3-30B-A3B example, with the same data and hyper-parameters on one | ||
| 4-GPU Blackwell node (tested on 4×B300; 4×B200 untested): training TP2 × EP2 with full recompute and the whole | ||
| optimizer on the CPU, the rollout vLLM with TP2. `PRECISION` selects the arm: `mxfp4_qat` (MXFP4 rollout experts and | ||
| MXFP4 QAT), `mxfp4` (MXFP4 rollout experts only) and `bf16` (the baseline). The script reads DAPO-Math-17k in verl's | ||
| format from `TRAIN_FILE`, as | ||
| [verl-recipe's `dapo/prepare_dapo_data.sh`](https://github.com/verl-project/verl-recipe/blob/main/dapo/prepare_dapo_data.sh) | ||
|
Comment on lines
+55
to
+56
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Align the data preparation and launcher paths. The linked preparation script defaults to
|
||
| downloads it, and runs without a validation set. | ||
|
|
||
| ```bash | ||
| PRECISION=mxfp4_qat TOTAL_TRAINING_STEPS=10 bash examples/mxfp4/run_qwen3_30b_a3b_megatron.sh | ||
| PRECISION=mxfp4 TOTAL_TRAINING_STEPS=10 bash examples/mxfp4/run_qwen3_30b_a3b_megatron.sh | ||
| PRECISION=bf16 TOTAL_TRAINING_STEPS=10 bash examples/mxfp4/run_qwen3_30b_a3b_megatron.sh | ||
| ``` | ||
|
|
||
| ### Results | ||
|
|
||
| **Configuration** | ||
|
|
||
| - One 4×B300 machine for every arm, the same seed and data order, so steps pair across arms; 10 steps | ||
| - The script's hyper-parameters, as in the MXFP8 example: 64 prompts × 8 responses per step, prompts and responses up | ||
| to 1024 and 8192 tokens, dynamic batching at 20,480 tokens per GPU, learning rate 1e-6; DAPO-Math-17k deduplicated | ||
| to its 17,398 unique prompts | ||
|
Comment on lines
+71
to
+72
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Provide the deduplicated input used for the results. The linked preparation script downloads the original parquet, and the dataset loader does not deduplicate it. The documented commands therefore do not reproduce the deduplicated dataset reported here. Provide the exact input or the deduplication and ordering steps used for the runs. (raw.githubusercontent.com)
|
||
| - verl main `e41dd697` plus this PR; the versions verl locks: vLLM 0.32.0, Transformer Engine | ||
| 2.20.2, Megatron-Core 0.19.2, Megatron-Bridge 0.6.2, ModelOpt 0.47.0, TileLang 0.1.15 | ||
|
|
||
| "X/Y" is training precision X, rollout precision Y; BF16+QAT trains in BF16 with MXFP4 QAT on the routed experts. | ||
|
|
||
| | mean of steps 1–10 | BF16/BF16 | BF16/MXFP4 | BF16+QAT/MXFP4 | | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added. The Results section now links one wandb run per arm from the ten-step run on 4×B300 at the PR head: QAT, MXFP4, BF16, with the per-step table and a paired steps-5–10 table for the QAT overhead. The example logs with |
||
| |---|---|---|---| | ||
| | step time (s) | 827 | 848 | 1000 | | ||
| | generation (s) | 107 | 114 | 115 | | ||
| | `old_log_prob` (s) | 118 | 118 | 149 | | ||
| | `ref` (s) | 113 | 117 | 130 | | ||
| | `update_actor` (s) | 476 | 477 | 582 | | ||
| | `update_weights` (s) | 8 | 18 | 19 | | ||
| | `rollout_corr/kl` | 0.0018 | 0.0150 | 0.0030 | | ||
| | score | −0.195 | −0.161 | −0.183 | | ||
| | response length | 6517 | 6411 | 6438 | | ||
|
|
||
| The QAT arm's steps 1–4 overlapped another tenant on the same node (the machine's other four GPUs at 100 %), so they | ||
| are slower than the rest; its steps 5–10 ran on an idle machine. The same six steps for all three arms: | ||
|
|
||
| | mean of steps 5–10 | BF16/BF16 | BF16/MXFP4 | BF16+QAT/MXFP4 | | ||
| |---|---|---|---| | ||
| | step time (s) | 818 | 833 | 900 | | ||
| | `old_log_prob` (s) | 117 | 116 | 135 | | ||
| | `update_actor` (s) | 472 | 468 | 511 | | ||
| | `rollout_corr/kl` | 0.0018 | 0.0153 | 0.0030 | | ||
|
|
||
|  | ||
| *Per step on one 4×B300 machine; the shaded steps of the QAT arm shared the node. wandb: | ||
| [QAT](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/z52e8qr2), | ||
| [MXFP4](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/mzx5r4ys), | ||
| [BF16](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/xd7yxpnw)* | ||
|
|
||
| Results and observations: | ||
|
|
||
| - QAT cuts the rollout–trainer mismatch of the MXFP4 rollout by 80 %, to 1.7× the BF16 arm's. Per step, | ||
| `rollout_corr/kl` ranged 0.0028–0.0033 with QAT, 0.0135–0.0173 without it and 0.0016–0.0020 in BF16. | ||
| - On the six idle steps QAT costs 8 % over the MXFP4 rollout alone (900 s vs 833 s; `old_log_prob` +19 s, | ||
| `update_actor` +43 s); the ten-step means above include four contended QAT steps and show 18 %. | ||
| - The TileLang grouped export of the expert weights is why the MXFP4 arms' `update_weights` is 18–19 s here; the | ||
| earlier per-expert export cost 45–46 s. | ||
| - The MXFP4 rollout does not speed this job up: generation takes about as long as in BF16 (+6 s) and the expert | ||
| export adds about 10 s per step. | ||
| - Ten steps are enough for speed and mismatch, not for the score. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,190 @@ | ||
| #!/usr/bin/env bash | ||
| # GRPO | Qwen3-30B-A3B (MoE) | DAPO-Math-17k | Megatron training | MXFP4 routed experts | NVIDIA Blackwell (SM100+) | ||
| # | ||
| # PRECISION=mxfp4_qat trains in BF16 with ModelOpt MXFP4 QAT on the routed experts and serves the rollout with MXFP4 | ||
| # experts; PRECISION=mxfp4 quantizes the rollout's experts only (no QAT); PRECISION=bf16 runs the same job in BF16, as | ||
| # the baseline. MXFP4 covers the routed experts only: dense layers, lm_head, embeddings and routers stay in BF16. | ||
| # The rollout serves the experts through vLLM's DeepGEMM FP4 MoE kernel, which needs SM100/SM120. The defaults were | ||
| # tested on one 4xB300 node (4xB200 untested) and run without a validation set. See docs/low_precision/mxfp4_qat.md. | ||
|
|
||
| set -xeuo pipefail | ||
| export CUDA_DEVICE_MAX_CONNECTIONS=1 | ||
|
|
||
| ########################### user-adjustable ########################### | ||
| INFER_BACKEND=${INFER_BACKEND:-vllm} | ||
| PRECISION=${PRECISION:-mxfp4_qat} | ||
|
|
||
| MODEL_PATH=${MODEL_PATH:-Qwen/Qwen3-30B-A3B} | ||
| TRAIN_FILE=${TRAIN_FILE:-$HOME/data/dapo-math-17k/train.parquet} | ||
| # only read when TEST_FREQ > 0 | ||
| VAL_FILE=${VAL_FILE:-$TRAIN_FILE} | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| NNODES=${NNODES:-1} | ||
| NGPUS_PER_NODE=${NGPUS_PER_NODE:-4} | ||
|
|
||
| train_batch_size=${TRAIN_BATCH_SIZE:-64} | ||
| ppo_mini_batch_size=${PPO_MINI_BATCH_SIZE:-32} | ||
| max_prompt_length=${MAX_PROMPT_LENGTH:-1024} | ||
| max_response_length=${MAX_RESPONSE_LENGTH:-8192} | ||
| max_token_len_per_gpu=${MAX_TOKEN_LEN_PER_GPU:-20480} | ||
|
|
||
| actor_lr=${ACTOR_LR:-1e-6} | ||
| kl_loss_coef=${KL_LOSS_COEF:-0.001} | ||
| entropy_coeff=${ENTROPY_COEFF:-0} | ||
|
|
||
| actor_tp=${ACTOR_TP:-2} | ||
| actor_pp=${ACTOR_PP:-1} | ||
| actor_ep=${ACTOR_EP:-2} | ||
| # offload parameters and optimizer state while the rollout runs | ||
| offload=${OFFLOAD:-True} | ||
| # keep the whole optimizer on the CPU: Adam's FP32 state for 30B does not fit next to training on four GPUs | ||
| optimizer_cpu_offload=${OPTIMIZER_CPU_OFFLOAD:-True} | ||
|
|
||
| rollout_tp=${ROLLOUT_TP:-2} | ||
| rollout_gpu_mem_util=${ROLLOUT_GPU_MEM_UTIL:-0.4} | ||
| rollout_max_num_seqs=${ROLLOUT_MAX_NUM_SEQS:-256} | ||
| rollout_n=${ROLLOUT_N:-8} | ||
|
|
||
| total_epochs=${TOTAL_EPOCHS:-1} | ||
| total_training_steps=${TOTAL_TRAINING_STEPS:-null} | ||
| save_freq=${SAVE_FREQ:--1} | ||
| test_freq=${TEST_FREQ:--1} | ||
| # the same seed gives every arm the same data order, so steps pair across arms | ||
| seed=${SEED:-1} | ||
|
|
||
| project_name=${PROJECT_NAME:-verl_grpo_dapo_mxfp4} | ||
| experiment_name=${EXPERIMENT_NAME:-qwen3_30b_a3b_megatron_${INFER_BACKEND}_${PRECISION}} | ||
| ########################### end user-adjustable ########################### | ||
|
|
||
| ########################### parameter arrays ########################### | ||
|
|
||
| TF=actor_rollout_ref.actor.megatron.override_transformer_config | ||
| case "${PRECISION}" in | ||
| mxfp4_qat) | ||
| PRECISION_ARGS=( | ||
| # MXFP4 rollout experts: verl exports MXFP4 expert weights (1x32 groups, E8M0 scales); DeepGEMM quantizes the | ||
| # expert activations to FP8 (1x128) at run time | ||
| actor_rollout_ref.rollout.quantization.moe=MXFP4 | ||
| # ModelOpt QAT fake-quantizes the routed experts the same way during training (MXFP4 weights, MXFP8 inputs) | ||
| actor_rollout_ref.actor.megatron.qat.enable=True | ||
| actor_rollout_ref.actor.megatron.qat.mode=mxfp4 | ||
| ) | ||
| ;; | ||
| mxfp4) | ||
| PRECISION_ARGS=( | ||
| actor_rollout_ref.rollout.quantization.moe=MXFP4 | ||
| ) | ||
| ;; | ||
| bf16) | ||
| PRECISION_ARGS=() | ||
| ;; | ||
| *) | ||
| echo "PRECISION must be mxfp4_qat, mxfp4 or bf16, got '${PRECISION}'" >&2 | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
| DATA=( | ||
| algorithm.adv_estimator=grpo | ||
| algorithm.use_kl_in_reward=False | ||
| data.train_files="$TRAIN_FILE" | ||
| data.val_files="$VAL_FILE" | ||
| data.train_batch_size=${train_batch_size} | ||
| data.max_prompt_length=${max_prompt_length} | ||
| data.max_response_length=${max_response_length} | ||
| data.filter_overlong_prompts=True | ||
| data.truncation='error' | ||
| data.seed=${seed} | ||
| ) | ||
|
|
||
| MODEL=( | ||
| actor_rollout_ref.model.path="$MODEL_PATH" | ||
| ) | ||
|
|
||
| OPTIM=+actor_rollout_ref.actor.optim.override_optimizer_config | ||
| ACTOR=( | ||
| actor_rollout_ref.actor.optim.lr=${actor_lr} | ||
| actor_rollout_ref.actor.ppo_mini_batch_size=${ppo_mini_batch_size} | ||
| actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1 | ||
| actor_rollout_ref.actor.use_dynamic_bsz=True | ||
| actor_rollout_ref.actor.ppo_max_token_len_per_gpu=${max_token_len_per_gpu} | ||
| actor_rollout_ref.actor.use_kl_loss=True | ||
| actor_rollout_ref.actor.kl_loss_coef=${kl_loss_coef} | ||
| actor_rollout_ref.actor.kl_loss_type=low_var_kl | ||
| actor_rollout_ref.actor.entropy_coeff=${entropy_coeff} | ||
| actor_rollout_ref.actor.megatron.tensor_model_parallel_size=${actor_tp} | ||
| actor_rollout_ref.actor.megatron.pipeline_model_parallel_size=${actor_pp} | ||
| actor_rollout_ref.actor.megatron.expert_model_parallel_size=${actor_ep} | ||
| actor_rollout_ref.actor.megatron.param_offload=${offload} | ||
| actor_rollout_ref.actor.megatron.optimizer_offload=${offload} | ||
| ${OPTIM}.optimizer_cpu_offload=${optimizer_cpu_offload} | ||
| ${OPTIM}.overlap_cpu_optimizer_d2h_h2d=${optimizer_cpu_offload} | ||
| ${OPTIM}.use_precision_aware_optimizer=${optimizer_cpu_offload} | ||
| ${OPTIM}.optimizer_offload_fraction=1.0 | ||
| ++${TF}.recompute_granularity=full | ||
| ++${TF}.recompute_method=uniform | ||
| ++${TF}.recompute_num_layers=1 | ||
| ++${TF}.moe_grouped_gemm=True | ||
| ++${TF}.moe_token_dispatcher_type=alltoall | ||
| ++${TF}.moe_router_dtype=fp32 | ||
| ) | ||
|
|
||
| ROLLOUT=( | ||
| actor_rollout_ref.rollout.name=${INFER_BACKEND} | ||
| actor_rollout_ref.rollout.tensor_model_parallel_size=${rollout_tp} | ||
| actor_rollout_ref.rollout.gpu_memory_utilization=${rollout_gpu_mem_util} | ||
| actor_rollout_ref.rollout.max_num_seqs=${rollout_max_num_seqs} | ||
| actor_rollout_ref.rollout.n=${rollout_n} | ||
| actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=2 | ||
| actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=True | ||
| actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=${max_token_len_per_gpu} | ||
| ) | ||
|
|
||
| REF=( | ||
| actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=2 | ||
| actor_rollout_ref.ref.log_prob_use_dynamic_bsz=True | ||
| actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=${max_token_len_per_gpu} | ||
| actor_rollout_ref.ref.megatron.tensor_model_parallel_size=${actor_tp} | ||
| actor_rollout_ref.ref.megatron.pipeline_model_parallel_size=${actor_pp} | ||
| actor_rollout_ref.ref.megatron.expert_model_parallel_size=${actor_ep} | ||
| actor_rollout_ref.ref.megatron.param_offload=${offload} | ||
| ) | ||
|
|
||
| TRAINER=( | ||
| trainer.critic_warmup=0 | ||
| trainer.logger='["console","wandb"]' | ||
| trainer.project_name=${project_name} | ||
| trainer.experiment_name=${experiment_name} | ||
| trainer.n_gpus_per_node=${NGPUS_PER_NODE} | ||
| trainer.nnodes=${NNODES} | ||
| trainer.save_freq=${save_freq} | ||
| trainer.test_freq=${test_freq} | ||
| trainer.val_before_train=False | ||
| trainer.total_epochs=${total_epochs} | ||
| trainer.total_training_steps=${total_training_steps} | ||
| ) | ||
|
|
||
| EXTRA=( | ||
| model_engine=megatron | ||
| ) | ||
|
|
||
| ########################### launch ########################### | ||
| # uv (set VERL_USE_UV=0 for system python): GPU vllm/sglang × megatron run the driver and every Ray worker | ||
| # (runtime_env.py_executable) through `uv run` on the matching extras of the committed uv.lock; | ||
| # other backends / NPU fall back to ambient python. Run from the verl repo root. | ||
| LAUNCH=(python3) | ||
| RAY=(ray_kwargs.ray_init.runtime_env.py_executable=null) | ||
| if [ "${VERL_USE_UV:-1}" != 0 ] && [ "${DEVICE:-gpu}" = gpu ] && { [ "${INFER_BACKEND}" = vllm ] || [ "${INFER_BACKEND}" = sglang ]; }; then | ||
| LAUNCH=(uv run --frozen --all-packages --extra "${INFER_BACKEND}" --extra megatron python3) | ||
| RAY=(ray_kwargs.ray_init.runtime_env.py_executable="uv -v run --frozen --all-packages --extra ${INFER_BACKEND} --extra megatron") | ||
| fi | ||
| "${LAUNCH[@]}" -m verl.trainer.main_ppo \ | ||
| "${DATA[@]}" \ | ||
| "${MODEL[@]}" \ | ||
| "${ACTOR[@]}" \ | ||
| "${ROLLOUT[@]}" \ | ||
| "${REF[@]}" \ | ||
| "${TRAINER[@]}" \ | ||
| "${PRECISION_ARGS[@]}" \ | ||
| "${EXTRA[@]}" \ | ||
| "${RAY[@]}" \ | ||
| "$@" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Script executed:
Repository: verl-project/verl
Length of output: 4208
Nest the rollout and QAT settings under
actor_rollout_ref.In YAML,
actor_rollout_ref.rollout.quantization:is one literal key. It does not create nestedrolloutandquantizationmappings. Copying this block into a config file leaves rollout quantization and QAT unset, and the run gives no error.Suggested fix