Repository navigation
[megatron]feat: Deepseek v4 RL MXFP4 experts qat support - #7895
Draft
sophiayyya wants to merge 28 commits into
Draft
sophiayyya wants to merge 28 commits into
sophiayyya wants to merge 28 commits into
Conversation
The DeepSeek-V4 report describes FP4 quantization-aware training applied to MoE expert weights during post-training. This adds `qat.mode=mxfp4_experts`: a weight-only ModelOpt fake-quant onto the MXFP4 grid (E2M1 values, one E8M0 power-of-two scale per 32 contiguous K elements) scoped to routed experts via `*mlp.experts*weight_quantizer`, with `algorithm=None` because dynamic block quantization derives its scale per call and ModelOpt asserts it is never calibrated. The GEMMs keep running in real FP8 through TE's blockwise autocast rather than falling back to a simulated BF16 matmul. ModelOpt refuses to run any quantizer under `fp8_autocast` -- a guard that keeps pure simulation faithful, but here the stacking is the point (FP4 weights feeding a real FP8 GEMM, which is what the checkpoint's own mixed MXFP4/FP8 layout implies). `qat.bypass_te_fp8_assert` lifts it at runtime without patching ModelOpt. Activations are quantized once, by TE; ModelOpt's input_quantizer stays disabled so they are not quantized twice. Two adjacent fixes were required to make QAT usable on such a model: - The ModelOpt layer spec is no longer forced on providers that declare an `experimental_attention_variant` (DeepSeek-V4 "dsv4_hybrid", GLM5 "dsa", Qwen3-Next "gated_delta_net"). Those install their own heterogeneous spec in `provider_bridge()`, and the generic ModelOpt GPT spec would silently replace it and build a different model. ModelOpt swaps QuantModules by class, so the spec change is not needed for quantization to apply. - `export_qat_weights` is gated to the NVFP4 modes it actually implements. For `mxfp4_experts` the bridge already re-quantizes routed experts to MXFP4 on every weight sync, so the training forward and the rollout weights land on one grid without a second exporter. `apply_qat_to_modules` now fails fast if the experts-only config did not land where intended: it requires at least one enabled routed-expert weight quantizer and no other enabled quantizer (a mis-ordered cfg would leave every remaining linear on ModelOpt's default per-tensor INT8), and logs whether the fake-quant is the identity on the loaded weights -- it is, exactly, when the checkpoint already stores those experts on the MXFP4 grid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: OpenAI Codex <noreply@openai.com>
Mirrors run_deepseek_v4_flash_megatron.sh with TE fp8 blockwise training, fp8 vLLM rollout and the optional `mxfp4_experts` QAT (off by default). Documents why the FP4 step is simulated on Hopper while the FP8 GEMM is not, and why filter_overlong_prompts must stay off for DeepSeek-V4. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Preserve native Megatron checkpoint methods around stateless expert MXFP4 quantization, and restore offloaded optimizer shards before loading HDO state. Add independent length-only and MXFP4 expert QAT recipes with a 4096-token response budget, 40 total steps and validation every 10 steps. Require model and dataset locations from the environment and use portable output defaults. Validation: checkpoint regressions passed (6 and 7 tests); the original length-only and QAT runs each completed 40 steps including save/resume. Published launch arguments match both completed experiments at 20/40 steps. Shell syntax, required-input checks and credential/path scans passed. Two pre-existing repository checks are excluded: stale generated reference configuration and naming of an older example. New example names pass. Co-authored-by: OpenAI Codex <noreply@openai.com>
Remove the two checkpoint regression test files from this branch as requested. Co-authored-by: OpenAI Codex <noreply@openai.com>
Record the HDO FP32-shard detach prerequisite, upstream fix and validation provenance in the standalone MXFP4 QAT script. Remove the length-only recipe and its stale reference from the QAT header. Validation: bash -n and git diff --check pass; the executable QAT script body is byte-for-byte unchanged. Co-authored-by: OpenAI Codex <noreply@openai.com>
Rename the original DeepSeek-V4-Flash QAT example to use the required Megatron backend suffix, preserving its contents. Refresh the generated Megatron reference config with the existing bypass_te_fp8_assert defaults. Co-authored-by: Codex
|
Sophia Yang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
7 of 8 tasks
4 of 8 tasks
wuxibin89
pushed a commit
that referenced
this pull request
Sep 29, 2026
…8062) ### What does this PR do? The FP8 end-to-end section of `docs/low_precision/fp8.md` lists `NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1` as a general requirement. It only applies on Hopper. On Blackwell (SM100+), Transformer Engine has no native block-scaled GEMM. It runs the block-wise recipe through MXFP8 GEMMs, whose E8M0 scales hold only an exponent. So both of its block-quantize kernels assert power-of-two scales on SM100+, and a run with the variable set stops at the first FP8 quantization: ``` transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:534 in function quantize_transpose_vector_blockwise: Assertion failed: pow2_scale. On Blackwell and newer, the FP8 block scaling recipe is emulated with MXFP8, which requires using power of two scaling factors. ``` This is the failure reported in #6172 (8×B200) when following this doc and the verl-recipe FP8 E2E script. This PR marks the variable as Hopper-only and says that on Blackwell it must stay unset; power-of-two scales are the Transformer Engine default. Docs only. Refs #6172. This does not fix #6172: that issue also reports garbled FP8 rollouts on Blackwell once the variable is removed, and this PR does not address that. ### Checklist Before Starting - [x] Search for similar PRs: - [NVTE_FP8_BLOCK_SCALING_FP32_SCALES](https://github.com/verl-project/verl/pulls?q=NVTE_FP8_BLOCK_SCALING_FP32_SCALES) - [blockwise Blackwell](https://github.com/verl-project/verl/pulls?q=blockwise+Blackwell) - No PR changes this requirement. Related: #6172 (the issue above); #7895 (sets the variable in its DeepSeek-V4 example scripts, does not touch this doc); #7519 (MXFP8 training, does not change this line). - [x] Format the PR title as `[{modules}] {type}: {description}` ### Test Documentation-only change. - `pre-commit run --files docs/low_precision/fp8.md`: all hooks pass, including "Check docs Last updated info". - Reproduced on 1×B200 (SM 10.0, torch 2.11.0+cu130, CUDA 13.0, Transformer Engine 2.16.1). `te.Linear` (2048→2048) and `te.GroupedLinear` (4 × 2048→768) ran forward and backward under `Float8BlockScaling(fp8_format=Format.E4M3)` and were compared with a bf16 run of the same module: | `NVTE_FP8_BLOCK_SCALING_FP32_SCALES` | `te.Linear` out / dgrad / wgrad rel. err | `te.GroupedLinear` out / dgrad / wgrad rel. err | |---|---|---| | unset (power-of-two scales) | 0.0377 / 0.0423 / 0.0462 | 0.0376 / 0.0475 / 0.0510 | | `1` | `RuntimeError`: the `pow2_scale` assertion above | same | The same modules under `MXFP8BlockScaling` give the same errors as the unset row, with or without the variable. Source references (Transformer Engine 2.16.1, commit c9877beb): - `transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:533-536` and `transformer_engine/common/transpose/quantize_transpose_square_blockwise.cu:490-493`: `if (sm_arch() >= 100) NVTE_CHECK(pow2_scale, ...)`. - `transformer_engine/pytorch/csrc/extensions/gemm.cpp:289-300`: "Emulate the FP8 block scaling recipe with MXFP8 on Blackwell and newer as it is not natively supported by cublasLt". - `transformer_engine/common/swizzle/swizzle_block_scaling.cu:116-118, 205-206`: each FP32 block scale becomes an E8M0 scale by keeping only its exponent byte, `(sf >> 23) & 0xFF`. This is why the scales must be powers of two. ### API and Usage Example None. ### Design & Code Changes `docs/low_precision/fp8.md`: - The requirement line now says that the variable is Hopper-only, why it fails on Blackwell, and what to do there. - "Last updated" is bumped. The verl-recipe script `low_precision/run_dapo_qwen3_moe_30b_megatron_fp8e2e.sh` also exports the variable unconditionally. That lives in the recipe repository and can get a matching note separately. ### Checklist Before Submitting - [ ] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/verl-project/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: documentation-only change; the behaviour it documents was checked on B200 as described under Test. - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1). - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit: not applicable. AI assistance: this PR was prepared with AI assistance (Claude Code). I reviewed the change and the evidence above, and the checks listed under Test were run for this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
6 of 7 tasks
5 of 6 tasks
This was referenced Oct 29, 2026
wuxibin89
added a commit
that referenced
this pull request
Nov 2, 2026
…h live refit (#7971) ### What does this PR do? Add end-to-end native NVFP4 W4A4 training and rollout on NVIDIA Blackwell GPUs, demonstrated with Qwen3-30B-A3B-Base DAPO using Megatron/Transformer Engine and vLLM. This contributes to Sophia's [low-precision roadmap #5972](#5972), specifically **NVFP4 real quant** and **NVFP4 training in Megatron**. In brief: - **Training:** Transformer Engine runs selected routed-expert MLPs in NVFP4 with per-token activation scaling and dequantized backward. Persistent policy weights remain BF16, with full optimizer state. - **Weight synchronization:** following the trainer-side quantization boundary of #8287, the trainer exports the routed experts as NVFP4 with `moe: NVFP4` and `moe_activation: nvfp4_per_token` after each policy update. - **Rollout:** vLLM loads the NVFP4 expert weights and quantizes activations per token, matching training. Refit preserves the storage referenced by CUDA graphs. Attention and the configured boundary-layer MLPs remain BF16. The Qwen3 example keeps the first two and last four MLP layers in BF16 and applies NVFP4 to the remaining 42 routed-expert MLP layers. ### Checklist Before Starting - [x] Search for similar PRs: [NVFP4](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+NVFP4), [roadmap #5972](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+5972+in%3Abody). - [x] Format the PR title as `[{modules}] {type}: {description}`. Related work: #6455 targets ModelOpt/TRT-LLM synchronization; #6992 provides FSDP W4A8 simulation; #7895 targets DeepSeek MXFP4 on TE FP8; #7519 adds MXFP8 training and rollout. #8287 adds ModelOpt NVFP4 QAT with static activation scales; this PR adds native TE NVFP4 training with per-token activation scaling, on the same export path. ### Test Unit tests (CPU and GB200) pass; scoped Ruff, formatting and lockfile checks pass. The trainer-side export is bit-identical to vLLM's online `nvfp4_per_token` quantization, and 6-step runs on Qwen3-30B-A3B (8 × 4 GB200) and a small DeepSeek-V3-layout model pass. #### Long run BF16 and W4A4 runs of the example script below (Qwen3-30B-A3B-Base, DAPO, R3, token-level TIS) on 8 nodes × 4 GB200 GPUs, 700 steps with checkpoint resume across jobs. **W4A4 vs BF16:** **Training score (`critic/score/mean`)** <img width="1813" height="957" alt="image" src="https://github.com/user-attachments/assets/d0693c0b-6789-4285-8658-44a885d72a72" /> **Entropy (`actor/entropy`)** <img width="1814" height="952" alt="image" src="https://github.com/user-attachments/assets/e63b552c-fe9b-4b23-9751-62c38976732c" /> **Train–rollout mismatch (`rollout_corr/kl`)** <img width="1811" height="952" alt="image" src="https://github.com/user-attachments/assets/13131fe5-c670-41ea-91e1-68e5fff123ee" /> **Gradient norm (`actor/grad_norm`)** <img width="1813" height="953" alt="image" src="https://github.com/user-attachments/assets/b121c562-b6f7-4432-8a7e-ed5768d78e69" /> **Performance:** at matched response length, NVFP4 rollout generation is about 1.3× faster than BF16 once the mean response exceeds ~2.5k tokens, giving roughly 1.2× shorter end-to-end steps; NVFP4 training itself is still slower than BF16 in Transformer Engine. ### API and Usage Example With a Ray cluster running on eight nodes, provide model and dataset paths shared by all workers: ```bash MODEL_PATH=/shared/models/Qwen3-30B-A3B-Base \ TRAIN_FILE=/shared/data/train.parquet \ TEST_FILE=/shared/data/validation.parquet \ EXP_NAME=qwen3-native-nvfp4 \ PRECISION_MODE=real_nvfp4 \ bash examples/real_nvfp4/run_qwen3_30b_megatron.sh ``` Use `PRECISION_MODE=bf16` for the baseline. NVFP4 training is enabled like FP8 training, through `actor_rollout_ref.actor.megatron.override_transformer_config` (`fp4: e2m1`, `fp4_recipe: nvfp4`, and `first_last_layers_bf16` with `num_layers_at_start_in_bf16` / `num_layers_at_end_in_bf16`), and rollout through `actor_rollout_ref.rollout.quantization.moe=NVFP4` with `moe_activation=nvfp4_per_token`, whose `ignore` list keeps the same layers in BF16; the trainer checks that the two agree. The example uses the standard verl PPO entry point, parquet datasets and built-in reward manager, with the DAPO recipe (clip-higher, token-level loss, dynamic sampling and the overlong penalty), R3 routing replay and token-level TIS. See [the example guide](https://github.com/zhangyimi/verl/blob/feat/native-nvfp4/examples/real_nvfp4/README.md) for configuration and runtime setup. ### Design & Code Changes - **Training precision:** the Megatron per-module recipe (attention BF16, routed-expert MLPs NVFP4, carved-out layers BF16) is generated from the configuration rather than maintained as a file. After model construction, the effective precision of every attention and expert linear layer is checked against it. Packed THD sequences use the TE alignment for FP4 as for FP8. - **Rollout config and refit:** `moe_activation=nvfp4_per_token` selects a compressed-tensors scheme (NVFP4 weights, fully dynamic NVFP4 inputs). The trainer quantizes the experts with its own Triton kernel, sharing one global scale across `gate_proj`/`up_proj`, bit-identical to vLLM's online `nvfp4_per_token` quantization. In vLLM the scheme selects FlashInfer TRT-LLM with per-token activations; refits load each bucket in place into the expert storage captured by CUDA graphs and post-process each MoE layer, so the extra memory is one MoE layer. - **Routing replay:** use vLLM’s native fused-MoE routing capture and align replay data with FP4 training padding. - **Runtime compatibility:** Transformer Engine 2.20.2 from the verl wheelhouse. No vLLM source patch is needed. The validated Megatron + vLLM stack uses: | Component | Version | |---|---| | PyTorch | 2.13 / CUDA 13 | | vLLM | 0.32.0 | | Transformers | 5.16.1 | | Megatron-Core | 0.19.2 | | Megatron-Bridge | 0.6.2 | | Transformer Engine | 2.20.2 | | FlashInfer | 0.7.0.post1 | | CUTLASS DSL / QuACK | 4.8.0 / 0.6.5 | SGLang end-to-end validation remains follow-up work. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [ ] Run the complete all-files pre-commit suite. Focused lint, formatting, shell and lockfile checks have passed. - [x] Add usage, design and runtime compatibility documentation. - [x] Add focused unit and end-to-end coverage. Project CI integration remains to be completed. - [x] Request project CI and review. - [x] Check recipe coordination: the recipe submodule reference is unchanged; no companion recipe PR is required. AI assistance: OpenAI Codex and Anthropic Claude assisted with implementation, testing and this PR description. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Codex <noreply@openai.com> Co-authored-by: wuxibin <wuxibin@bytedance.com>
7 of 8 tasks
wuxibin89
added a commit
that referenced
this pull request
Nov 2, 2026
### What does this PR do? Add shared TileLang quantization kernels for FP8_BLOCK, MXFP8, and MXFP4 QAT and parameter export. QAT uses fused quantize-dequantize with straight-through gradients, while export uses the same kernels without dequantization so values and scales follow the same numerical recipe. NVFP4 QAT uses triton kernel in #8494. This overlaps with [#7895](#7895), which adds DeepSeek-specific routed-expert MXFP4 weight-only QAT using ModelOpt's CUDA extension. This PR instead provides reusable TileLang kernels for FP8_BLOCK, MXFP8, and MXFP4, applies them to both weight/activation fake quantization and checkpoint export, and keeps the existing QAT mode interface. AI assistance was used for implementation, numerical comparison, testing, debugging, and preparation of this PR description. The human submitter directed the numerical design and must review every changed line and validation result before requesting maintainer review. ### Checklist Before Starting - [x] Search for similar PRs: [QAT quant kernel search](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+QAT+quant+kernel), [MXFP4 QAT search](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+MXFP4+QAT). - [x] Format the PR title as `[{modules}] {type}: {description}`. ### Test ```bash # FP8, MXFP8, MXFP4 kernels, export integration, and ModelOpt integration pytest -q tests/utils/test_qat_fp8_on_gpu.py tests/utils/test_qat_fp4_on_gpu.py tests/utils/test_qat_export_kernels_on_gpu.py tests/utils/test_modelopt_mxfp8_on_gpu.py # 51 passed # CPU CI-equivalent environment uv run --frozen --all-packages --extra cpu pytest -s -x --asyncio-mode=auto tests/utils/real_nvfp4 # 24 passed, 1 skipped uvx --with hydra-core pre-commit run --all-files --show-diff-on-failure --color=always # all hooks passed git diff --check origin/main...HEAD # passed ``` FP8 and MXFP4 quantized values, scales, and fused QDQ outputs were also compared bitwise against the DeepSeek-V4.1 inference reference kernels. ### API and Usage Example ```python from verl.utils.qat.fp4 import fp4_quant from verl.utils.qat.fp8 import fp8_quant fp8_weight, fp8_scale = fp8_quant(weight, (128, 128)) mxfp8_weight, mxfp8_scale = fp8_quant(weight, (1, 32), scale_fmt="e8m0") mxfp4_weight, mxfp4_scale = fp4_quant(weight) # Fused fake quantization used by QAT: dequantized_activation = fp8_quant(activation, (1, 32), scale_fmt="e8m0", dequant=True) ``` ### Design & Code Changes - Add 2D FP8_BLOCK and 1D MXFP8 TileLang kernels with optional fused dequantization and FP32/E8M0 scale output. - Add a packed 1x32 MXFP4 TileLang kernel with optional fused dequantization. - Route ModelOpt FP8_BLOCK, MXFP8, and MXFP4 fake quantizers through the new kernels while retaining straight-through gradients. - Route compressed-tensors FP8_BLOCK, MXFP8, and MXFP4 CUDA weight export through the same kernels; retain CPU and NVFP4 fallbacks. - Add numerical and export integration tests and document the shared QAT/export behavior. - Upgrade ModelOpt to 0.47.0 and update the lockfile. - Isolate Megatron stubs in the Muon CPU tests so they do not pollute later tests through `sys.modules`. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [x] Add unit tests to cover the kernels, QAT integration, and export integration. - [ ] Once the PR is ready for CI, send a message in the `ci-request` channel. - [x] This PR is not related to the `recipe` submodule.
…eam/main) - mxfp4_experts moves from verl/utils/modelopt to verl/utils/qat/modelopt.py as one more QAT mode - bypass_te_fp8_assert config knob removed: the mode itself lifts modelopt's TE-FP8 guard, and the QAT x TE-FP8 mutual-exclusion check exempts it - DeepSeek-V4 native fp8 quantization_config accepts qat.mode=mxfp4_experts in dispatch_rollout_weight_quantizer - drop rollout.quantization=fp8 and the length-only recipe; HDO resume fix already upstream (verl-project#8235) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… in the DSv4 QAT recipe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in the DSv4 QAT recipe Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Covers the quantize config (routed-expert weight_quantizers only), the rollout dispatch pairing with DeepSeek-V4's native fp8 quantization_config, the on-GPU fake-quant (idempotent on the MXFP4 grid, other linears untouched) and the fail-fast layout check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…; test imports the mcore patch first _check_mxfp4_experts matched ".mlp.experts." with a leading dot, so a routed-expert quantizer on a module rooted at mlp.experts (a bare MoE block, as in the unit test) was reported as unexpected. The test also imports verl.models.mcore before modelopt, as the Megatron engine does, so its megatron plugin does not trip over flash_attn.cute on the mcore dev branch, and builds the qat namespace directly instead of importing the worker config package. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dev Megatron-Core maps apply_dsa_kernel_fusion=True to dsa_kernel_backend=cudnn, which hard-requires flash_mla (nv_dev) and fails fast when it is not installed. Expose the switch so the recipe can run on the PyTorch fallback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…antizers
modelopt 0.47 quantizes a TE grouped linear with a GroupedQuantizer whose per-expert quantizers are
named `...linear_fc1.weight_quantizer.{i}`; _check_mxfp4_experts only accepted names ending in
`weight_quantizer`, so on DeepSeek-V4 it saw 0 expert quantizers and raised. Add a single-GPU test
on a real Megatron-Core MoE (TE grouped GEMM) that pins this layout.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n it applies Megatron-Core dev removed dist_checkpointing.strategies.async_utils (NVIDIA/Megatron-LM#7052), so the unconditional import made `import verl.models.mcore` fail on any newer mcore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CPU offload Megatron-Core dev asserts that optimizer_cpu_offload (HybridDeviceOptimizer) runs on the precision-aware optimizer path. Main grads and Adam moments stay fp32 by default, so numerics are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eek-V4) models - RouterReplay accepts is_mtp_layer (NVIDIA/Megatron-LM#7310 passes it from TopKRouter) and every router is registered once: drop the replay object Megatron-Core builds itself before adding ours. - The hash-routing wrapper passes new arguments (dense_output, verl-project#7450) through. - is_moe_layer reads hybrid_layer_pattern when present: Megatron-Bridge >= 0.7 builds DeepSeek-V4 as attention + MoE hybrid sublayers, so only the 'E' positions own a router. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On Megatron-Core dev, mcore_fsdp_adapter.FullyShardedDataParallel is a factory dispatching to FullyShardedDataParallelV1/V2, so using it in isinstance raised TypeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… that has it HybridModel (DeepSeek-V4 via Megatron-Bridge >= 0.7) exposes Megatron's output_processor hook, so it needs no GPT-specific forward patch. When a model can be neither hooked nor patched, the engine now disables fused kernels instead of calling the unpatched forward with fused-only kwargs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stale rollout.qat - PIPELINE_MODEL_PARALLEL_LAYOUT may be empty (PP=1 / provider-built layout). - ROLLOUT_MOE_BACKEND: verl defaults quantized models to DeepGEMM, whose MXFP4 MoE kernel is Blackwell-only; use marlin on Hopper. - actor_rollout_ref.rollout.qat no longer exists on RolloutConfig. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atron-Bridge's helper Megatron-Bridge >= 0.7 builds DeepSeek-V4 as attention + MoE hybrid sublayers. With PP > 1 the hybrid_layer_pattern has to be split at stage boundaries as well, which a logical-layer pipeline_model_parallel_layout string cannot express. When no layout is given, call Bridge's set_deepseek_v4_pipeline_model_parallel_layout (what its own recipes use). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stage Megatron-Core requires hash-routed MoE layers on the stage that owns input_ids. Bridge's helper splits logical layers evenly, which breaks this when the first stage gets fewer blocks than num_hash_layers (e.g. 16 layers over PP8). Grow the first stage when needed (only then; the full 43-layer Flash model keeps the even split) and pass the counts to the helper. CPU unit test included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wuxibin89
pushed a commit
that referenced
this pull request
Nov 10, 2026
### What does this PR do? Adds the MXFP4 recipe from the low-precision roadmap (#8186): the MXFP4 counterpart of the MXFP8 recipe in #8127, for the routed experts of MoE models. - `examples/mxfp4/run_qwen3_30b_a3b_megatron.sh`: GRPO on Qwen3-30B-A3B with DAPO-Math-17k on one 4-GPU Blackwell node (tested on 4×B300; 4×B200 untested). It uses the data, hyper-parameters and layout of `examples/mxfp8/run_qwen3_30b_a3b_megatron.sh` (training TP2 × EP2, full recompute, optimizer on the CPU; rollout vLLM TP2). `PRECISION` selects the arm: - `mxfp4_qat`: MXFP4 rollout experts (`rollout.quantization.moe=MXFP4`) and ModelOpt MXFP4 QAT on the routed experts (`qat.mode=mxfp4`) - `mxfp4`: MXFP4 rollout experts only - `bf16`: the baseline - `docs/low_precision/mxfp4_qat.md`: requirements, configuration, notes and the example; linked from `docs/index.rst`. The page is named after QAT because Transformer Engine has no MXFP4 training: the routed experts are fake-quantized, everything else trains in BF16. - The example sets `data.seed` (default 1), so every arm sees the same data order. Results, ten GRPO steps per arm on one 4×B300 machine (details in the doc page and under Test): QAT cuts the rollout–trainer mismatch of the MXFP4 rollout by 80 % (`rollout_corr/kl` 0.0150 → 0.0030; BF16 0.0018) and makes a step 6 % longer than the MXFP4 rollout alone on an idle machine (900 s vs 848 s; BF16 827 s). The QAT cost assumes the grouped QAT and export kernels of #8498 and #8500, both merged; this PR no longer depends on an open PR. ### Checklist Before Starting - [x] Search for similar PRs: no open PR adds an MXFP4 recipe or example (searched `mxfp4`; #7895 is DeepSeek-V4 MXFP4 expert QAT, #8398 the Hopper MoE backend default). - [x] Format the PR title as `[{modules}] {type}: {description}`. ### Test ```bash python3 tests/special_sanity/check_docs_time_info.py # ✅ python3 tests/special_sanity/check_example_naming.py --root examples # ✅ 115 scripts python3 tests/special_sanity/check_uv_gpu_only.py # ✅ python3 tests/special_sanity/validate_structure.py # ✅ ``` All three arms of the new script compose (`--cfg job`, `VERL_USE_UV=0`): the QAT arm resolves to `qat.enable=true`, `qat.mode=mxfp4`, `rollout.quantization.moe=MXFP4`, TP2 × EP2 for actor and ref. GPU, B300 (SM103). The versions verl locks: vLLM 0.32.0, Transformer Engine 2.20.2, Megatron-Core 0.19.2, Megatron-Bridge 0.6.2, ModelOpt 0.47.0, TileLang 0.1.15. 1. verl main 174888f, one GPU, `scripts/rollout_logprob_diff.py --r3-router-replay` on a small random Qwen3-MoE (4 layers, 32 experts, top-4), run before the QAT path moved to the TileLang kernels: | Arm | vLLM MoE backend | k3 KL | |---|---|---| | BF16 | FlashInfer TRT-LLM (unquantized) | 1.0e-5 | | MXFP4 rollout experts | `DEEPGEMM_MXFP4` | 1.0e-3 | | MXFP4 rollout experts + MXFP4 QAT | `DEEPGEMM_MXFP4` | 4.0e-5 | QAT brings the training forward 26× closer to the MXFP4 rollout. 2. Four GPUs, the configuration of this script with `PRECISION=mxfp4_qat` (host-specific cache paths added), one GRPO step on Qwen3-30B-A3B (TP2 × EP2): QAT builds on the Transformer Engine spec with tensor and expert parallelism combined, weight sync and the step complete with `rollout_corr/kl` 0.0030. With the grouped kernels this first step takes 1096 s (generation 121 s, `old_log_prob` 170 s, `ref` 110 s, `update_actor` 635 s, `update_weights` 18 s). 3. Four GPUs, the three arms of this script for 10 steps each, one after the other on the same machine, verl main e41dd69 plus this PR. Host-specific cache paths, Ray `num_cpus=64`, the venv's python started directly instead of through `uv run`, the console, file and wandb loggers, DAPO-Math-17k deduplicated to its 17,398 unique prompts as in #8127. | mean of steps 1–10 | BF16/BF16 | BF16/MXFP4 | BF16+QAT/MXFP4 | |---|---|---|---| | step time (s) | 827 | 848 | 1000 | | generation (s) | 107 | 114 | 115 | | `old_log_prob` (s) | 118 | 118 | 149 | | `ref` (s) | 113 | 117 | 130 | | `update_actor` (s) | 476 | 477 | 582 | | `update_weights` (s) | 8 | 18 | 19 | | `rollout_corr/kl` | 0.0018 | 0.0150 | 0.0030 | | score | −0.195 | −0.161 | −0.183 | | response length | 6517 | 6411 | 6438 | The QAT arm's steps 1–4 overlapped another tenant on the same node (the machine's other four GPUs at 100 %); its steps 5–10 ran on an idle machine: step time 900 s, `old_log_prob` 135 s, `update_actor` 511 s, `rollout_corr/kl` 0.0030. wandb: [QAT](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/z52e8qr2), [MXFP4](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/mzx5r4ys), [BF16](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/xd7yxpnw). ### Design & Code Changes - New example script; shares its parameter arrays and launch block with the MXFP8 example, with `PRECISION` arms for MXFP4. - New doc page with the results; no code changes. - The example sets `data.seed=${SEED:-1}`. With `data.seed` unset, as in the MXFP8 example, the shuffle is unseeded and each run sees a different data order. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply pre-commit checks. - [x] Add / Update the documentation. - [ ] Add unit or end-to-end test(s): example script only, no CI test; the three-arm E2E run is under Test. AI assistance was used for this change. I reviewed every changed line and ran the checks above. --------- Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
sophiayyya
force-pushed
the
mxfp4-experts-qat
branch
from
November 11, 2026 03:49
4e811b5 to
2de28c7
Compare
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sophiayyya
force-pushed
the
mxfp4-experts-qat
branch
from
November 11, 2026 04:02
2de28c7 to
7b18d41
Compare
thecourageyu
pushed a commit
to thecourageyu/verl
that referenced
this pull request
Nov 11, 2026
…erl-project#8062) ### What does this PR do? The FP8 end-to-end section of `docs/low_precision/fp8.md` lists `NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1` as a general requirement. It only applies on Hopper. On Blackwell (SM100+), Transformer Engine has no native block-scaled GEMM. It runs the block-wise recipe through MXFP8 GEMMs, whose E8M0 scales hold only an exponent. So both of its block-quantize kernels assert power-of-two scales on SM100+, and a run with the variable set stops at the first FP8 quantization: ``` transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:534 in function quantize_transpose_vector_blockwise: Assertion failed: pow2_scale. On Blackwell and newer, the FP8 block scaling recipe is emulated with MXFP8, which requires using power of two scaling factors. ``` This is the failure reported in verl-project#6172 (8×B200) when following this doc and the verl-recipe FP8 E2E script. This PR marks the variable as Hopper-only and says that on Blackwell it must stay unset; power-of-two scales are the Transformer Engine default. Docs only. Refs verl-project#6172. This does not fix verl-project#6172: that issue also reports garbled FP8 rollouts on Blackwell once the variable is removed, and this PR does not address that. ### Checklist Before Starting - [x] Search for similar PRs: - [NVTE_FP8_BLOCK_SCALING_FP32_SCALES](https://github.com/verl-project/verl/pulls?q=NVTE_FP8_BLOCK_SCALING_FP32_SCALES) - [blockwise Blackwell](https://github.com/verl-project/verl/pulls?q=blockwise+Blackwell) - No PR changes this requirement. Related: verl-project#6172 (the issue above); verl-project#7895 (sets the variable in its DeepSeek-V4 example scripts, does not touch this doc); verl-project#7519 (MXFP8 training, does not change this line). - [x] Format the PR title as `[{modules}] {type}: {description}` ### Test Documentation-only change. - `pre-commit run --files docs/low_precision/fp8.md`: all hooks pass, including "Check docs Last updated info". - Reproduced on 1×B200 (SM 10.0, torch 2.11.0+cu130, CUDA 13.0, Transformer Engine 2.16.1). `te.Linear` (2048→2048) and `te.GroupedLinear` (4 × 2048→768) ran forward and backward under `Float8BlockScaling(fp8_format=Format.E4M3)` and were compared with a bf16 run of the same module: | `NVTE_FP8_BLOCK_SCALING_FP32_SCALES` | `te.Linear` out / dgrad / wgrad rel. err | `te.GroupedLinear` out / dgrad / wgrad rel. err | |---|---|---| | unset (power-of-two scales) | 0.0377 / 0.0423 / 0.0462 | 0.0376 / 0.0475 / 0.0510 | | `1` | `RuntimeError`: the `pow2_scale` assertion above | same | The same modules under `MXFP8BlockScaling` give the same errors as the unset row, with or without the variable. Source references (Transformer Engine 2.16.1, commit c9877beb): - `transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:533-536` and `transformer_engine/common/transpose/quantize_transpose_square_blockwise.cu:490-493`: `if (sm_arch() >= 100) NVTE_CHECK(pow2_scale, ...)`. - `transformer_engine/pytorch/csrc/extensions/gemm.cpp:289-300`: "Emulate the FP8 block scaling recipe with MXFP8 on Blackwell and newer as it is not natively supported by cublasLt". - `transformer_engine/common/swizzle/swizzle_block_scaling.cu:116-118, 205-206`: each FP32 block scale becomes an E8M0 scale by keeping only its exponent byte, `(sf >> 23) & 0xFF`. This is why the scales must be powers of two. ### API and Usage Example None. ### Design & Code Changes `docs/low_precision/fp8.md`: - The requirement line now says that the variable is Hopper-only, why it fails on Blackwell, and what to do there. - "Last updated" is bumped. The verl-recipe script `low_precision/run_dapo_qwen3_moe_30b_megatron_fp8e2e.sh` also exports the variable unconditionally. That lives in the recipe repository and can get a matching note separately. ### Checklist Before Submitting - [ ] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/verl-project/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: documentation-only change; the behaviour it documents was checked on B200 as described under Test. - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1). - [ ] If your PR is related to the `recipe` submodule, please also update the reference to the submodule commit: not applicable. AI assistance: this PR was prepared with AI assistance (Claude Code). I reviewed the change and the evidence above, and the checks listed under Test were run for this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…e); TileLang weight QDQ
DeepSeek-V4's routed-expert FP4 QAT is now the regular `mxfp4` mode instead of a separate mode:
- New `qat.activation` ("mxfp8" default, or "none"). "none" drops the routed-expert MXFP8 input quantizer, so
the experts are weight-only -- for an MXFP4 W4A16 rollout (BF16 training) or for TE FP8 training.
- `mxfp4` + `activation=none` is the one QAT configuration allowed together with
override_transformer_config.fp8 (TE's FP8 GEMM quantizes the activations once); only then is ModelOpt's
TE-FP8 guard lifted. Every other QAT + TE FP8 combination is still rejected.
- Expert weights use verl's TileLang MXFP4 QDQ (`verl_mxfp4` backend, DeepSeek-V4 UE8M0 scale recipe), batched
over all experts of a TE grouped linear; on real DeepSeek-V4-Flash weights it matches ModelOpt's native MX
fake-quant bit for bit.
- DeepSeek-V4 native fp8 checkpoints pair with `mxfp4` in dispatch_rollout_weight_quantizer.
- The stateless-checkpoint guard and the routed-expert quantizer check apply to `mxfp4`.
- Tests renamed to test_mxfp4_weight_only*; recipe, qat.md and megatron.yaml updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Hopper With FP32 block scales (NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1) the TE FP8 cast re-rounds 48% of the MXFP4-grid routed-expert weights of DeepSeek-V4-Flash; with TE's default power-of-two scales it changes none (measured on 12 expert matrices of the 0731 checkpoint), matching DeepSeek-V4's lossless FP4->FP8 dequant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d re-registered params in place - Humming joins the refittable MXFP4 backends. With VLLM_HUMMING_INPUT_QUANT_CONFIG selecting FP8 activations (1x128 dynamic groups) it serves MXFP4 weights x FP8 activations on Hopper via WGMMA dequant. - Humming's conversion deletes and re-creates every w13_*/w2_* parameter with setattr and publishes only some of them through replace_parameter. A refit therefore left the CUDA graph reading stale values of the others: on a 6-layer DeepSeek-V4-Flash slice the rollout/trainer probs diff was 0.16 with Humming W4A8 vs 0.028 with Marlin W4A16. Every live parameter of a staged MoE module is now recorded before staging, and any parameter re-registered outside replace_parameter is copied back into the captured storage afterwards (shape/dtype must match). With this the Humming W4A8 probs diff is 0.027-0.029. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rollout ROLLOUT_HUMMING_W4A8=True selects vLLM's Humming MoE backend and passes VLLM_HUMMING_INPUT_QUANT_CONFIG (FP8 activations, dynamic 1x128 groups, FP32 scales) to the rollout workers through Ray's runtime_env, so the MXFP4 experts run as W4A8 on Hopper instead of Marlin's W4A16. Checked on a 6-layer DeepSeek-V4-Flash slice: the knob alone (no extra overrides) gives the Humming backend and a probs diff of 0.028-0.029. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mxfp4 now builds its quantizer config from the upstream ModelOpt presets, so the extra startup check (enabled quantizer census and on-grid identity log) no longer guards anything the unit tests do not already cover. The mcore test asserts the per-expert quantizer count and that the QDQ equals the TileLang kernel directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
MXFP4 QAT precision and data flow
This change adds routed-expert weight-only MXFP4 fake quantization on top of Transformer Engine (TE) FP8 training for DeepSeek-V4-Flash-0731. Checkpoint storage, trainable parameter dtype, and GEMM precision are separate:
w1,w2,w3I8byte; one E8M0 scale per 32 contiguous input-channel elementsmlp.experts.linear_fc1/linear_fc2wq_a,wq_b,wkv,wo_a,wo_b; indexerwq_bweights_projhc_attn_fn/hc_ffn_fn,attn_sink, and compressorapeI8is the byte container for packed FP4 values, not INT8 quantization. The checkpoint precision map establishes tensor storage and scale geometry; it does not establish runtime kernel precision. Bridge's E2M1 unpacking and the QAT configuration provide the implementation-level interpretation of the packed expert weights. Only indexerwq_bis FP8 in this table; the entire indexer is not uniformly FP8.flowchart TD C["0731 checkpoint: MXFP4 experts / FP8 projections / BF16 / FP32"] I["Bridge import: unpack and dequantize quantized weights"] P["Trainable model parameters: BF16 plus native FP32 tensors"] Q["Routed expert weights: dynamic MXFP4 Q/DQ; BF16 temporary output"] T["TE FP8 quantization and GEMM: hybrid / blockwise recipe"] O["Adam update: FP32 main parameters and optimizer state; HDO CPU offload"] E["Bridge export: requantize updated parameters to MXFP4 / FP8 weight-scale pairs"] V["vLLM rollout / validation: quantization=fp8; KV cache=fp8"] C --> I --> P P -->|routed expert weights| Q Q --> T P -->|other TE FP8 linear paths| T T --> O O -->|update model parameters| P P -->|weight synchronization| E E --> VQAT scope and kernel.
qat.mode=mxfp4_expertsdisables all ModelOpt quantizers first, then enables only*mlp.experts*weight_quantizer, withnum_bits=(2, 1)andblock_sizes={-1: 32, "type": "dynamic", "scale_bits": (8, 0)}. Scales are computed on each call;algorithm=Nonerequires no calibration pass. Shared experts, attention, router, and all ModelOpt input/output quantizers remain disabled. This change does not implement FP4 indexer QK computation or additional KV/output-projection activation fake-quantization.In the validated ModelOpt 0.44 implementation, MXFP4 fake-quant uses
modelopt_cuda_ext_mx.fused_amax_convert→ CUDAamax_and_convert_kernel. It returns BF16 values on the MXFP4 grid without permanently packing the trainable parameters. TE then quantizes weights and activations for its FP8 GEMMs. On the validated H100 setup, this simulates MXFP4 weight error while computing the supported GEMMs in FP8; it is not native FP4 GEMM.bypass_te_fp8_assert=Trueallows this explicit composition. Direct FP8 parameter loading andfp8_paramare not enabled.Scale and rollout handling. Checkpoint E8M0 scales are distinct from runtime compute scales: the recipe sets
NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1for TE andVLLM_USE_DEEP_GEMM_E8M0=0for the vLLM DeepGEMM path. Bridge exports the updated model parameters and independently recreates packed MXFP4 expert weights and FP8 projection weights with their scales. It does not export a previous forward pass's temporary fake-quant result. The generic NVFP4QATWeightExporteris skipped for this mode, androllout.qat.enable=Falsestill retains Bridge's quantized weight synchronization. The diagram describes export formats and rollout configuration, not the compute precision of every vLLM kernel.Checkpoint compatibility. Dynamic MXFP4 quantizers have no persistent calibration tensors.
preserve_mxfp4_checkpoint_methodstherefore preserves native TE/Megatron checkpoint methods while retaining the QAT forward path, with a guard rejecting persistent quantizer state. HDO resume restores optimizer main shards to the GPU before loading optimizer state. Neither fix changes the quantization recipe. Native FP32 parameters with HDO additionally require the detach fix in NVIDIA/Megatron-LM#6982, as documented in the example script.Implementation: quantization configuration, checkpoint compatibility, and 40-step QAT example.
Quantization experiment evidence
The completed H100 experiment ran 40 steps with a 4096-token response limit, validation at steps 10/20/30/40, checkpoints at steps 20/40, and optimizer-state resume between the two 20-step segments. Step-40 AIME validation accuracy was 64.21% with MXFP4 QAT, versus 64.59% for the TE FP8 control without ModelOpt QAT. These are single-run measurements, not evidence of a statistically significant difference. See the QAT run and control run.
This experiment predates the final rebase. It used the implementation before publication cleanup and Megatron
1ff25ca7ewith the equivalent local FP32-shard detach patch; the runtime patch helper is not included in this PR. These results validate that experiment, not a fresh 40-step run of the rebased head.Checklist Before Starting
[{modules}] {type}: {description}(This will be checked by the CI){modules}includefsdp,megatron,veomni,sglang,vllm,rollout,trainer,ci,training_utils,recipe,hardware,deployment,ray,worker,single_controller,misc,perf,model,algo,env,tool,ckpt,doc,data,cfg,reward,fully_async,one_step_off,like[megatron, fsdp, doc]{type}is infeat,fix,refactor,chore,test[BREAKING]to the beginning of the title.[BREAKING][fsdp, megatron] feat: dynamic batchingTest
API and Usage Example
# Add code snippet or script demonstrating how to use thisDesign & Code Changes
Checklist Before Submitting
Important
Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.
pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=alwaysci-requestchannel in theverlSlack workspace. (If not accessible, please try the Feishu group (飞书群).)recipesubmodule, please also update the reference to the submodule commit viagit submodule update --remoteorcd recipe && git pull origin main.