Repository navigation
[megatron, vllm, env] feat: add native NVFP4 training and rollout with live refit - #7971
Conversation
Add a fail-closed Megatron/Transformer Engine W4A4 path, native vLLM 0.26 per-token NVFP4 rollout and layerwise refit, R3 capture support, and a versioned scheduler-gated experiment bundle. Assisted-by: OpenAI Codex
Build the runtime from tracked archives and record the source commit and lockfile digest so later scheduler stages cannot mix an old image with changed code. Assisted-by: OpenAI Codex
Restrict the duplicate-submission guard to this experiment bundle instead of blocking on unrelated user jobs. Assisted-by: OpenAI Codex
Keep the lock check on the submission host and rely on frozen sync inside the target aarch64 container, avoiding cross-architecture uv execution before container startup. Assisted-by: OpenAI Codex
Use git grep for the no-loss audit and let the probe inventory pip when the base image has no uv binary. Assisted-by: OpenAI Codex
Install the architecture-native uv wheel when the aarch64 base image does not provide a usable executable, and verify the exact lock-tool version before syncing the frozen runtime. Assisted-by: OpenAI Codex
Match the exact local-version metadata emitted by the locked Megatron Core source build in both image and preflight checks. Assisted-by: OpenAI Codex
Exercise 16 tokens per expert as required by the NVFP4 block size. Permit harness-only bundle changes after an image build while continuing to reject any runtime payload drift. Assisted-by: OpenAI Codex
Default the real-NVFP4 worker flag to false for legacy and test-created workers, align the R3 test double with vLLM 0.26 source structure, and derive a fresh v4 immutable job bundle for the runtime change. Assisted-by: OpenAI Codex
v11 changed three runtime variables at once and its rollout collapsed to near-uniform output. Add a v12 bundle that applies only enable_pdl=False on top of the v8 image, so the startup-hang fix is separated from the post-0.26 online-NVFP4 backports. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The single-hunk #50074 backport keeps the first NVFP4 MoE quant config alive across reloads while _quantize_weights rebinds every scale parameter, so after the first refit the kernel pairs dummy-init scales with freshly packed weights. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add the v13 long-run bundle on top of the v12 runtime and require a recorded numeric gate before release. v11 cleared every structural gate while its rollout emitted near-uniform output, so the audit asserts KL, ESS, entropy, clip ratio, grad norm and length slope against the verified v8 band. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step time tracks response length (v8 averaged 110s/step at ~1.0k tokens) and a healthy run grows length, so equal 80-step chunks would overrun the partition limit near the end. Target 80/150/210/265 instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gb200_30B_bf16_megatron_0603 is 12 same-named runs, one per 5h segment, whose per-run _step is not the global training step. Record the run ids, the stitching rule and the resulting curve so the long-run comparison is reproducible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
train.job carries its own copy of the chunk target list and rejects a mismatch at runtime, so shrinking the chunks in submit.sh alone would have failed chunk 2 only after chunk 1 had run for hours. Sync both copies, align every target to the save_freq cadence, and assert both invariants during static validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both rollout attestations read the layer's scale parameters, which were correct in v11; only the kernel's captured quant config was stale. Record the identity assertion to add before the upstream PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reward and response length both move a lot between step 20 and step 265, so the comparison is only meaningful step-for-step. Stitch the BF16 segments once, cache them, and report matched-step values and slopes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v12 reproduces the v8 numeric band while keeping 32/32 startup, so enable_pdl=False is numerics-neutral and the rollout collapse came from the post-0.26 backports. Record the comparison, the BF16 matched-step result and the two chain defects caught during v13 preparation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| QAT requires a quantized rollout: set `rollout.quantization` or use a checkpoint with a native `quantization_config` that verl supports for export and refit. verl checks at startup that the QAT mode matches the effective rollout configuration. Megatron QAT uses [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) for fake quantization. MXFP8 uses ModelOpt's native `MXFP8_DEFAULT_CFG`; FP8_BLOCK uses a verl backend because ModelOpt's `FP8_DEFAULT_CFG` is per-tensor and its 2D-block preset is weight-only. MXFP4 and NVFP4 use compressed-tensors-compatible QDQ backends so fake quantization is bit-exact with the exported checkpoint format. Both FP4 modes apply only to routed experts. Use `mxfp4_mxfp8` for the vLLM combination where dense linear layers also use MXFP8. | ||
| Its `ignore_patterns` use Megatron module names and `fnmatch` syntax. Keep them aligned with the rollout ignore list; for Qwen3.5 MXFP8, add `*linear_attn.in_proj_a` and `*linear_attn.in_proj_b`. | ||
|
|
||
| ### Native NVFP4 training (Megatron + Transformer Engine, experimental) |
There was a problem hiding this comment.
Please move NVFP4 training to separate nvfp4.md, just like:
- FP8 training: https://verl.readthedocs.io/en/latest/low_precision/fp8.html
- MXFP8 training: https://verl.readthedocs.io/en/latest/low_precision/fp8.html
| return any(fnmatch.fnmatch(module_path, pattern) for pattern in ROUTED_EXPERT_PATTERNS) | ||
|
|
||
|
|
||
| def real_nvfp4_bf16_layers(override_transformer_config: Mapping[str, Any]) -> Optional[tuple[int, int]]: |
There was a problem hiding this comment.
Can we make it a common function? I think other recipe(e.g mxfp8/mxfp4) could also reuse it.
There was a problem hiding this comment.
Split it in f7b8003: real_nvfp4_enabled only checks the fp4 settings, and the BF16 first/last layer selection is now bf16_layers_at_start_end in verl/utils/megatron/low_precision.py, usable by any recipe.
| logger.info(f"Rollout compressed-tensors quantization_config: {quantization_config}") | ||
| return {"quantization_config": quantization_config} | ||
|
|
||
| def _check_nvfp4_per_token_rollout(self, compilation_config: dict, engine_kwargs: dict) -> None: |
There was a problem hiding this comment.
Move _check_nvfp4_per_token_rollout to verl/utils/real_nvfp4/config.py to keep vllm_async_server.py neutral to nvfp4.
There was a problem hiding this comment.
Moved to verl/utils/real_nvfp4/config.py as check_nvfp4_per_token_rollout (cb01d30); vllm_async_server.py only calls it.
| attest_nvfp4_per_token_runtime(self.model_runner.get_model(), self._nvfp4_per_token_config) | ||
|
|
||
| def _update_nvfp4_per_token_weights(self, use_shm: bool): | ||
| """Refit through vLLM's layerwise reload, one bucket at a time. |
There was a problem hiding this comment.
P0: Do not use layerwise reload since it accumulate tensors until all weights in single layer are collected. For large MoE model, this may cause OOM.
There was a problem hiding this comment.
Replaced the layerwise reload with an in-place refit like the NVFP4 one in vllm_fp4_utils (cb01d30): buckets load into checkpoint-layout views over the live expert storage, and each layer is post-processed after the last bucket. Extra memory is now one MoE layer (325 MiB vs 2.6 GiB for Qwen3-30B-A3B layers), with bit-identical logprobs.
- Load refit buckets into checkpoint-layout views over the live expert storage and post-process each layer after the last bucket, instead of vLLM's layerwise reload; the extra memory is one MoE layer. - Move the NVFP4 training docs to docs/low_precision/nvfp4.md. - Share the BF16 first/last layer selection in verl/utils/megatron/low_precision.py. - Move the NVFP4_PER_TOKEN rollout check to verl/utils/real_nvfp4/config.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
real_nvfp4_enabled only checks override_transformer_config.fp4; callers take the BF16 first/last layers from verl.utils.megatron.low_precision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4 docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@wuxibin89 On the vLLM patches this PR still carries (vLLM 0.32):
|
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| # get_model() unwraps vLLM's CUDA-graph wrapper. | ||
| attest_nvfp4_per_token_runtime(self.model_runner.get_model(), self._nvfp4_per_token_config) | ||
|
|
||
| def _update_nvfp4_per_token_weights(self, use_shm: bool): |
There was a problem hiding this comment.
Why vLLMColocateWorkerExtension still need special process for nvfp4_per_token , can we align with nvfp4 weight refit in verl/utils/vllm/vllm_fp4_utils.py?
There was a problem hiding this comment.
Aligned in a1677fe: NVFP4_PER_TOKEN now uses the shared quantized refit, with no per-token code in update_weights_from_ipc.
The worker no longer has its own NVFP4_PER_TOKEN refit: the per-token experts go through prepare/process_quanted_weights like the other formats, with the per-token refit check and attestation around it. The NVFP4 refit in vllm_fp4_utils now takes the per-token TRT-LLM experts and restores the parameters TRT-LLM re-registers, and the shared post-process re-runs the BF16 routed-expert layers of a quantized model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update_weights_from_ipc no longer has NVFP4_PER_TOKEN code: the per-bucket refit check is gone and the g1_scale_c check moves into the shared NVFP4 kernel scale attestation. The worker only attests the routed-expert precision once at start-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rollout.quantization.moe=NVFP4 with moe_activation=nvfp4_per_token replaces the NVFP4_PER_TOKEN preset. The activation name follows vLLM's quantization_config.moe.activation. The exported quantization_config and the numerics are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…8494) ### What does this PR do? NVFP4 QAT now uses per-token activations, the same scheme as the NVFP4 rollout from #7971. The static per-tensor activation scale is removed. Why per-token: - Each token is quantized with a scale from its own amax, independent of the other tokens in the batch and of earlier training steps. The static scale was a running max of the observed amax, so tokens much smaller than the largest activation seen so far lost block-scale range, and it was 1.0 until the first training step. - Much lower training-inference mismatch in RL training: mean `rollout_corr/k3_kl` over 100 steps 0.060 → 0.0033, the same as native NVFP4 training (see Test). Changes: - `rollout.quantization.moe: NVFP4` always uses per-token activations; `moe_activation` now defaults to `nvfp4_per_token`. - Fake quantization matches FlashInfer's per-token activation quantization and the trainer-side weight export bitwise. - **Breaking:** removes the static NVFP4 refit for SGLang. verl does not support per-token NVFP4 with SGLang yet. - **Breaking:** removes FSDP QAT (`QATLinear`, `QATQuantizer`, `actor.qat` and the FSDP engine's `qat` config). It has been unreachable since #8287: its `w4a16`/`w4a4` modes match no rollout preset, and the FSDP engine has no trainer-side rollout quantizer. QAT requires the Megatron engine. - Merges `nvfp4_qat.md` into `qat.md`. ### Test RL training, Qwen3-30B-A3B-Base on DAPO-Math-17k, 8x4 GB200, 100 steps. Routed experts in NVFP4 in training and rollout, layers 0-1 and 44-47 in BF16. <img width="3133" height="755" alt="image" src="https://github.com/user-attachments/assets/f0858fdf-e8df-4784-bcfc-393acf4506e9" /> Mean over 100 steps: | Training | Rollout activation | `rollout_logprobs_diff_mean` | `k3_kl` | `rollout_is_mean` | |---|---|---|---|---| | NVFP4 QAT, before this PR | static per-tensor | 0.124 | 0.060 | 0.979 | | NVFP4 QAT | per-token | 0.021 | 0.0033 | 0.9996 | | Native NVFP4 training (#7971) | per-token | 0.021 | 0.0033 | 0.9996 | `scripts/rollout_logprob_diff.py` on Qwen3.5-35B-A3B, Transformer Engine 2.20.2, ModelOpt 0.47. `qat.md` mismatch table setting (EP=8, vLLM TP=2), median of 3 runs: | Rollout | Training | k3_kl | |---|---|---| | NVFP4 MoE | BF16 | 0.00966 | | NVFP4 MoE | NVFP4 QAT | 0.00766 | Unit tests: - new `tests/utils/test_nvfp4_qat_fake_quant_on_gpu.py`: bitwise checks against vLLM / FlashInfer; - updated `tests/utils/test_modelopt_mxfp8_on_gpu.py`: NVFP4 per-token backends; - updated `tests/utils/real_nvfp4/test_nvfp4_per_token_on_cpu.py`. ### API and Usage Example ```yaml actor_rollout_ref: actor: megatron: qat: enable: true mode: nvfp4 rollout: quantization: moe: NVFP4 ``` ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/verl-project/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: the GPU tests need Blackwell. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: wuxibin <wuxibin@bytedance.com>
Only docs/low_precision/qat.md's 'Last updated' line conflicted. The CPU test loads vllm_quant_utils by path with its vLLM-facing siblings stubbed, so it now also stubs the two modules main imports there since verl-project#7971 / verl-project#8494 / verl-project#8483 (vllm_nvfp4_per_token_utils, vllm_unquant_utils) and drops the removed install_nvfp4_cutlass_compressed_tensors_patch stub. Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
…layout changes shape (verl-project#7987) ### What does this PR do? Fixes verl-project#7978: the bf16 MoE weight sync crashes on Blackwell when vLLM picks its default FlashInfer TRT-LLM MoE backend, because that backend keeps the expert weights in a 4-D block layout that the checkpoint-format loads no longer fit. This revision stages the checkpoint layout **in place** (a view over the live storage) and folds the re-derived kernel layout back layer by layer. It replaces the first revision's use of vLLM's layerwise reload, which buffered whole layers (see review feedback below). **Root cause (verified against vLLM 0.24.0 and 0.29.0 source).** `update_weights_from_ipc` feeds the bucketed checkpoint-format tensors straight into the live parameters via `model.load_weights` and runs `process_weights_after_loading` once at the end. That holds only while the kernel layout equals the checkpoint layout up to a same-shape in-place transform (Triton MoE, FlashInfer CUTLASS). On SM100 vLLM auto-selects the FlashInfer TRT-LLM bf16 MoE backend (`fused_moe/oracle/unquantized.py::_get_priority_backends` puts it first on non-Hopper CUDA), whose `convert_moe_weights_to_flashinfer_trtllm_block_layout` turns `w13_weight` / `w2_weight` into 4-D BlockMajorK tensors. `RoutedExperts.weight_loader` then indexes `param.data[expert_id]` (3-D) and `_load_w13(shard_dim=0)` rejects it at the first sync: ``` ValueError: shard_dim=0 is not a valid data dimension for a 3D tensor (expected 1 or 2) ``` The conversion is one-way (it cannot be re-applied to its own 4-D output), so the end-of-sync `process_weights_after_loading` could not repair it either. Observed 2026-09-21 on 1×B200 (random-init Qwen3-MoE, TP1, vLLM 0.24.0); any bf16 MoE RL rollout on Blackwell with the default backend hits it. **Why not vLLM's layerwise reload (the first revision of this PR).** As @wuxibin89 pointed out, that lifecycle holds every incoming tensor of a layer until the layer is complete (`online_process_loader` appends each call to `info.loaded_weights` before counting it). In verl those tensors are views into the receiver's reused IPC bucket, so they had to be cloned, and what is held is the sender's full tensor: before TP narrowing, and including experts that are not local to the rank (the loader is called for every expert; the non-local calls return `False` but have already been appended). With verl's export order (all experts' gate/up before any `down_proj`, e.g. `unfuse_moe_params` on the FSDP path) a MoE layer completes only at its last `down_proj`, so the extra peak per GPU is about one full, un-sharded MoE layer on top of the rebuilt layer, whatever the TP/EP: | model | one MoE layer, bf16, all experts (G) | layerwise revision ≈ G + 2G/P | this revision ≈ G/P | |---|---|---|---| | Qwen3-30B-A3B, experts over P=2 GPUs | 1.21 GB | ≈ 2.4 GB | ≈ 0.6 GB | | Qwen3-235B-A22B, P=8 | 4.83 GB | ≈ 6.0 GB | ≈ 0.6 GB | | DeepSeek-V3-sized layer, P=16 | 22.5 GB | ≈ 25 GB | ≈ 1.4 GB | Estimates from the code paths and the model configs. The layerwise revision also held the clones of the current bucket (up to 2 GB). Measured on the tiny model below: 460.1 MB vs 6.0 MB per sync. **Fix.** The TRT-LLM bf16 block layout has the same byte count as the checkpoint layout (`w13`: `[E, 2H/128, 2I, 64]` vs `[E, 2I, H]`, both bf16), so no second copy is needed: 1. **Stage** (before the first bucket, `stage_unquantized_moe_params`). For each `UnquantizedFusedMoEMethod` layer whose live `w13_weight` / `w2_weight` no longer has the checkpoint shape, put a `Parameter` on the layer that views the start of the live storage in checkpoint shape and set the live parameter aside. The checkpoint shapes come from the metadata vLLM records for every layer at `initialize_model` for its own reloads (`reload.layerwise.LAYERWISE_INFO[layer].restore_metadata`), so nothing is patched before the engine starts. Layers whose kernel keeps the checkpoint shape (Triton; FlashInfer CUTLASS's same-shape gate/up swap) are not staged and load as before. 2. **Stream** (per bucket, unchanged from `main`). `model.load_weights(bucket)`: the expert loaders write each expert's rows straight into the live storage through the view. Nothing is buffered or cloned; the receiver reuses its IPC buffer as before. 3. **Fold** (after the last bucket, `fold_unquantized_moe_params` around the existing `process_weights_after_loading`). vLLM's `_setup_kernel` re-derives the kernel layout from the view into a temporary and calls `replace_parameter(..., prefer_copy=True)`. With the view on the layer that call's shape check fails, so for the duration of this call `replace_parameter` in `unquantized_fused_moe_method` is patched to copy the temporary into the set-aside live parameter and put that parameter back. This happens inside the layer's own hook, so the temporary is freed before the next layer is processed. CUDA-graph addresses and the live `Parameter` objects are unchanged. A staged weight that nothing re-derived raises instead of being left in checkpoint layout. The set-aside parameters stay recorded on the layer until each one is folded back, so a sync that fails anywhere in between leaves the next sync what it needs to restore them. Between steps 1 and 3 the live storage holds checkpoint-layout bytes, so no forward pass may run in between. That is the assumption `main`'s sync already makes (it loads into the live parameters bucket by bucket and post-processes at the end) and the one the FP8 staging path relies on. vLLM 0.29 pads TRT-LLM's intermediate dim to a multiple of 128 before the block conversion when the per-rank size is not one (e.g. Qwen3-235B-A22B or GLM-4.5 at TP8 without EP: 192 → 256). The padded live storage still has room for the checkpoint layout, so it is staged in place too. Only a kernel layout smaller than the checkpoint layout, which no unquantized backend produces today, would get a fresh NaN-filled staging buffer, logged with its size. **Which backends this touches.** Staging keys on the layer's live layout, not on the backend name. For the CUDA list in vLLM's `_get_priority_backends`, per `convert_to_unquantized_kernel_format` (the same in 0.24 and 0.29): | backend | kernel prep of `w13` / `w2` | shape changes | this PR | |---|---|---|---| | `FLASHINFER_TRTLLM` (default on SM100) | 4-D block layout (0.29: after padding the intermediate dim to a multiple of 128) | yes | staged, then folded | | `FLASHINFER_CUTLASS` | gated MoE: gate/up halves swapped (`swap_w13_to_w31`); otherwise none | no | untouched | | `TRITON` (default on SM90) | none (`.contiguous()`) | no | untouched | | `BATCHED_TRITON` | none | no | untouched | For the untouched backends `stage_unquantized_moe_params` returns nothing and the fold is a no-op (nothing is patched), so the sync is exactly `main`'s. The same-shape CUTLASS swap needs no staging on this path: the full refit rewrites every row of every local expert in checkpoint layout, and the post-load pass, which runs on every layer, re-derives the swap and copies it in place (`prefer_copy`). The FP8 path in verl-project#7986 had to detect same-shape repacks because it re-processes only the layers it staged. What this relies on in vLLM (the module-level `replace_parameter` import in `unquantized_fused_moe_method`, `_setup_kernel`'s `prefer_copy` call, `LAYERWISE_INFO` filled by `record_metadata_for_reloading` at `initialize_model`, and the TRT-LLM conversion returning bf16) is the same in vLLM 0.24.0, the version run on B200, and in 0.29.0, which `main` pins (checked in source). On a vLLM without these pieces, staging is skipped and the sync behaves as on `main`. Everything else in the sync is as on `main`: buffer updates per bucket; LoRA, QAT, ModelOpt and FP8 paths untouched. Folding reuses the FP8 refit helpers `_copy_param_subclass_attrs` and `_fold_into_live_param` (verl-project#7986). Workaround until this lands: `+actor_rollout_ref.rollout.engine_kwargs.vllm.moe_backend=triton` (verified on B200). ### Checklist Before Starting - [x] Similar PRs (re-checked 2026-09-23): the only open PR mentioning verl-project#7978 is this one; searches for "unquantized moe", "moe refit", "flashinfer moe weight update" find no open PR on this path (verl-project#7971 is NVFP4 live refit, a quantized path). Closed verl-project#7136 explored vLLM's layerwise lifecycle for the *FP8* resync on vLLM 0.20. verl-project#6847 (closed) is a different failure of the FlashInfer-MoE-plus-weight-update family (stale kernel tensor addresses on 0.20.2, fixed upstream by `prefer_copy`). - [x] Title format `[rollout, vllm] fix: ...` ### Test ```bash python -m pytest tests/utils/test_vllm_moe_refit_staging_on_cpu.py -q # 10 passed python -m pytest tests/utils/test_vllm_weight_name_normalization_on_cpu.py -q # 21 passed, 6 failed: the 6 are the pytest.importorskip("vllm") tests (tied-alias and real-receiver bucket tests) # that need the vllm CI venv; a clean main checkout gives the same 6 failures (19 passed) pre-commit run --files <the 4 changed files> # ruff, ruff-format, mypy: pass ``` `tests/utils/test_vllm_moe_refit_staging_on_cpu.py` runs against stub vLLM modules, with a toy block-transpose kernel layout that has the property that matters (a new shape over the same bytes, not convertible in place): - the block layout is staged as a view of the live storage, the loads land in it, and the fold restores the same `Parameter` objects holding the re-derived layout; - each layer is folded before the next layer is processed (one temporary at a time); - layout-preserving backends (identity, same-shape swap), layers without recorded metadata and layers with another quant method are not staged; - a padded kernel layout (vLLM 0.29) is staged in the live storage; a layout smaller than the checkpoint layout gets a fresh buffer (NaN until written); - a staged weight that nothing re-derived fails loudly; - a layer left staged by a failed sync is restaged from its live parameters; - a fold that raises, or a staging failure part way through a layer, keeps the set-aside parameters recorded, and the next sync restores and refits the layer. Each of these tests was checked to fail when the behaviour it covers is removed (no fold, a fresh allocation instead of a view, the padded storage not reused, the room check dropped, the fold deferred to the end of the model, the re-derivation check dropped, the stale stash ignored, the record dropped before the fold, the record attached only after staging). In `tests/utils/test_vllm_weight_name_normalization_on_cpu.py`: the standard sync stages before the first bucket, hands `load_weights` the receiver's bucket views (no copies) and runs `process_weights_after_loading` inside the fold; a LoRA adapter sync neither stages nor folds. #### Hardware — 1×B200, this PR at `162b3db7` (2026-09-23) Setup as in the earlier round (collapsed below): RunPod 1×B200 (US-NE-1), image `verlai/verl:vllm024.dev2` (vLLM 0.24.0, FlashInfer 0.6.12), random-init 2-layer / 8-expert Qwen3-MoE (hidden 512, `moe_intermediate_size` 256, top-2), Megatron bf16 training + vLLM bf16 rollout, TP1, GRPO on gsm8k, FlashInfer autotune off. Pod-only debug hooks (not in this PR) measured the sync memory and checked the refit bytes; each arm's backend is vLLM's own `Using … Unquantized MoE backend` line. All arms ran on the same pod. | arm | code | MoE backend | result | |---|---|---|---| | T1, negative control | `main` `0ed63af8` | FlashInfer TRT-LLM (auto) | crashes at the first sync: `ValueError: shard_dim=0 is not a valid data dimension for a 3D tensor (expected 1 or 2)` | | T2 | this PR, lr 1e-6, 4 steps | FlashInfer TRT-LLM (auto) | exit 0; `Staged 2 unquantized MoE layers in checkpoint layout for the refit` on each of the 5 syncs; no fresh staging buffer; `rollout_corr/kl` 0.00177 / 0.00182 / 0.00181 / 0.00190 (Triton baseline 0.0019) | | T4, exactness | this PR, lr 1e-3, 3 steps | FlashInfer TRT-LLM (auto) | 4/4 syncs: the staged view shares the live storage; incoming expert-0 `gate`/`up`/`down` bytes equal the staged bytes; after the fold the live `w13` / `w2` are the original `Parameter`s at the original address and bit-equal to vLLM's own `convert_to_unquantized_kernel_format` of the delivered weights | | T5 | this PR, lr 1e-3, `moe_backend=triton` | Triton | not staged (no staging line); 4/4 syncs: live `w13` / `w2` bit-equal to the conversion (identity) of the delivered weights, same objects and addresses; kl 0.00192 / 0.00184 / 0.00183 | | T6 | this PR, lr 1e-3, `moe_backend=flashinfer_cutlass` | FlashInfer CUTLASS | not staged; 4/4 syncs: live bit-equal to the conversion (gate/up swap) of the delivered weights, same objects and addresses; kl 0.00187 / 0.00179 / 0.00179 | | A-mem | layerwise revision `1bfda7dc`, 2 steps | FlashInfer TRT-LLM (auto) | exit 0, lifecycle 3/3; memory below | Sync memory, per sync: peak `torch.cuda.max_memory_allocated` during `update_weights_from_ipc` minus the allocation before it. | revision | peak extra per sync | |---|---| | layerwise revision (`1bfda7dc`) | 460.1 MB (3/3 syncs) | | this PR (`162b3db7`) | 6.0 MB (5/5 syncs) | On this tiny model the layerwise number is mostly the clones of the whole bucket (the 0.33 GB model fits in one 2 GB bucket, embedding and `lm_head` included) plus the rebuilt layers. This PR's 6.0 MB is one MoE layer's temporary (`w13` 4 MB + `w2` 2 MB), as expected. Real-size numbers will come with the Qwen3-30B-A3B run. T4 per sync (sha1 prefixes; lr 1e-3 so the weights move every step): | sync | incoming expert-0 `gate_proj` | staged expert-0 gate | live `w13` after the fold | |---|---|---|---| | step 0 | `1f07f6a9fe2e601f` | `1f07f6a9fe2e601f` | `a905365b9495ecec` | | step 1 | `93b57a89c31877fc` | `93b57a89c31877fc` | `4d2e0f3c969a1ec4` | | step 2 | `d646a30eb7b97f0a` | `d646a30eb7b97f0a` | `c186bb52902b7269` | | step 3 | `2b7b47ba68802d43` | `2b7b47ba68802d43` | `b08f175176a1184e` | Step 0's live `w13` is byte-identical to the layerwise revision's step 0 in the earlier round (`a905365b9495ecec`): same random-init weights, same kernel layout. **Still pending:** Qwen3-30B-A3B TP2/EP2 on 4×B200. <details> <summary>Earlier layerwise revision (<code>1bfda7dc</code>, superseded) — 1×B200 results, 2026-09-22</summary> **T1 — negative control: the sync path `main` has** (the layerwise revision with `VERL_VLLM_LAYERWISE_RELOAD=0`), default backend. Crashes at the first weight sync, exactly the failure in verl-project#7978: ```text (Worker pid=146003) ERROR 09-22 17:00:55 [multiproc_executor.py:1000] ValueError: shard_dim=0 is not a valid data dimension for a 3D tensor (expected 1 or 2) ``` (`RoutedExperts.weight_loader → _load_w13 → _get_hidden_dim` on the 4-D `w13_weight`.) Setup: RunPod 1×B200 (SM100, 183 GB), image `verlai/verl:vllm024.dev2` (vLLM **0.24.0**, FlashInfer 0.6.12, Megatron-Core 0.18, Megatron-Bridge 0.5.2 — `main` now pins vLLM 0.29; the reload API and the SM100 backend selection are unchanged there, source-checked, but 0.29 itself was not run). Model: random-init 2-layer / 8-expert Qwen3-MoE (`Qwen3MoeForCausalLM`, hidden 512, `moe_intermediate_size` 256, top-k 2, per-expert HF layout), Megatron bf16 training + vLLM **bf16** rollout, TP1, GRPO on gsm8k, FlashInfer autotune off, `val_before_train=False`. The random model's outputs are near-uniform (entropy ≈ ln V) and all gsm8k rewards are 0, so the runs check the *mechanism* (crash / no crash, exact weight delivery, lifecycle bookkeeping), not model quality. The bf16-Triton baseline numbers come from the same setup on 2026-09-21. **T2 — layerwise revision (default backend, lr 1e-6, 4 steps).** `TRAIN_EXIT=0`, 5 syncs (step 0 + 4): ```text INFO:2026-09-22 16:32:28,251:vLLM layerwise reload: parameters restored to checkpoint layout for this sync WARNING 09-22 16:32:28 [layerwise.py:255] RotaryEmbedding: Failed to load weights INFO:2026-09-22 16:32:28,965:vLLM layerwise reload finalized, deferred buffers applied: 0 ``` `parameters restored` ×5 / `finalized` ×5; the only `Failed to load weights` is `RotaryEmbedding` (buffer-only layer whose tensors the stream never carries — vLLM puts its original tensors back, one warning per sync); no "Allocating … buffers" warnings; restore→finalize span 0.7 s on the first sync, 0.1 s after. Same-setup bf16 baseline on the Triton backend (old path) for comparison: | | `rollout_corr/kl` steps 1–4 | `rollout_probs_diff_mean` | |---|---|---| | layerwise revision, FlashInfer TRT-LLM (default on SM100) | 0.00177 / 0.00182 / 0.00193 / 0.00184 | 4.7e-08 | | old path, `moe_backend=triton` (2026-09-21) | 0.00190 / 0.00187 / 0.00185 / 0.00186 | 4.7e-08 | **T4 — exact delivery and freshness (lr 1e-3 so the weights move every step; 3 steps = 4 syncs).** A pod-only debug hook (not in this PR) logged the sha1 of the raw bf16 bytes of `layers.0.self_attn.o_proj.weight` as it arrived and of the live parameters after finalize: | sync | incoming `o_proj` | live `o_proj` after finalize | live `w13_weight` (4-D kernel layout) | |---|---|---|---| | step 0 | `01a611e6f6b7a3e1` | `01a611e6f6b7a3e1` | `a905365b9495ecec` | | step 1 | `f53bac539b9ef503` | `f53bac539b9ef503` | `4b6718bfe919be02` | | step 2 | `8afd6ca3bc4a656d` | `8afd6ca3bc4a656d` | `2cdc3ffff9aecfee` | | step 3 | `6101d8425d0ff80d` | `6101d8425d0ff80d` | `4722bf20ef2da3b7` | Live equals incoming bit-for-bit on every sync (delivered into the CUDA-graph storage, not a stale copy), the incoming hash changes every step (the trainer's weights really move), and the 4-D `w13_weight` changes every sync (the experts are re-derived into the kernel layout each time). The same hook on the Triton old path shows the same equalities (4/4), as expected. Also observed: a fresh pod's first vLLM start with this backend took 1125 s (`torch.compile and initial profiling/warmup`), 2.6 s once the FlashInfer JIT cache was warm — unrelated to this PR but worth knowing when timing bf16 MoE rollouts on Blackwell. </details> ### API and Usage Example No API change and no new environment variable (the layerwise revision's `VERL_VLLM_LAYERWISE_RELOAD` is gone). ### Design & Code Changes - `verl/utils/vllm/vllm_moe_refit_utils.py` (new): `stage_unquantized_moe_params(model)` and the `fold_unquantized_moe_params(layers)` context manager described above. - `verl/workers/rollout/vllm_rollout/utils.py` (+13/−2): the standard branch of step 1 stages every synced model (main model and MTP drafter); the standard branch of step 3 runs `process_weights_after_loading` inside the fold. `_update_weights` is unchanged from `main`. - Tests as above. ### Checklist Before Submitting - [x] Read the Contribute Guide - [x] pre-commit hooks on the changed files (ruff, ruff-format, mypy); full `pre-commit run --all-files` not run locally (the config-generation hook needs hydra) - [x] Documentation: the change is internal to the sync path and described in the module docstring. - [x] Unit tests added (CPU) - [ ] ci-request — 1×B200 validation done (see Hardware); Qwen3-30B-A3B 4×B200 run pending AI assistance was used for the diagnosis (vLLM source reading) and for drafting the code, tests and this description (Claude); the author reviewed every changed line and ran the tests above. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: wuxibin <wuxibin@bytedance.com>
…and low-precision rollout (verl-project#8287) ### What does this PR do? This PR refactors Megatron QAT and low-precision rollout around a unified trainer-side weight quantization and rollout-side refit boundary. - Add compressed-tensors-compatible trainer-side exporters for FP8_BLOCK, MXFP8, MXFP4, and NVFP4. - Add ModelOpt-backed Megatron QAT modes: `fp8`, `mxfp8`, `mxfp4`, `mxfp4_mxfp8`, and `nvfp4`. - Replace the string `rollout.quantization=fp8` API with structured `rollout.quantization.linear`, `moe`, `ignore`, and `config_file` settings. - Load and refit pre-quantized weights in vLLM and SGLang instead of quantizing BF16 weights inside rollout workers. - Add vLLM MXFP4/NVFP4 refit support, update FP8 refit handling, and update SGLang routing replay support. - Add a unified QAT guide, migration notes, examples, CPU tests, and a rollout log-prob comparison tool. - Remove the legacy `verl.utils.modelopt` implementation and TRT-LLM rollout quantization support. Related to verl-project#5972 and verl-project#4066. This PR has partial scope overlap with open low-precision work, but uses a materially different architecture: verl-project#7519 implements Transformer Engine shared-grid MXFP8 training and rollout, while this PR implements ModelOpt QAT plus a generalized trainer-side compressed-tensors export boundary for multiple formats; verl-project#7971 implements native Transformer Engine NVFP4 real quantization, while this PR implements NVFP4 fake-quant QAT and pre-quantized refit; verl-project#8205 fixes the legacy FSDP NVFP4 compressor API, while this PR refactors the Megatron and rollout path and does not add this feature to FSDP. AI assistance was used to review the branch, fix two review findings, add a regression test, and prepare this PR description. The human submitter reviewed the changes and is responsible for the contribution end to end. ### Checklist Before Starting - [x] Search for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+QAT+compressed-tensors+rollout - [x] Format the PR title as `[{modules}] {type}: {description}`. ### Test Passed: ```bash git diff --check origin/main...wuxibin/qat python -m compileall -q verl/utils/qat verl/utils/vllm verl/workers/rollout/sglang_rollout verl/workers/rollout/trtllm_rollout PATH="$PWD/.venv/bin:$PATH" pre-commit run --all-files --show-diff-on-failure --color=always ``` The full pre-commit suite passed, including Ruff, Ruff format, mypy, generated-config verification, repository sanity checks, and compileall. The targeted pytest suites and GPU end-to-end tests were not run in this local environment because PyTorch, ModelOpt, and compressed-tensors were unavailable. CI/GPU validation should cover Megatron with vLLM and SGLang using FP8_BLOCK, plus MXFP8/MXFP4/NVFP4 on supported Blackwell hardware. ### API and Usage Example The previous API: ```bash actor_rollout_ref.rollout.quantization=fp8 ``` is replaced by: ```bash actor_rollout_ref.rollout.quantization.linear=FP8_BLOCK \ actor_rollout_ref.rollout.quantization.moe=FP8_BLOCK ``` Enable matching Megatron QAT with: ```bash actor_rollout_ref.actor.megatron.qat.enable=true \ actor_rollout_ref.actor.megatron.qat.mode=fp8 ``` Dense and routed-expert precision can be configured independently. For example, use BF16 dense layers and MXFP4 experts with: ```bash actor_rollout_ref.rollout.quantization.linear=null \ actor_rollout_ref.rollout.quantization.moe=MXFP4 \ actor_rollout_ref.actor.megatron.qat.enable=true \ actor_rollout_ref.actor.megatron.qat.mode=mxfp4 ``` TRT-LLM no longer accepts `rollout.quantization`; use vLLM or SGLang for supported low-precision rollout configurations. ### Design & Code Changes - Introduce `WeightQuantizer` dispatch based on rollout overrides or the model's native Hugging Face quantization config. - Export canonical HF-named tensors from Megatron, quantize them on the trainer side, and transfer the final checkpoint layout through the checkpoint engine. - Build compressed-tensors configs from per-layer presets and enforce QAT/rollout precision compatibility at startup. - Add vLLM staging and post-load processing for FP8, MXFP8, MXFP4, and NVFP4 refits. - Pass compressed-tensors model overrides to SGLang for FP8_BLOCK rollout. - Keep default high-precision layers unquantized for both generated compressed-tensors configs and native FP8 configs. - Consolidate low-precision documentation and provide `scripts/rollout_logprob_diff.py` for numerical comparison. - Remove superseded ModelOpt/vLLM monkey-patch utilities and the TRT-LLM FP8 E2E job. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [ ] Add unit or end-to-end tests to CI to cover all code. New CPU tests and updated FP8 E2E jobs are included, but ModelOpt/compressed-tensors test dependency coverage and MXFP4/NVFP4 E2E coverage still need confirmation. - [ ] Once this PR is ready for CI, send a message in the [`ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1). - [x] Recipe submodule update is not applicable; this PR does not change the recipe submodule. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…ect#8306) ### What does this PR do? Follow up verl-project#8287, add SGLang online weight refit for MXFP8, routed-expert MXFP4, and routed-expert NVFP4 rollouts. The trainer exports pre-quantized checkpoint tensors, while a bucket-aware SGLang custom loader stages canonical checkpoint layouts and rebuilds backend-specific kernel layouts after the final sync bucket without replacing CUDA-graph-captured storage. The implementation: - translates an all-MXFP8 compressed-tensors rollout configuration to SGLang's native `quant_method: mxfp8` schema and exports `weight_scale_inv`; - supports BF16 dense layers with MXFP4×MXFP8 or NVFP4 W4A4 routed experts; - stages MXFP8 scales and MXFP4/NVFP4 expert tensors across all weight-sync buckets, then reprocesses them once after the final bucket; - restores live storage if loading or post-processing fails; - pins validated FP8/MXFP8/MXFP4 SGLang paths to DeepGEMM while leaving NVFP4 backend selection to SGLang; - rejects the unsupported SGLang combination `linear=MXFP8, moe=MXFP4`; - rejects `delta_flush` for custom quantized refit because finalization requires the complete bucket sequence; - documents configuration, backend constraints, support status, and measured trainer–rollout mismatch. This PR overlaps with [verl-project#8130](verl-project#8130) on SGLang MXFP8 export and refit. The additional scope here is a unified refit lifecycle shared by MXFP8, MXFP4, and NVFP4, including FP4 checkpoint-layout staging, backend-specific in-place reconstruction, and Blackwell validation for all three formats. It also overlaps conceptually with [verl-project#7971](verl-project#7971), but that PR implements native NVFP4 training and vLLM rollout; this PR adds the SGLang NVFP4 refit path. AI assistance was used for implementation review, debugging, and preparation of this PR description. The human submitter must review every changed line and confirm the validation results before submission. ### Checklist Before Starting - [x] Search for similar PRs: [SGLang QAT](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+sglang+qat), [SGLang MXFP4](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+sglang+mxfp4), and [SGLang weight refit](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+sglang+weight+refit). - [x] Format the PR title as `[{modules}] {type}: {description}`. ### Test The mismatch measurements below use Qwen3.5-35B-A3B with Megatron EP=8, SGLang rollout TP=2, Blackwell, and the default CUDA Graph configuration: | Rollout | Backend | `k3_kl` without QAT | `k3_kl` with matching QAT | |---|---|---:|---:| | MXFP8 linear + MoE | DeepGEMM linear + MoE | 0.00324 | 0.00327 | | BF16 linear + MXFP4 MoE | DeepGEMM MoE | 0.00571 | 0.00200 | | BF16 linear + NVFP4 MoE | FlashInfer CUTLASS MoE | 0.00946 | 0.00653 | Local checks performed while preparing the branch: ```bash python -m compileall -q verl/utils/sglang/sglang_quant_utils.py git diff --check ``` The SGLang CPU test suite could not be completed in the current environment because importing `torchcodec` failed to resolve `libc10_cuda.so`. The final branch currently does not add committed unit tests; reviewers should treat the Blackwell measurements above as the available validation and require focused CPU coverage if needed. ### API and Usage Example All-MXFP8 rollout: ```yaml actor_rollout_ref: rollout: name: sglang quantization: linear: MXFP8 moe: MXFP8 ignore: - "re:.*linear_attn\\.in_proj_[ab]$" actor: megatron: qat: enable: true mode: mxfp8 ignore_patterns: - "*linear_attn.in_proj_a" - "*linear_attn.in_proj_b" ``` MXFP4 routed experts with BF16 dense layers: ```yaml actor_rollout_ref: rollout: name: sglang quantization: linear: null moe: MXFP4 actor: megatron: qat: enable: true mode: mxfp4 ``` NVFP4 routed experts with BF16 dense layers: ```yaml actor_rollout_ref: rollout: name: sglang quantization: linear: null moe: NVFP4 actor: megatron: qat: enable: true mode: nvfp4 ``` SGLang intentionally rejects `linear: MXFP8` together with `moe: MXFP4`. Use BF16 dense layers for SGLang MXFP4 rollout, or use vLLM for the mixed MXFP8-linear + MXFP4-MoE configuration. ### Design & Code Changes - `verl/utils/sglang/sglang_quant_utils.py` - resolves native and override quantization configurations; - translates all-MXFP8 compressed-tensors configurations to native SGLang MXFP8; - selects the required refit kind and registers one custom loader; - stages each bucket and sends a final marker to trigger post-load reconstruction; - fails fast for unsupported mixed MXFP8-linear + MXFP4-MoE rollout. - `verl/utils/sglang/sglang_fp8_utils.py` - rebuilds dense MXFP8 derived scale layouts; - stages canonical MoE scales and folds rebuilt layouts into captured storage. - `verl/utils/sglang/sglang_fp4_utils.py` - stages MXFP4 and NVFP4 expert checkpoint tensors; - publishes packed-weight aliases expected by SGLang loaders; - rebuilds backend layouts and preserves CUDA Graph storage. - `verl/utils/qat/compressed_tensors.py` - detects all-MXFP8 compressed-tensors configurations; - selects an SGLang-specific MXFP8 exporter that emits native scale names. - `verl/workers/rollout/sglang_rollout/async_sglang_server.py` - prepares the effective SGLang quantization configuration; - registers the refit loader and validated GEMM backends. - `verl/workers/rollout/sglang_rollout/sglang_rollout.py` - uses the custom loader for every bucket; - sends finalization after the last bucket; - rejects the incompatible `delta_flush` wire format. - `docs/low_precision/qat.md` - documents the support matrix, configuration, engine constraints, mismatch measurements, and limitations. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [ ] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always`. - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). - [ ] Add unit or end-to-end test(s) to the CI workflow. Focused tests were developed during the branch history but are not present in the final diff; the current evidence is the Blackwell mismatch validation above. - [ ] Once the PR is ready for CI, send a message in the `ci-request` channel. - [x] This PR is not related to the `recipe` submodule. --------- Co-authored-by: root <root@maas-gpu-gb200-devbox-1.byted.org>
…roject#8127) ### What does this PR do? Adds the **MXFP8 training recipe** (`fp8_recipe: mxfp8`, Blackwell) to verl's Megatron path, an end-to-end example and its documentation, `docs/low_precision/mxfp8.md`. Together with what main already has, verl-project#8287 (the trainer quantizes the rollout weights with compressed-tensors) and verl-project#8306 (SGLang MXFP8 refit), this gives an MXFP8 RL recipe: MXFP8 GEMMs in training through Transformer Engine (TE), and an MXFP8 rollout on vLLM or SGLang. - **Training.** `fp8_recipe: mxfp8` already flows through `override_transformer_config`. The one functional gap was THD sequence padding: - TE's MXFP8 quantizer needs both GEMM dimensions divisible by 32, and the wgrad scale blocks run along the token dimension. - So packed sequences are padded to `lcm(32, align)` per sequence under MXFP8, using Megatron-Core's own `get_fp8_align_size` (the mapping it applies to its MoE FP8 padding). - Every other recipe keeps its previous alignment (`lcm(16, align)`). - **QAT combined with a TE FP8 recipe is rejected** at engine init: QAT fake-quantizes weights inside BF16 GEMMs, while an FP8 recipe switches the GEMMs themselves to FP8. - **Examples**, each with `PRECISION=mxfp8` (MXFP8 end to end) or `PRECISION=bf16` (the same job as the BF16 baseline): - `examples/mxfp8/run_qwen3_8b_megatron.sh`: GRPO on Qwen3-8B with GSM8K; the defaults fit one B200. - `examples/mxfp8/run_qwen3_30b_a3b_megatron.sh`: GRPO on Qwen3-30B-A3B (MoE) with DAPO-Math-17k on one 4×B200 node, experts included. - **Docs.** `docs/low_precision/mxfp8.md` (new, in the Low Precision toctree): the configuration, which kernels the vLLM and SGLang rollouts use, the examples with their results, and earlier runs on DAPO-Math-17k. Nothing changes unless an FP8 recipe is configured. This was part 1 of verl-project#7519's split; as the maintainers asked, it now stands alone (the consolidated verl-project#8314 was closed in favor of it). Making the THD packing helpers recipe-agnostic, as suggested in the review, follows in a separate PR. ### Checklist Before Starting - [x] Search for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+mxfp8. No other open PR adds MXFP8 training on the Megatron path. verl-project#7971 (NVFP4 training, open) changes the same THD padding code for FP4; whichever lands second rebases. - [x] PR title formatted as `[megatron] feat: ...` ### Test ```bash python -m pytest tests/utils/test_megatron_fp8_padding_on_cpu.py tests/utils/test_mxfp8_ignored_layers_on_cpu.py -q # 11 passed (with compressed-tensors 0.17.0; without it 2 are skipped) # existing tests of the paths this PR touches: python -m pytest tests/utils/test_megatron_bshd_preprocess.py tests/utils/test_config_router_replay_on_cpu.py \ tests/utils/test_modelopt_mxfp8_on_cpu.py tests/workers/engine/megatron/test_router_replay_utils_on_cpu.py \ tests/utils/megatron/test_router_replay_model_walk_on_cpu.py -q # 94 passed, 1 skipped, 1 failed python -m pytest tests/models/test_model_forward_fused.py -q # 6 passed, 3 failed ``` - **The failure in the second set**, `test_modelopt_mxfp8_on_cpu.py::test_nvfp4_qat_uses_experts_only_export_compatible_backends`, fails the same way on main (954bd28) in the same environment (ModelOpt 0.46.0). - **The three failures in `test_model_forward_fused.py`** check the installed Megatron-Core's forward signature and fail the same way on main in this environment. `test_engine_hook_context_and_current_thd_arguments` passes; it now expects the `fp8_recipe` THD argument. - **Collection order:** `test_modelopt_mxfp8_on_cpu.py` has to be collected before `test_router_replay_model_walk_on_cpu.py`, which installs a `MagicMock` in `sys.modules` for Megatron modules at import. `pre-commit run` passes on all changed files. ### End-to-end results on B200 **The example itself**, one B200, verl's locked environment (vLLM 0.29.0, TE 2.16.1, Megatron-Core 0.19.2, Megatron-Bridge 0.6.2, compressed-tensors 0.17.0), `TOTAL_TRAINING_STEPS=10`, the GSM8K test set (1,319 problems, greedy) at steps 0, 5 and 10: | | GSM8K test accuracy (step 0 / 5 / 10) | training reward (step 1 → 10) | `rollout_corr/kl` | |---|---|---|---| | BF16 | 0.329 / 0.610 / 0.819 | 0.24 → 0.82 | 0.0006–0.0008 | | MXFP8 | 0.372 / 0.616 / 0.803 | 0.25 → 0.80 | 0.0063–0.0071 |  *Blue: BF16, vermillion: MXFP8 end to end.* wandb: [BF16](https://forge.coreweave.com/wandb/wengeezhang-bytedance/verl_grpo_gsm8k_mxfp8/runs/zodbg1pr), [MXFP8](https://forge.coreweave.com/wandb/wengeezhang-bytedance/verl_grpo_gsm8k_mxfp8/runs/cr5dk8fs). - No non-finite metric in either run. - Both are single runs of 10 steps, too few to resolve the 1.6-point gap at step 10 (21 of 1,319 problems). - FlashInfer was available to vLLM's workers, so vLLM picked FlashInfer CuTe-DSL for the MXFP8 dense GEMMs. The example run does not log the choice (verl starts the workers at `VLLM_LOGGING_LEVEL=WARN`); its step-0 accuracy equals, to every digit, that of the CuTe-DSL run below, which logged it. **The same hyper-parameters on each rollout kernel**, launched through our own script with `VLLM_LOGGING_LEVEL=INFO` (validation at steps 0 and 10): | rollout | dense GEMM kernel | GSM8K test accuracy (step 0 → 10) | `rollout_corr/kl` | |---|---|---|---| | vLLM 0.29.0, FlashInfer available | FlashInfer CuTe-DSL (MXFP8 weights and activations) | 0.372 → 0.806 | 0.0064–0.0073 | | vLLM 0.29.0, FlashInfer unavailable | Marlin (MXFP8 weights, BF16 activations) | 0.362 → 0.805 | 0.0041–0.0050 | | SGLang 0.5.20 | DeepGEMM, pinned by verl (MXFP8 weights and activations) | 0.356 → 0.830 | 0.0065–0.0071 | No non-finite metric in any run. Qwen3-8B is dense, so these runs exercise no MoE kernel. **Earlier runs on DAPO-Math-17k** (settings D and B of verl-project#7519, before its rebase on verl-project#8287: vLLM 0.24 served MXFP8 through ModelOpt, and FP8 blockwise through its own FP8 method; the training side is this PR's). wandb: [verl_mxfp8_b200](https://forge.coreweave.com/wandb/wengeezhang-bytedance/verl_mxfp8_b200). Qwen3-30B-A3B, the 30B example's hyper-parameters, one 4×B200 machine for every arm, 10 steps (mean of steps 1–10; "X/Y" is training precision X, rollout precision Y): | | BF16/BF16 | FP8-TW/BF16 | FP8-BW/FP8-BW | MXFP8/MXFP8 | |---|---|---|---|---| | step time (s) | 506 | 487 | 489 | 452 | | generation (s) | 134 | 133 | 115 | 102 | | `update_actor` (s) | 244 | 225 | 241 | 221 | | `rollout_corr/kl` | 0.0018 | 0.0058 | 0.0090 | 0.0045 |  - MXFP8 cuts the step time by 11% (generation −24%, `update_actor` −9%), with the lowest mismatch of the three low-precision arms. Ten steps do not show convergence. - The MXFP8 arm, rerun for 5 steps on the compressed-tensors path (verl-project#7519 rebuilt on verl-project#8287, vLLM 0.29, another 4×B200 machine), read `rollout_corr/kl` 0.0037–0.0040. The 30B example itself has not run on this branch. Qwen3-8B, 2×B200, 32 steps, every arm with an FP8 KV cache (which raises every arm's KL) and the MXFP8 rollout's `gate_up_proj` in BF16 (mean of steps 23–32): | | BF16/BF16 | FP8-TW/BF16 | MXFP8/MXFP8 | |---|---|---|---| | `rollout_corr/kl` | 0.0088 | 0.0297 | 0.0088 | | entropy | 0.220 | 0.083 | 0.338 | | step time (s) | 407 | 345 | 386 |  - MXFP8's mismatch stays at the BF16 level, while FP8-TW with a BF16 rollout drifts from 0.009 to 0.037 and its entropy nearly collapses (0.307 at step 1, 0.062 at step 32). Also from verl-project#7519 (GSM8K, 60 steps): held-out accuracy 0.9393 with MXFP8 vs. 0.9363 with BF16, and the `update_actor` cost per token cut by 43%. ### API and Usage Example No API change. ```yaml # Blackwell (SM100+), Megatron-Core >= 0.13, TE >= 2.1, Megatron-Bridge model path (default) actor_rollout_ref.actor.megatron.override_transformer_config: fp8: "e4m3" # or "hybrid" fp8_recipe: "mxfp8" # MXFP8 rollout on vLLM or SGLang actor_rollout_ref.rollout.quantization: linear: MXFP8 moe: MXFP8 ``` ```bash PRECISION=mxfp8 bash examples/mxfp8/run_qwen3_8b_megatron.sh # Qwen3-8B, GSM8K, one B200 PRECISION=mxfp8 bash examples/mxfp8/run_qwen3_30b_a3b_megatron.sh # Qwen3-30B-A3B, DAPO-Math-17k, 4×B200 # PRECISION=bf16 for the baselines ``` ### Design & Code Changes - `verl/models/mcore/util.py`: - `_compute_fp8_thd_align_size(align, fp8_recipe)` uses `get_fp8_align_size`; - `get_fp8_padding_options` replaces the padding decision repeated at each call site. - `model_forward.py`, `model_forward_fused.py`, `verl/utils/megatron/router_replay_utils.py`: pass `fp8_recipe` to every THD preprocess call. - `verl/workers/engine/megatron/transformer_impl.py`: QAT + TE FP8 rejected. - `examples/mxfp8/run_qwen3_8b_megatron.sh`, `run_qwen3_30b_a3b_megatron.sh` (new): the examples above. - `docs/low_precision/mxfp8.md` (new), listed in `docs/index.rst`. - Tests: - `test_megatron_fp8_padding_on_cpu.py`: alignment math and both THD preprocess paths; - `test_mxfp8_ignored_layers_on_cpu.py`: - the exporter's default ignore list keeps `lm_head`, the embedding (also under a VL model's `language_model` prefix) and the routers in BF16, but not `gate_up_proj`; - an MXFP8 export with a user ignore list leaves `lm_head` and the embedding in BF16; - `test_model_forward_fused.py`: the expected THD arguments include `fp8_recipe`. ### AI assistance disclosure Developed with AI assistance. I reviewed every changed line, ran the tests and the B200 runs above, and can defend the change end to end. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs): `docs/low_precision/mxfp8.md`. - [x] Add unit test(s): CPU tests above; B200 runs above. - [ ] Once your PR is ready for CI, send a message in the `ci-request` channel. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…of top-k distillation tensors (verl-project#8325) ### What does this PR do? Follow-up to verl-project#8127, for two of its review threads. 1. **Recipe-agnostic THD padding** ([review](verl-project#8127 (comment))). `preprocess_packed_seqs` and `preprocess_thd_engine` took two FP8-specific arguments, `use_fp8_padding` and `fp8_recipe`. They now take one integer, `quant_align_size`: the token alignment each packed sequence needs for the model's low-precision GEMMs (`None`: no low-precision padding). `get_thd_quant_align_size(config)` derives it from the `TransformerConfig`, for FP8 recipes through Megatron-Core's `get_fp8_align_size` (32 for MXFP8, 16 otherwise), so a new format such as NVFP4 or MXFP4 (verl-project#7971) adds a branch there instead of another parameter. The model forward, the fused forward and router replay all call it. The padding itself is unchanged. 2. **Top-k distillation under FP8** ([review](verl-project#8127 (comment))). `compute_forward_kl_topk` packs the teacher's top-k tensors itself and asserts they match the student logits, but it packed them without the student's FP8 padding, so Megatron top-k distillation failed that assertion under any FP8 recipe; the gap predates MXFP8. The engine now passes `quant_align_size` to the Megatron loss, which packs the teacher tensors with it on THD and with `use_fp8_padding` derived from it on BSHD. Left out: on BSHD, `pad_bshd_to_minibatch_max` (on by default) pads the student micro-batch to the mini-batch max (`forced_max_seqlen`) while the teacher tensors are not padded to match. That fails the same assertion even in BF16, and verl-project#8061 fixes it. The two PRs touch neighbouring lines in `compute_forward_kl_topk`, so whichever lands second needs a small rebase. ### Checklist Before Starting - [x] Search for similar PRs: [quant_align_size](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+quant_align_size), [distillation padding](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+distillation+padding), [pad_bshd_to_minibatch_max](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+pad_bshd_to_minibatch_max). - Not a duplicate: verl-project#8061 fixes only the BSHD `forced_max_seqlen` mismatch, which this PR leaves to it; verl-project#7887 handles metrics of empty-response micro-batches; verl-project#7971 (native NVFP4 training) adds its own padding flag to the same helpers and could add an NVFP4 branch to `get_thd_quant_align_size` instead. - [x] Format the PR title as `[{modules}] {type}: {description}`. `[BREAKING]` because two public helpers change keyword arguments (see below). ### Test CPU unit tests and pre-commit: ```bash pytest -q tests/utils/test_megatron_fp8_padding_on_cpu.py tests/workers/test_megatron_distillation_only_on_cpu.py \ tests/utils/test_megatron_bshd_preprocess.py tests/utils/test_config_router_replay_on_cpu.py \ tests/workers/engine/megatron/test_router_replay_utils_on_cpu.py tests/utils/megatron/test_router_replay_model_walk_on_cpu.py # 90 passed, 1 skipped pytest -q tests/models/test_model_forward_fused.py # 6 passed, 3 failed; the same 3 fail on main in the same environment pre-commit run --files <changed files> # passed ``` New tests cover the alignment helper, both THD preprocess paths with and without the padding, the teacher's top-k packing on THD (MXFP8 and a 16-token FP8 recipe) and BSHD, and that the engine and `compute_topk_loss` pass `quant_align_size` to the Megatron loss only. On hosts without Megatron, the tests stub Megatron-Core's `get_fp8_align_size` with the same 32/16 rule. End to end on 2×B200: - A tiny random Qwen3-MoE (2 layers, 8 experts, top-2) on GSM8K, 5 steps per run, the trainer on one GPU (TP, EP and CP 1). - MXFP8 training with Transformer Engine (`fp8_recipe=mxfp8`) and an MXFP8 vLLM rollout, except the last row. - The distillation runs put the teacher (the same model, BF16, served by vLLM) on the second GPU, with `forward_kl_topk` and top-16. - vLLM 0.29.0, Transformer Engine 2.16.1, Megatron-Core 0.19.2. - "This PR" is this PR's files, byte for byte, on verl-project#8127's head before it merged. That predates verl-project#8308, so the vLLM rollout used FlashInfer CuTe-DSL for the dense layers and FlashInfer TRT-LLM for the experts. "Before" is verl-project#8127's head without this PR. | run | code | result | |---|---|---| | GRPO, THD | this PR | passes: 5/5 steps, no non-finite metric, `rollout_corr/kl` 0.0019–0.0023 | | GRPO + router replay R2, THD | this PR | passes: `rollout_corr/kl` 0.0022–0.0027 | | top-k distillation, THD | this PR | passes: `rollout_corr/kl` 0.0018–0.0026, distillation metrics logged | | top-k distillation, THD | before | fails the shape assertion in `compute_forward_kl_topk` | | GRPO, THD | before | passes: `rollout_corr/kl` 0.0020–0.0030 (control) | | top-k distillation, BSHD | this PR | fails the same assertion: the `forced_max_seqlen` mismatch that verl-project#8061 fixes | | top-k distillation, BSHD | this PR + the micro-batch's `forced_max_seqlen` passed to the teacher packing, as in verl-project#8061 | passes: `rollout_corr/kl` 0.0018–0.0027 | | top-k distillation, BSHD, BF16 | before | fails the same assertion: that mismatch does not depend on FP8 | `rollout_corr/kl` is the KL between vLLM's and the trainer's log-probs. The fused forward, FP8 recipes other than MXFP8, and TP or CP above 1 were not run on hardware; the unit tests above cover them. ### API and Usage Example No config or CLI change. `preprocess_packed_seqs` and `preprocess_thd_engine` take `quant_align_size` instead of `use_fp8_padding` and `fp8_recipe`. `preprocess_bshd_engine` keeps `use_fp8_padding`, which verl-omni passes. Within verl-project, nothing else calls the THD helpers with the removed arguments. ```python from verl.models.mcore.util import get_thd_quant_align_size, preprocess_thd_engine quant_align_size = get_thd_quant_align_size(tf_config) # 32 for MXFP8, 16 for other FP8 recipes, None without FP8 packed, *_ = preprocess_thd_engine(input_ids, pre_process=True, quant_align_size=quant_align_size) ``` ### Design & Code Changes - `verl/models/mcore/util.py`: `get_thd_quant_align_size(config)`; the THD helpers take `quant_align_size`. Each sequence is padded to a multiple of `lcm(quant_align_size, align_size)` and the total to `align_size * 128`, as before. - `verl/models/mcore/model_forward.py`, `model_forward_fused.py`: derive `quant_align_size` from the helper; BSHD uses `use_fp8_padding = quant_align_size is not None`. - `verl/utils/megatron/router_replay_utils.py`: router replay records and replays with the helper instead of inlining the rule. - `verl/workers/engine/megatron/transformer_impl.py`: `_lm_head_logits_processor` passes `quant_align_size` to the top-k loss. - `verl/trainer/distillation/losses.py`: `compute_topk_loss` and `distillation_ppo_loss` forward it to the Megatron loss only. - `verl/trainer/distillation/megatron/losses.py`: `compute_forward_kl_topk` packs the teacher's top-k tensors with it. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting). - [x] Add / Update [the documentation](https://github.com/verl-project/verl/tree/main/docs). No user-facing change to document. - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/verl-project/verl/tree/main/.github/workflows) to cover all the code: the new tests are CPU unit tests (`*_on_cpu.py`) that CI runs. - [ ] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1). - [x] Not related to the `recipe` submodule. AI assistance was used to write the code, the tests and this description. The submitter reviewed the changes and ran the tests above. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…r QAT (verl-project#8361) ### What does this PR do? Fixes two ways Megatron QAT breaks because of the layer spec the engine builds the model from, and corrects the Hopper note in the QAT guide. Both came out of the MXFP4 recipe checks reported in verl-project#8186. **1. Models whose Megatron-Bridge bridge sets its own layer spec fail under QAT.** With QAT enabled, the engine replaced every provider's `transformer_layer_spec` with `modelopt_transformer_layer_spec`. Megatron-Bridge maps the HF weights onto the modules of the spec the bridge chose, and more than twenty bridges set their own (OLMoE, Qwen3-Next, text-only Qwen3.5, DeepSeek-V2/V3/V4, GLM, Kimi, MiniMax, ...). Under the ModelOpt spec those models lose modules. OLMoE fails at load: ``` ValueError: Shape mismatch for megatron param decoder.layers.0.self_attention.k_layernorm.weight ``` By the same code path, Qwen3-Next and text-only Qwen3.5 would lose their linear-attention layers (from reading the bridges; not run). Qwen3.5 checkpoints (`Qwen3_5MoeForConditionalGeneration`) only escape because the Qwen3.5 VL provider builds its language block without reading `transformer_layer_spec`, so their QAT already runs on the TE spec. The engine now switches to the ModelOpt spec only when the provider uses Bridge's default GPT spec (`default_layer_spec`). Any other spec is kept, and ModelOpt quantizes its Transformer Engine modules in place (`TELayerNormColumnParallelLinear`, `TERowParallelLinear`, the grouped expert linears, ...). Models on the default spec are unchanged. **2. Packed THD sequences fail at the first forward when context parallelism is 1** (`use_remove_padding=True`, the default): ``` AssertionError: Attention mask type must be padding or padding_causal for qkv_format=thd! ``` When `use_arbitrary_attention_mask` is unset, `modelopt_transformer_layer_spec` turns it on for CP=1; the arbitrary mask is meant for speculative-decoding training. Transformer Engine then rejects verl's THD inputs. This PR defaults the flag to `False` under the ModelOpt spec, which keeps the causal mask that the non-QAT TE spec uses. An explicit `override_transformer_config.use_arbitrary_attention_mask` still wins. **3. Docs.** `docs/low_precision/qat.md`: - The `MXFP4` preset has no Marlin fallback: its MXFP4-weight × FP8-activation experts are served only by vLLM's DeepGEMM FP4 MoE kernel (SM100/SM120), so on Hopper the rollout fails at engine start. - New limitation: the ModelOpt spec's expert module (`SequentialMLP`) rejects tensor and expert parallelism together, so default-spec models need TP=1 or EP=1 under QAT. Models with their own spec keep it. ### Checklist Before Starting - [x] Search for similar PRs: no open PR or issue mentions `use_arbitrary_attention_mask`, `modelopt_transformer_layer_spec`, the THD assertion, or QAT with a model-specific layer spec (searched `use_arbitrary_attention_mask`, `qat thd`, `modelopt_transformer_layer_spec`, `QAT layer spec`, `qat transformer_layer_spec`, `QAT olmoe`, `QAT custom spec`, and the error text). verl-project#7971 changes the same file but not the QAT layer spec. - [x] Format the PR title as `[{modules}] {type}: {description}`. ### Test ```bash python -m pytest -q tests/workers/test_megatron_qat_layer_spec_on_cpu.py tests/workers/test_megatron_*_on_cpu.py \ tests/workers/test_engine_*_on_cpu.py tests/workers/test_router_replay_engine_helpers_on_cpu.py \ tests/utils/test_mxfp8_ignored_layers_on_cpu.py # 80 passed bash scripts/generate_trainer_config.sh # All good pre-commit run --files <changed files> # ruff, ruff format, mypy, docs time, docstring, license and structure checks pass ``` The CPU test covers the default-spec switch, an explicit mask override, and keeping a bridge-specific spec. Spec chosen by `_use_qat_layer_spec` on real Megatron-Bridge 0.6.2 providers (CPU, built from the HF configs as the engine does, no weights loaded): | Model | Bridge spec | Spec under QAT | |---|---|---| | OLMoE-1B-7B-0125-Instruct | `olmoe_layer_spec` | kept (was replaced) | | Qwen3-MoE | `default_layer_spec` | `modelopt_transformer_layer_spec` (unchanged) | | Qwen3.5-35B-A3B | experimental attention variant | kept (the VL provider ignores the field anyway) | | Qwen3.5 MoE, text-only config | experimental attention variant | kept (was replaced) | GPU, 1× GH200 with the stock wheelhouse TE, `scripts/rollout_logprob_diff.py --n-gpus 1 --tp 1 --gen-tp 1 --ep 1 --quant-linear FP8_BLOCK --quant-moe FP8_BLOCK`, OLMoE-1B-7B-0125-Instruct: | Code | QAT | Result | k3 KL | |---|---|---|---| | THD fix only (first commit) | `--qat-mode fp8` | fails at load with the `k_layernorm` shape mismatch above | — | | this PR | `--qat-mode fp8` | runs; 128 enabled quantizers, all on TE modules (layernorm-column, row, grouped fc1 and fc2 linears) | 6.3e-3 | | this PR | off | runs | 9.1e-3 | - The rollout starts from dummy weights and receives the trainer's export through `update_weights()`. The QAT run therefore also exercises exporting the quantized TE modules, grouped experts included, through the OLMoE bridge. - The quantizer count came from a temporary debug print that is not part of this PR. - OLMoE is the only model with its own spec that I ran on GPU. The other bridge-specific models now keep their own spec by the same code path; I did not run them. THD part, `scripts/rollout_logprob_diff.py` with Megatron QAT, CP=1, `use_remove_padding=True`, on a small Qwen3-MoE (random init, 4 layers, 32 experts): - **Before the fix**, on GH200 with the stock wheelhouse TE (`--quant-linear FP8_BLOCK --quant-moe FP8_BLOCK --qat-mode fp8`) and on an RTX 5090 (`--quant-moe MXFP4 --qat-mode mxfp4`): the assertion above fires at the first actor forward. - **With the equivalent override** `+actor_rollout_ref.actor.megatron.override_transformer_config.use_arbitrary_attention_mask=False`: both run to completion. On the 5090, MXFP4 QAT gives k3 KL 2.3e-4, against 1.7e-3 for the same MXFP4 rollout without QAT. - These THD runs used the override, not this branch. The code path in this PR sets the same provider field through `provider_overrides`. Hopper doc claim: on GH200 with vLLM 0.29, loading the trainer-exported MXFP4 preset fails for every backend: - `auto` raises `NotImplementedError: No MXFP4 MoE backend supports the deployment configuration`; - `deep_gemm` reports an unsupported device; - `marlin` and `triton` reject the FP8 1x128 activation scheme; - `rollout_logprob_diff.py --quant-moe MXFP4` fails at engine start. ### Design & Code Changes - `verl/workers/engine/megatron/transformer_impl.py`: QAT spec selection moves into `_use_qat_layer_spec(provider, provider_overrides)`. It returns early unless the provider uses `default_layer_spec`; otherwise it sets `modelopt_transformer_layer_spec` and `provider_overrides.setdefault("use_arbitrary_attention_mask", False)`. - `tests/workers/test_megatron_qat_layer_spec_on_cpu.py`: new CPU test. - `docs/low_precision/qat.md`: Hopper note for the `MXFP4` preset and the QAT layer-spec limitation. ### Checklist Before Submitting - [x] Read the [Contribute Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md). - [x] Apply pre-commit checks. - [x] Add / Update the documentation. - [x] Add unit tests (CPU). AI assistance was used for this change. I reviewed every changed line and ran the tests above. --------- Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: wuxibin <wuxibin@bytedance.com>
What does this PR do?
Add end-to-end native NVFP4 W4A4 training and rollout on NVIDIA Blackwell GPUs, demonstrated with Qwen3-30B-A3B-Base DAPO using Megatron/Transformer Engine and vLLM.
This contributes to Sophia's low-precision roadmap #5972, specifically NVFP4 real quant and NVFP4 training in Megatron. In brief:
moe: NVFP4andmoe_activation: nvfp4_per_tokenafter each policy update.Attention and the configured boundary-layer MLPs remain BF16. The Qwen3 example keeps the first two and last four MLP layers in BF16 and applies NVFP4 to the remaining 42 routed-expert MLP layers.
Checklist Before Starting
[{modules}] {type}: {description}.Related work: #6455 targets ModelOpt/TRT-LLM synchronization; #6992 provides FSDP W4A8 simulation; #7895 targets DeepSeek MXFP4 on TE FP8; #7519 adds MXFP8 training and rollout. #8287 adds ModelOpt NVFP4 QAT with static activation scales; this PR adds native TE NVFP4 training with per-token activation scaling, on the same export path.
Test
Unit tests (CPU and GB200) pass; scoped Ruff, formatting and lockfile checks pass. The trainer-side export is bit-identical to vLLM's online
nvfp4_per_tokenquantization, and 6-step runs on Qwen3-30B-A3B (8 × 4 GB200) and a small DeepSeek-V3-layout model pass.Long run
BF16 and W4A4 runs of the example script below (Qwen3-30B-A3B-Base, DAPO, R3, token-level TIS) on 8 nodes × 4 GB200 GPUs, 700 steps with checkpoint resume across jobs. W4A4 vs BF16:
Training score (
critic/score/mean)Entropy (
actor/entropy)Train–rollout mismatch (
rollout_corr/kl)Gradient norm (
actor/grad_norm)Performance: at matched response length, NVFP4 rollout generation is about 1.3× faster than BF16 once the mean response exceeds ~2.5k tokens, giving roughly 1.2× shorter end-to-end steps; NVFP4 training itself is still slower than BF16 in Transformer Engine.
API and Usage Example
With a Ray cluster running on eight nodes, provide model and dataset paths shared by all workers:
Use
PRECISION_MODE=bf16for the baseline. NVFP4 training is enabled like FP8 training, throughactor_rollout_ref.actor.megatron.override_transformer_config(fp4: e2m1,fp4_recipe: nvfp4, andfirst_last_layers_bf16withnum_layers_at_start_in_bf16/num_layers_at_end_in_bf16), and rollout throughactor_rollout_ref.rollout.quantization.moe=NVFP4withmoe_activation=nvfp4_per_token, whoseignorelist keeps the same layers in BF16; the trainer checks that the two agree. The example uses the standard verl PPO entry point, parquet datasets and built-in reward manager, with the DAPO recipe (clip-higher, token-level loss, dynamic sampling and the overlong penalty), R3 routing replay and token-level TIS. See the example guide for configuration and runtime setup.Design & Code Changes
moe_activation=nvfp4_per_tokenselects a compressed-tensors scheme (NVFP4 weights, fully dynamic NVFP4 inputs). The trainer quantizes the experts with its own Triton kernel, sharing one global scale acrossgate_proj/up_proj, bit-identical to vLLM's onlinenvfp4_per_tokenquantization. In vLLM the scheme selects FlashInfer TRT-LLM with per-token activations; refits load each bucket in place into the expert storage captured by CUDA graphs and post-process each MoE layer, so the extra memory is one MoE layer.The validated Megatron + vLLM stack uses:
SGLang end-to-end validation remains follow-up work.
Checklist Before Submitting
AI assistance: OpenAI Codex and Anthropic Claude assisted with implementation, testing and this PR description.