Skip to content

[megatron, vllm, env] feat: add native NVFP4 training and rollout with live refit - #7971

Merged
wuxibin89 merged 180 commits into
verl-project:mainfrom
zhangyimi:feat/native-nvfp4
Nov 2, 2026
Merged

wuxibin89 merged 180 commits into
verl-project:mainfrom
zhangyimi:feat/native-nvfp4

Conversation

@zhangyimi

@zhangyimi zhangyimi commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Add end-to-end native NVFP4 W4A4 training and rollout on NVIDIA Blackwell GPUs, demonstrated with Qwen3-30B-A3B-Base DAPO using Megatron/Transformer Engine and vLLM.

This contributes to Sophia's low-precision roadmap #5972, specifically NVFP4 real quant and NVFP4 training in Megatron. In brief:

  • Training: Transformer Engine runs selected routed-expert MLPs in NVFP4 with per-token activation scaling and dequantized backward. Persistent policy weights remain BF16, with full optimizer state.
  • Weight synchronization: following the trainer-side quantization boundary of [BREAKING][megatron, vllm, sglang, rollout, doc] refactor: unify QAT and low-precision rollout #8287, the trainer exports the routed experts as NVFP4 with moe: NVFP4 and moe_activation: nvfp4_per_token after each policy update.
  • Rollout: vLLM loads the NVFP4 expert weights and quantizes activations per token, matching training. Refit preserves the storage referenced by CUDA graphs.

Attention and the configured boundary-layer MLPs remain BF16. The Qwen3 example keeps the first two and last four MLP layers in BF16 and applies NVFP4 to the remaining 42 routed-expert MLP layers.

Checklist Before Starting

  • Search for similar PRs: NVFP4, roadmap #5972.
  • Format the PR title as [{modules}] {type}: {description}.

Related work: #6455 targets ModelOpt/TRT-LLM synchronization; #6992 provides FSDP W4A8 simulation; #7895 targets DeepSeek MXFP4 on TE FP8; #7519 adds MXFP8 training and rollout. #8287 adds ModelOpt NVFP4 QAT with static activation scales; this PR adds native TE NVFP4 training with per-token activation scaling, on the same export path.

Test

Unit tests (CPU and GB200) pass; scoped Ruff, formatting and lockfile checks pass. The trainer-side export is bit-identical to vLLM's online nvfp4_per_token quantization, and 6-step runs on Qwen3-30B-A3B (8 × 4 GB200) and a small DeepSeek-V3-layout model pass.

Long run

BF16 and W4A4 runs of the example script below (Qwen3-30B-A3B-Base, DAPO, R3, token-level TIS) on 8 nodes × 4 GB200 GPUs, 700 steps with checkpoint resume across jobs. W4A4 vs BF16:

Training score (critic/score/mean)

image

Entropy (actor/entropy)

image

Train–rollout mismatch (rollout_corr/kl)

image

Gradient norm (actor/grad_norm)

image

Performance: at matched response length, NVFP4 rollout generation is about 1.3× faster than BF16 once the mean response exceeds ~2.5k tokens, giving roughly 1.2× shorter end-to-end steps; NVFP4 training itself is still slower than BF16 in Transformer Engine.

API and Usage Example

With a Ray cluster running on eight nodes, provide model and dataset paths shared by all workers:

MODEL_PATH=/shared/models/Qwen3-30B-A3B-Base \
TRAIN_FILE=/shared/data/train.parquet \
TEST_FILE=/shared/data/validation.parquet \
EXP_NAME=qwen3-native-nvfp4 \
PRECISION_MODE=real_nvfp4 \
bash examples/real_nvfp4/run_qwen3_30b_megatron.sh

Use PRECISION_MODE=bf16 for the baseline. NVFP4 training is enabled like FP8 training, through actor_rollout_ref.actor.megatron.override_transformer_config (fp4: e2m1, fp4_recipe: nvfp4, and first_last_layers_bf16 with num_layers_at_start_in_bf16 / num_layers_at_end_in_bf16), and rollout through actor_rollout_ref.rollout.quantization.moe=NVFP4 with moe_activation=nvfp4_per_token, whose ignore list keeps the same layers in BF16; the trainer checks that the two agree. The example uses the standard verl PPO entry point, parquet datasets and built-in reward manager, with the DAPO recipe (clip-higher, token-level loss, dynamic sampling and the overlong penalty), R3 routing replay and token-level TIS. See the example guide for configuration and runtime setup.

Design & Code Changes

  • Training precision: the Megatron per-module recipe (attention BF16, routed-expert MLPs NVFP4, carved-out layers BF16) is generated from the configuration rather than maintained as a file. After model construction, the effective precision of every attention and expert linear layer is checked against it. Packed THD sequences use the TE alignment for FP4 as for FP8.
  • Rollout config and refit: moe_activation=nvfp4_per_token selects a compressed-tensors scheme (NVFP4 weights, fully dynamic NVFP4 inputs). The trainer quantizes the experts with its own Triton kernel, sharing one global scale across gate_proj/up_proj, bit-identical to vLLM's online nvfp4_per_token quantization. In vLLM the scheme selects FlashInfer TRT-LLM with per-token activations; refits load each bucket in place into the expert storage captured by CUDA graphs and post-process each MoE layer, so the extra memory is one MoE layer.
  • Routing replay: use vLLM’s native fused-MoE routing capture and align replay data with FP4 training padding.
  • Runtime compatibility: Transformer Engine 2.20.2 from the verl wheelhouse. No vLLM source patch is needed.

The validated Megatron + vLLM stack uses:

Component Version
PyTorch 2.13 / CUDA 13
vLLM 0.32.0
Transformers 5.16.1
Megatron-Core 0.19.2
Megatron-Bridge 0.6.2
Transformer Engine 2.20.2
FlashInfer 0.7.0.post1
CUTLASS DSL / QuACK 4.8.0 / 0.6.5

SGLang end-to-end validation remains follow-up work.

Checklist Before Submitting

  • Read the Contribute Guide.
  • Run the complete all-files pre-commit suite. Focused lint, formatting, shell and lockfile checks have passed.
  • Add usage, design and runtime compatibility documentation.
  • Add focused unit and end-to-end coverage. Project CI integration remains to be completed.
  • Request project CI and review.
  • Check recipe coordination: the recipe submodule reference is unchanged; no companion recipe PR is required.

AI assistance: OpenAI Codex and Anthropic Claude assisted with implementation, testing and this PR description.

zhangyimi and others added 30 commits August 31, 2026 06:21
Add a fail-closed Megatron/Transformer Engine W4A4 path, native vLLM 0.26 per-token NVFP4 rollout and layerwise refit, R3 capture support, and a versioned scheduler-gated experiment bundle.

Assisted-by: OpenAI Codex
Build the runtime from tracked archives and record the source commit and lockfile digest so later scheduler stages cannot mix an old image with changed code.

Assisted-by: OpenAI Codex
Restrict the duplicate-submission guard to this experiment bundle instead of blocking on unrelated user jobs.

Assisted-by: OpenAI Codex
Keep the lock check on the submission host and rely on frozen sync inside the target aarch64 container, avoiding cross-architecture uv execution before container startup.

Assisted-by: OpenAI Codex
Use git grep for the no-loss audit and let the probe inventory pip when the base image has no uv binary.

Assisted-by: OpenAI Codex
Install the architecture-native uv wheel when the aarch64 base image does not provide a usable executable, and verify the exact lock-tool version before syncing the frozen runtime.

Assisted-by: OpenAI Codex
Match the exact local-version metadata emitted by the locked Megatron Core source build in both image and preflight checks.

Assisted-by: OpenAI Codex
Exercise 16 tokens per expert as required by the NVFP4 block size. Permit harness-only bundle changes after an image build while continuing to reject any runtime payload drift.

Assisted-by: OpenAI Codex
Default the real-NVFP4 worker flag to false for legacy and test-created workers, align the R3 test double with vLLM 0.26 source structure, and derive a fresh v4 immutable job bundle for the runtime change.

Assisted-by: OpenAI Codex
v11 changed three runtime variables at once and its rollout collapsed to
near-uniform output. Add a v12 bundle that applies only enable_pdl=False on top
of the v8 image, so the startup-hang fix is separated from the post-0.26
online-NVFP4 backports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The single-hunk #50074 backport keeps the first NVFP4 MoE quant config alive
across reloads while _quantize_weights rebinds every scale parameter, so after
the first refit the kernel pairs dummy-init scales with freshly packed weights.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add the v13 long-run bundle on top of the v12 runtime and require a recorded
numeric gate before release. v11 cleared every structural gate while its rollout
emitted near-uniform output, so the audit asserts KL, ESS, entropy, clip ratio,
grad norm and length slope against the verified v8 band.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step time tracks response length (v8 averaged 110s/step at ~1.0k tokens) and a
healthy run grows length, so equal 80-step chunks would overrun the partition
limit near the end. Target 80/150/210/265 instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gb200_30B_bf16_megatron_0603 is 12 same-named runs, one per 5h segment, whose
per-run _step is not the global training step. Record the run ids, the stitching
rule and the resulting curve so the long-run comparison is reproducible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
train.job carries its own copy of the chunk target list and rejects a mismatch
at runtime, so shrinking the chunks in submit.sh alone would have failed chunk 2
only after chunk 1 had run for hours. Sync both copies, align every target to
the save_freq cadence, and assert both invariants during static validation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both rollout attestations read the layer's scale parameters, which were correct
in v11; only the kernel's captured quant config was stale. Record the identity
assertion to add before the upstream PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reward and response length both move a lot between step 20 and step 265, so the
comparison is only meaningful step-for-step. Stitch the BF16 segments once,
cache them, and report matched-step values and slopes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
v12 reproduces the v8 numeric band while keeping 32/32 startup, so enable_pdl=False
is numerics-neutral and the rollout collapse came from the post-0.26 backports.
Record the comparison, the BF16 matched-step result and the two chain defects
caught during v13 preparation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/low_precision/qat.md Outdated
QAT requires a quantized rollout: set `rollout.quantization` or use a checkpoint with a native `quantization_config` that verl supports for export and refit. verl checks at startup that the QAT mode matches the effective rollout configuration. Megatron QAT uses [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) for fake quantization. MXFP8 uses ModelOpt's native `MXFP8_DEFAULT_CFG`; FP8_BLOCK uses a verl backend because ModelOpt's `FP8_DEFAULT_CFG` is per-tensor and its 2D-block preset is weight-only. MXFP4 and NVFP4 use compressed-tensors-compatible QDQ backends so fake quantization is bit-exact with the exported checkpoint format. Both FP4 modes apply only to routed experts. Use `mxfp4_mxfp8` for the vLLM combination where dense linear layers also use MXFP8.
Its `ignore_patterns` use Megatron module names and `fnmatch` syntax. Keep them aligned with the rollout ignore list; for Qwen3.5 MXFP8, add `*linear_attn.in_proj_a` and `*linear_attn.in_proj_b`.

### Native NVFP4 training (Megatron + Transformer Engine, experimental)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move NVFP4 training to separate nvfp4.md, just like:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to docs/low_precision/nvfp4.md (cb01d30, 1d4d593), like fp8/mxfp8; qat.md is back to main.

Comment thread verl/utils/real_nvfp4/config.py Outdated
return any(fnmatch.fnmatch(module_path, pattern) for pattern in ROUTED_EXPERT_PATTERNS)


def real_nvfp4_bf16_layers(override_transformer_config: Mapping[str, Any]) -> Optional[tuple[int, int]]:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it a common function? I think other recipe(e.g mxfp8/mxfp4) could also reuse it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split it in f7b8003: real_nvfp4_enabled only checks the fp4 settings, and the BF16 first/last layer selection is now bf16_layers_at_start_end in verl/utils/megatron/low_precision.py, usable by any recipe.

logger.info(f"Rollout compressed-tensors quantization_config: {quantization_config}")
return {"quantization_config": quantization_config}

def _check_nvfp4_per_token_rollout(self, compilation_config: dict, engine_kwargs: dict) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move _check_nvfp4_per_token_rollout to verl/utils/real_nvfp4/config.py to keep vllm_async_server.py neutral to nvfp4.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to verl/utils/real_nvfp4/config.py as check_nvfp4_per_token_rollout (cb01d30); vllm_async_server.py only calls it.

attest_nvfp4_per_token_runtime(self.model_runner.get_model(), self._nvfp4_per_token_config)

def _update_nvfp4_per_token_weights(self, use_shm: bool):
"""Refit through vLLM's layerwise reload, one bucket at a time.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0: Do not use layerwise reload since it accumulate tensors until all weights in single layer are collected. For large MoE model, this may cause OOM.

@zhangyimi zhangyimi Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced the layerwise reload with an in-place refit like the NVFP4 one in vllm_fp4_utils (cb01d30): buckets load into checkpoint-layout views over the live expert storage, and each layer is post-processed after the last bucket. Extra memory is now one MoE layer (325 MiB vs 2.6 GiB for Qwen3-30B-A3B layers), with bit-identical logprobs.

zhangyimi and others added 4 commits October 28, 2026 00:31
- Load refit buckets into checkpoint-layout views over the live expert storage
  and post-process each layer after the last bucket, instead of vLLM's
  layerwise reload; the extra memory is one MoE layer.
- Move the NVFP4 training docs to docs/low_precision/nvfp4.md.
- Share the BF16 first/last layer selection in verl/utils/megatron/low_precision.py.
- Move the NVFP4_PER_TOKEN rollout check to verl/utils/real_nvfp4/config.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
real_nvfp4_enabled only checks override_transformer_config.fp4; callers take the
BF16 first/last layers from verl.utils.megatron.low_precision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…4 docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zhangyimi

zhangyimi commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

@wuxibin89 On the vLLM patches this PR still carries (vLLM 0.32):

  1. Per-token compressed-tensors MoE (install_nvfp4_per_token_compressed_tensors_patch): vLLM supports per-token NVFP4 activations for ModelOpt ([quantization] Add per-token NVFP4 quantization MoE support for ReLU2 vllm-project/vllm#56740) but not for compressed-tensors. Upstream PR: [Quantization] Support nvfp4_per_token MoE activation override for compressed-tensors vllm-project/vllm#64209. Once it lands, verl only passes the activation override, and the subclass goes away.
  2. clear_kv_cache at start-up: with dummy weights the scales are zero, so per-token quantization of the all-zero MoE rows writes NaN into the KV cache. vLLM main has since changed dummy initialization; we will drop this after the next vLLM upgrade, once verified.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# get_model() unwraps vLLM's CUDA-graph wrapper.
attest_nvfp4_per_token_runtime(self.model_runner.get_model(), self._nvfp4_per_token_config)

def _update_nvfp4_per_token_weights(self, use_shm: bool):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why vLLMColocateWorkerExtension still need special process for nvfp4_per_token , can we align with nvfp4 weight refit in verl/utils/vllm/vllm_fp4_utils.py?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aligned in a1677fe: NVFP4_PER_TOKEN now uses the shared quantized refit, with no per-token code in update_weights_from_ipc.

zhangyimi and others added 4 commits October 28, 2026 22:31
The worker no longer has its own NVFP4_PER_TOKEN refit: the per-token experts go
through prepare/process_quanted_weights like the other formats, with the per-token
refit check and attestation around it. The NVFP4 refit in vllm_fp4_utils now takes
the per-token TRT-LLM experts and restores the parameters TRT-LLM re-registers, and
the shared post-process re-runs the BF16 routed-expert layers of a quantized model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update_weights_from_ipc no longer has NVFP4_PER_TOKEN code: the per-bucket
refit check is gone and the g1_scale_c check moves into the shared NVFP4
kernel scale attestation. The worker only attests the routed-expert
precision once at start-up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rollout.quantization.moe=NVFP4 with moe_activation=nvfp4_per_token replaces
the NVFP4_PER_TOKEN preset. The activation name follows vLLM's
quantization_config.moe.activation. The exported quantization_config and the
numerics are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wuxibin89
wuxibin89 previously approved these changes Oct 30, 2026
@wuxibin89
wuxibin89 merged commit 6879e1a into verl-project:main Nov 2, 2026
87 of 111 checks passed
wuxibin89 added a commit that referenced this pull request Nov 3, 2026
…8494)

### What does this PR do?

NVFP4 QAT now uses per-token activations, the same scheme as the NVFP4
rollout from #7971. The static per-tensor activation scale is removed.

Why per-token:
- Each token is quantized with a scale from its own amax, independent of
the other tokens in the batch and of earlier training steps. The static
scale was a running max of the observed amax, so tokens much smaller
than the largest activation seen so far lost block-scale range, and it
was 1.0 until the first training step.
- Much lower training-inference mismatch in RL training: mean
`rollout_corr/k3_kl` over 100 steps 0.060 → 0.0033, the same as native
NVFP4 training (see Test).

Changes:
- `rollout.quantization.moe: NVFP4` always uses per-token activations;
`moe_activation` now defaults to `nvfp4_per_token`.
- Fake quantization matches FlashInfer's per-token activation
quantization and the trainer-side weight export bitwise.
- **Breaking:** removes the static NVFP4 refit for SGLang. verl does not
support per-token NVFP4 with SGLang yet.
- **Breaking:** removes FSDP QAT (`QATLinear`, `QATQuantizer`,
`actor.qat` and the FSDP engine's `qat` config). It has been unreachable
since #8287: its `w4a16`/`w4a4` modes match no rollout preset, and the
FSDP engine has no trainer-side rollout quantizer. QAT requires the
Megatron engine.
- Merges `nvfp4_qat.md` into `qat.md`.

### Test

RL training, Qwen3-30B-A3B-Base on DAPO-Math-17k, 8x4 GB200, 100 steps.
Routed experts in NVFP4 in training and rollout, layers 0-1 and 44-47 in
BF16.

<img width="3133" height="755" alt="image"
src="https://github.com/user-attachments/assets/f0858fdf-e8df-4784-bcfc-393acf4506e9"
/>

Mean over 100 steps:

| Training | Rollout activation | `rollout_logprobs_diff_mean` | `k3_kl`
| `rollout_is_mean` |
|---|---|---|---|---|
| NVFP4 QAT, before this PR | static per-tensor | 0.124 | 0.060 | 0.979
|
| NVFP4 QAT | per-token | 0.021 | 0.0033 | 0.9996 |
| Native NVFP4 training (#7971) | per-token | 0.021 | 0.0033 | 0.9996 |

`scripts/rollout_logprob_diff.py` on Qwen3.5-35B-A3B, Transformer Engine
2.20.2, ModelOpt 0.47.

`qat.md` mismatch table setting (EP=8, vLLM TP=2), median of 3 runs:

| Rollout | Training | k3_kl |
|---|---|---|
| NVFP4 MoE | BF16 | 0.00966 |
| NVFP4 MoE | NVFP4 QAT | 0.00766 |

Unit tests:
- new `tests/utils/test_nvfp4_qat_fake_quant_on_gpu.py`: bitwise checks
against vLLM / FlashInfer;
- updated `tests/utils/test_modelopt_mxfp8_on_gpu.py`: NVFP4 per-token
backends;
- updated `tests/utils/real_nvfp4/test_nvfp4_per_token_on_cpu.py`.

### API and Usage Example

```yaml
actor_rollout_ref:
  actor:
    megatron:
      qat:
        enable: true
        mode: nvfp4
  rollout:
    quantization:
      moe: NVFP4
```

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [x] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/verl-project/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: the GPU tests need
Blackwell.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: wuxibin <wuxibin@bytedance.com>
linmuchuiyang added a commit to linmuchuiyang/verl that referenced this pull request Nov 4, 2026
Only docs/low_precision/qat.md's 'Last updated' line conflicted. The CPU test
loads vllm_quant_utils by path with its vLLM-facing siblings stubbed, so it now
also stubs the two modules main imports there since verl-project#7971 / verl-project#8494 / verl-project#8483
(vllm_nvfp4_per_token_utils, vllm_unquant_utils) and drops the removed
install_nvfp4_cutlass_compressed_tensors_patch stub.

Signed-off-by: linmuchuiyang <6169320+linmuchuiyang@users.noreply.github.com>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…layout changes shape (verl-project#7987)

### What does this PR do?

Fixes verl-project#7978: the bf16 MoE weight sync crashes on Blackwell when vLLM
picks its default FlashInfer TRT-LLM MoE backend, because that backend
keeps the expert weights in a 4-D block layout that the
checkpoint-format loads no longer fit. This revision stages the
checkpoint layout **in place** (a view over the live storage) and folds
the re-derived kernel layout back layer by layer. It replaces the first
revision's use of vLLM's layerwise reload, which buffered whole layers
(see review feedback below).

**Root cause (verified against vLLM 0.24.0 and 0.29.0 source).**
`update_weights_from_ipc` feeds the bucketed checkpoint-format tensors
straight into the live parameters via `model.load_weights` and runs
`process_weights_after_loading` once at the end. That holds only while
the kernel layout equals the checkpoint layout up to a same-shape
in-place transform (Triton MoE, FlashInfer CUTLASS). On SM100 vLLM
auto-selects the FlashInfer TRT-LLM bf16 MoE backend
(`fused_moe/oracle/unquantized.py::_get_priority_backends` puts it first
on non-Hopper CUDA), whose
`convert_moe_weights_to_flashinfer_trtllm_block_layout` turns
`w13_weight` / `w2_weight` into 4-D BlockMajorK tensors.
`RoutedExperts.weight_loader` then indexes `param.data[expert_id]` (3-D)
and `_load_w13(shard_dim=0)` rejects it at the first sync:

```
ValueError: shard_dim=0 is not a valid data dimension for a 3D tensor (expected 1 or 2)
```

The conversion is one-way (it cannot be re-applied to its own 4-D
output), so the end-of-sync `process_weights_after_loading` could not
repair it either. Observed 2026-09-21 on 1×B200 (random-init Qwen3-MoE,
TP1, vLLM 0.24.0); any bf16 MoE RL rollout on Blackwell with the default
backend hits it.

**Why not vLLM's layerwise reload (the first revision of this PR).** As
@wuxibin89 pointed out, that lifecycle holds every incoming tensor of a
layer until the layer is complete (`online_process_loader` appends each
call to `info.loaded_weights` before counting it). In verl those tensors
are views into the receiver's reused IPC bucket, so they had to be
cloned, and what is held is the sender's full tensor: before TP
narrowing, and including experts that are not local to the rank (the
loader is called for every expert; the non-local calls return `False`
but have already been appended). With verl's export order (all experts'
gate/up before any `down_proj`, e.g. `unfuse_moe_params` on the FSDP
path) a MoE layer completes only at its last `down_proj`, so the extra
peak per GPU is about one full, un-sharded MoE layer on top of the
rebuilt layer, whatever the TP/EP:

| model | one MoE layer, bf16, all experts (G) | layerwise revision ≈ G
+ 2G/P | this revision ≈ G/P |
|---|---|---|---|
| Qwen3-30B-A3B, experts over P=2 GPUs | 1.21 GB | ≈ 2.4 GB | ≈ 0.6 GB |
| Qwen3-235B-A22B, P=8 | 4.83 GB | ≈ 6.0 GB | ≈ 0.6 GB |
| DeepSeek-V3-sized layer, P=16 | 22.5 GB | ≈ 25 GB | ≈ 1.4 GB |

Estimates from the code paths and the model configs. The layerwise
revision also held the clones of the current bucket (up to 2 GB).
Measured on the tiny model below: 460.1 MB vs 6.0 MB per sync.

**Fix.** The TRT-LLM bf16 block layout has the same byte count as the
checkpoint layout (`w13`: `[E, 2H/128, 2I, 64]` vs `[E, 2I, H]`, both
bf16), so no second copy is needed:

1. **Stage** (before the first bucket, `stage_unquantized_moe_params`).
For each `UnquantizedFusedMoEMethod` layer whose live `w13_weight` /
`w2_weight` no longer has the checkpoint shape, put a `Parameter` on the
layer that views the start of the live storage in checkpoint shape and
set the live parameter aside. The checkpoint shapes come from the
metadata vLLM records for every layer at `initialize_model` for its own
reloads (`reload.layerwise.LAYERWISE_INFO[layer].restore_metadata`), so
nothing is patched before the engine starts. Layers whose kernel keeps
the checkpoint shape (Triton; FlashInfer CUTLASS's same-shape gate/up
swap) are not staged and load as before.
2. **Stream** (per bucket, unchanged from `main`).
`model.load_weights(bucket)`: the expert loaders write each expert's
rows straight into the live storage through the view. Nothing is
buffered or cloned; the receiver reuses its IPC buffer as before.
3. **Fold** (after the last bucket, `fold_unquantized_moe_params` around
the existing `process_weights_after_loading`). vLLM's `_setup_kernel`
re-derives the kernel layout from the view into a temporary and calls
`replace_parameter(..., prefer_copy=True)`. With the view on the layer
that call's shape check fails, so for the duration of this call
`replace_parameter` in `unquantized_fused_moe_method` is patched to copy
the temporary into the set-aside live parameter and put that parameter
back. This happens inside the layer's own hook, so the temporary is
freed before the next layer is processed. CUDA-graph addresses and the
live `Parameter` objects are unchanged. A staged weight that nothing
re-derived raises instead of being left in checkpoint layout. The
set-aside parameters stay recorded on the layer until each one is folded
back, so a sync that fails anywhere in between leaves the next sync what
it needs to restore them.

Between steps 1 and 3 the live storage holds checkpoint-layout bytes, so
no forward pass may run in between. That is the assumption `main`'s sync
already makes (it loads into the live parameters bucket by bucket and
post-processes at the end) and the one the FP8 staging path relies on.
vLLM 0.29 pads TRT-LLM's intermediate dim to a multiple of 128 before
the block conversion when the per-rank size is not one (e.g.
Qwen3-235B-A22B or GLM-4.5 at TP8 without EP: 192 → 256). The padded
live storage still has room for the checkpoint layout, so it is staged
in place too. Only a kernel layout smaller than the checkpoint layout,
which no unquantized backend produces today, would get a fresh
NaN-filled staging buffer, logged with its size.

**Which backends this touches.** Staging keys on the layer's live
layout, not on the backend name. For the CUDA list in vLLM's
`_get_priority_backends`, per `convert_to_unquantized_kernel_format`
(the same in 0.24 and 0.29):

| backend | kernel prep of `w13` / `w2` | shape changes | this PR |
|---|---|---|---|
| `FLASHINFER_TRTLLM` (default on SM100) | 4-D block layout (0.29: after
padding the intermediate dim to a multiple of 128) | yes | staged, then
folded |
| `FLASHINFER_CUTLASS` | gated MoE: gate/up halves swapped
(`swap_w13_to_w31`); otherwise none | no | untouched |
| `TRITON` (default on SM90) | none (`.contiguous()`) | no | untouched |
| `BATCHED_TRITON` | none | no | untouched |

For the untouched backends `stage_unquantized_moe_params` returns
nothing and the fold is a no-op (nothing is patched), so the sync is
exactly `main`'s. The same-shape CUTLASS swap needs no staging on this
path: the full refit rewrites every row of every local expert in
checkpoint layout, and the post-load pass, which runs on every layer,
re-derives the swap and copies it in place (`prefer_copy`). The FP8 path
in verl-project#7986 had to detect same-shape repacks because it re-processes only
the layers it staged.

What this relies on in vLLM (the module-level `replace_parameter` import
in `unquantized_fused_moe_method`, `_setup_kernel`'s `prefer_copy` call,
`LAYERWISE_INFO` filled by `record_metadata_for_reloading` at
`initialize_model`, and the TRT-LLM conversion returning bf16) is the
same in vLLM 0.24.0, the version run on B200, and in 0.29.0, which
`main` pins (checked in source). On a vLLM without these pieces, staging
is skipped and the sync behaves as on `main`.

Everything else in the sync is as on `main`: buffer updates per bucket;
LoRA, QAT, ModelOpt and FP8 paths untouched. Folding reuses the FP8
refit helpers `_copy_param_subclass_attrs` and `_fold_into_live_param`
(verl-project#7986).

Workaround until this lands:
`+actor_rollout_ref.rollout.engine_kwargs.vllm.moe_backend=triton`
(verified on B200).

### Checklist Before Starting

- [x] Similar PRs (re-checked 2026-09-23): the only open PR mentioning
verl-project#7978 is this one; searches for "unquantized moe", "moe refit",
"flashinfer moe weight update" find no open PR on this path (verl-project#7971 is
NVFP4 live refit, a quantized path). Closed verl-project#7136 explored vLLM's
layerwise lifecycle for the *FP8* resync on vLLM 0.20. verl-project#6847 (closed) is
a different failure of the FlashInfer-MoE-plus-weight-update family
(stale kernel tensor addresses on 0.20.2, fixed upstream by
`prefer_copy`).
- [x] Title format `[rollout, vllm] fix: ...`

### Test

```bash
python -m pytest tests/utils/test_vllm_moe_refit_staging_on_cpu.py -q
# 10 passed
python -m pytest tests/utils/test_vllm_weight_name_normalization_on_cpu.py -q
# 21 passed, 6 failed: the 6 are the pytest.importorskip("vllm") tests (tied-alias and real-receiver bucket tests)
# that need the vllm CI venv; a clean main checkout gives the same 6 failures (19 passed)
pre-commit run --files <the 4 changed files>   # ruff, ruff-format, mypy: pass
```

`tests/utils/test_vllm_moe_refit_staging_on_cpu.py` runs against stub
vLLM modules, with a toy block-transpose kernel layout that has the
property that matters (a new shape over the same bytes, not convertible
in place):
- the block layout is staged as a view of the live storage, the loads
land in it, and the fold restores the same `Parameter` objects holding
the re-derived layout;
- each layer is folded before the next layer is processed (one temporary
at a time);
- layout-preserving backends (identity, same-shape swap), layers without
recorded metadata and layers with another quant method are not staged;
- a padded kernel layout (vLLM 0.29) is staged in the live storage; a
layout smaller than the checkpoint layout gets a fresh buffer (NaN until
written);
- a staged weight that nothing re-derived fails loudly;
- a layer left staged by a failed sync is restaged from its live
parameters;
- a fold that raises, or a staging failure part way through a layer,
keeps the set-aside parameters recorded, and the next sync restores and
refits the layer.

Each of these tests was checked to fail when the behaviour it covers is
removed (no fold, a fresh allocation instead of a view, the padded
storage not reused, the room check dropped, the fold deferred to the end
of the model, the re-derivation check dropped, the stale stash ignored,
the record dropped before the fold, the record attached only after
staging).

In `tests/utils/test_vllm_weight_name_normalization_on_cpu.py`: the
standard sync stages before the first bucket, hands `load_weights` the
receiver's bucket views (no copies) and runs
`process_weights_after_loading` inside the fold; a LoRA adapter sync
neither stages nor folds.

#### Hardware — 1×B200, this PR at `162b3db7` (2026-09-23)

Setup as in the earlier round (collapsed below): RunPod 1×B200
(US-NE-1), image `verlai/verl:vllm024.dev2` (vLLM 0.24.0, FlashInfer
0.6.12), random-init 2-layer / 8-expert Qwen3-MoE (hidden 512,
`moe_intermediate_size` 256, top-2), Megatron bf16 training + vLLM bf16
rollout, TP1, GRPO on gsm8k, FlashInfer autotune off. Pod-only debug
hooks (not in this PR) measured the sync memory and checked the refit
bytes; each arm's backend is vLLM's own `Using … Unquantized MoE
backend` line. All arms ran on the same pod.

| arm | code | MoE backend | result |
|---|---|---|---|
| T1, negative control | `main` `0ed63af8` | FlashInfer TRT-LLM (auto) |
crashes at the first sync: `ValueError: shard_dim=0 is not a valid data
dimension for a 3D tensor (expected 1 or 2)` |
| T2 | this PR, lr 1e-6, 4 steps | FlashInfer TRT-LLM (auto) | exit 0;
`Staged 2 unquantized MoE layers in checkpoint layout for the refit` on
each of the 5 syncs; no fresh staging buffer; `rollout_corr/kl` 0.00177
/ 0.00182 / 0.00181 / 0.00190 (Triton baseline 0.0019) |
| T4, exactness | this PR, lr 1e-3, 3 steps | FlashInfer TRT-LLM (auto)
| 4/4 syncs: the staged view shares the live storage; incoming expert-0
`gate`/`up`/`down` bytes equal the staged bytes; after the fold the live
`w13` / `w2` are the original `Parameter`s at the original address and
bit-equal to vLLM's own `convert_to_unquantized_kernel_format` of the
delivered weights |
| T5 | this PR, lr 1e-3, `moe_backend=triton` | Triton | not staged (no
staging line); 4/4 syncs: live `w13` / `w2` bit-equal to the conversion
(identity) of the delivered weights, same objects and addresses; kl
0.00192 / 0.00184 / 0.00183 |
| T6 | this PR, lr 1e-3, `moe_backend=flashinfer_cutlass` | FlashInfer
CUTLASS | not staged; 4/4 syncs: live bit-equal to the conversion
(gate/up swap) of the delivered weights, same objects and addresses; kl
0.00187 / 0.00179 / 0.00179 |
| A-mem | layerwise revision `1bfda7dc`, 2 steps | FlashInfer TRT-LLM
(auto) | exit 0, lifecycle 3/3; memory below |

Sync memory, per sync: peak `torch.cuda.max_memory_allocated` during
`update_weights_from_ipc` minus the allocation before it.

| revision | peak extra per sync |
|---|---|
| layerwise revision (`1bfda7dc`) | 460.1 MB (3/3 syncs) |
| this PR (`162b3db7`) | 6.0 MB (5/5 syncs) |

On this tiny model the layerwise number is mostly the clones of the
whole bucket (the 0.33 GB model fits in one 2 GB bucket, embedding and
`lm_head` included) plus the rebuilt layers. This PR's 6.0 MB is one MoE
layer's temporary (`w13` 4 MB + `w2` 2 MB), as expected. Real-size
numbers will come with the Qwen3-30B-A3B run.

T4 per sync (sha1 prefixes; lr 1e-3 so the weights move every step):

| sync | incoming expert-0 `gate_proj` | staged expert-0 gate | live
`w13` after the fold |
|---|---|---|---|
| step 0 | `1f07f6a9fe2e601f` | `1f07f6a9fe2e601f` | `a905365b9495ecec`
|
| step 1 | `93b57a89c31877fc` | `93b57a89c31877fc` | `4d2e0f3c969a1ec4`
|
| step 2 | `d646a30eb7b97f0a` | `d646a30eb7b97f0a` | `c186bb52902b7269`
|
| step 3 | `2b7b47ba68802d43` | `2b7b47ba68802d43` | `b08f175176a1184e`
|

Step 0's live `w13` is byte-identical to the layerwise revision's step 0
in the earlier round (`a905365b9495ecec`): same random-init weights,
same kernel layout.

**Still pending:** Qwen3-30B-A3B TP2/EP2 on 4×B200.

<details>
<summary>Earlier layerwise revision (<code>1bfda7dc</code>, superseded)
— 1×B200 results, 2026-09-22</summary>

**T1 — negative control: the sync path `main` has** (the layerwise
revision with `VERL_VLLM_LAYERWISE_RELOAD=0`), default backend. Crashes
at the first weight sync, exactly the failure in verl-project#7978:

```text
(Worker pid=146003) ERROR 09-22 17:00:55 [multiproc_executor.py:1000] ValueError: shard_dim=0 is not a valid data dimension for a 3D tensor (expected 1 or 2)
```

(`RoutedExperts.weight_loader → _load_w13 → _get_hidden_dim` on the 4-D
`w13_weight`.)

Setup: RunPod 1×B200 (SM100, 183 GB), image `verlai/verl:vllm024.dev2`
(vLLM **0.24.0**, FlashInfer 0.6.12, Megatron-Core 0.18, Megatron-Bridge
0.5.2 — `main` now pins vLLM 0.29; the reload API and the SM100 backend
selection are unchanged there, source-checked, but 0.29 itself was not
run). Model: random-init 2-layer / 8-expert Qwen3-MoE
(`Qwen3MoeForCausalLM`, hidden 512, `moe_intermediate_size` 256, top-k
2, per-expert HF layout), Megatron bf16 training + vLLM **bf16**
rollout, TP1, GRPO on gsm8k, FlashInfer autotune off,
`val_before_train=False`. The random model's outputs are near-uniform
(entropy ≈ ln V) and all gsm8k rewards are 0, so the runs check the
*mechanism* (crash / no crash, exact weight delivery, lifecycle
bookkeeping), not model quality. The bf16-Triton baseline numbers come
from the same setup on 2026-09-21.

**T2 — layerwise revision (default backend, lr 1e-6, 4 steps).**
`TRAIN_EXIT=0`, 5 syncs (step 0 + 4):

```text
INFO:2026-09-22 16:32:28,251:vLLM layerwise reload: parameters restored to checkpoint layout for this sync
WARNING 09-22 16:32:28 [layerwise.py:255] RotaryEmbedding: Failed to load weights
INFO:2026-09-22 16:32:28,965:vLLM layerwise reload finalized, deferred buffers applied: 0
```

`parameters restored` ×5 / `finalized` ×5; the only `Failed to load
weights` is `RotaryEmbedding` (buffer-only layer whose tensors the
stream never carries — vLLM puts its original tensors back, one warning
per sync); no "Allocating … buffers" warnings; restore→finalize span 0.7
s on the first sync, 0.1 s after. Same-setup bf16 baseline on the Triton
backend (old path) for comparison:

| | `rollout_corr/kl` steps 1–4 | `rollout_probs_diff_mean` |
|---|---|---|
| layerwise revision, FlashInfer TRT-LLM (default on SM100) | 0.00177 /
0.00182 / 0.00193 / 0.00184 | 4.7e-08 |
| old path, `moe_backend=triton` (2026-09-21) | 0.00190 / 0.00187 /
0.00185 / 0.00186 | 4.7e-08 |

**T4 — exact delivery and freshness (lr 1e-3 so the weights move every
step; 3 steps = 4 syncs).** A pod-only debug hook (not in this PR)
logged the sha1 of the raw bf16 bytes of
`layers.0.self_attn.o_proj.weight` as it arrived and of the live
parameters after finalize:

| sync | incoming `o_proj` | live `o_proj` after finalize | live
`w13_weight` (4-D kernel layout) |
|---|---|---|---|
| step 0 | `01a611e6f6b7a3e1` | `01a611e6f6b7a3e1` | `a905365b9495ecec`
|
| step 1 | `f53bac539b9ef503` | `f53bac539b9ef503` | `4b6718bfe919be02`
|
| step 2 | `8afd6ca3bc4a656d` | `8afd6ca3bc4a656d` | `2cdc3ffff9aecfee`
|
| step 3 | `6101d8425d0ff80d` | `6101d8425d0ff80d` | `4722bf20ef2da3b7`
|

Live equals incoming bit-for-bit on every sync (delivered into the
CUDA-graph storage, not a stale copy), the incoming hash changes every
step (the trainer's weights really move), and the 4-D `w13_weight`
changes every sync (the experts are re-derived into the kernel layout
each time). The same hook on the Triton old path shows the same
equalities (4/4), as expected.

Also observed: a fresh pod's first vLLM start with this backend took
1125 s (`torch.compile and initial profiling/warmup`), 2.6 s once the
FlashInfer JIT cache was warm — unrelated to this PR but worth knowing
when timing bf16 MoE rollouts on Blackwell.

</details>

### API and Usage Example

No API change and no new environment variable (the layerwise revision's
`VERL_VLLM_LAYERWISE_RELOAD` is gone).

### Design & Code Changes

- `verl/utils/vllm/vllm_moe_refit_utils.py` (new):
`stage_unquantized_moe_params(model)` and the
`fold_unquantized_moe_params(layers)` context manager described above.
- `verl/workers/rollout/vllm_rollout/utils.py` (+13/−2): the standard
branch of step 1 stages every synced model (main model and MTP drafter);
the standard branch of step 3 runs `process_weights_after_loading`
inside the fold. `_update_weights` is unchanged from `main`.
- Tests as above.

### Checklist Before Submitting

- [x] Read the Contribute Guide
- [x] pre-commit hooks on the changed files (ruff, ruff-format, mypy);
full `pre-commit run --all-files` not run locally (the config-generation
hook needs hydra)
- [x] Documentation: the change is internal to the sync path and
described in the module docstring.
- [x] Unit tests added (CPU)
- [ ] ci-request — 1×B200 validation done (see Hardware); Qwen3-30B-A3B
4×B200 run pending

AI assistance was used for the diagnosis (vLLM source reading) and for
drafting the code, tests and this description (Claude); the author
reviewed every changed line and ran the tests above.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: wuxibin <wuxibin@bytedance.com>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…and low-precision rollout (verl-project#8287)

### What does this PR do?

This PR refactors Megatron QAT and low-precision rollout around a
unified trainer-side weight quantization and rollout-side refit
boundary.

- Add compressed-tensors-compatible trainer-side exporters for
FP8_BLOCK, MXFP8, MXFP4, and NVFP4.
- Add ModelOpt-backed Megatron QAT modes: `fp8`, `mxfp8`, `mxfp4`,
`mxfp4_mxfp8`, and `nvfp4`.
- Replace the string `rollout.quantization=fp8` API with structured
`rollout.quantization.linear`, `moe`, `ignore`, and `config_file`
settings.
- Load and refit pre-quantized weights in vLLM and SGLang instead of
quantizing BF16 weights inside rollout workers.
- Add vLLM MXFP4/NVFP4 refit support, update FP8 refit handling, and
update SGLang routing replay support.
- Add a unified QAT guide, migration notes, examples, CPU tests, and a
rollout log-prob comparison tool.
- Remove the legacy `verl.utils.modelopt` implementation and TRT-LLM
rollout quantization support.

Related to verl-project#5972 and verl-project#4066.

This PR has partial scope overlap with open low-precision work, but uses
a materially different architecture: verl-project#7519 implements Transformer Engine
shared-grid MXFP8 training and rollout, while this PR implements
ModelOpt QAT plus a generalized trainer-side compressed-tensors export
boundary for multiple formats; verl-project#7971 implements native Transformer
Engine NVFP4 real quantization, while this PR implements NVFP4
fake-quant QAT and pre-quantized refit; verl-project#8205 fixes the legacy FSDP
NVFP4 compressor API, while this PR refactors the Megatron and rollout
path and does not add this feature to FSDP.

AI assistance was used to review the branch, fix two review findings,
add a regression test, and prepare this PR description. The human
submitter reviewed the changes and is responsible for the contribution
end to end.

### Checklist Before Starting

- [x] Search for similar PRs:
https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+QAT+compressed-tensors+rollout
- [x] Format the PR title as `[{modules}] {type}: {description}`.

### Test

Passed:

```bash
git diff --check origin/main...wuxibin/qat
python -m compileall -q verl/utils/qat verl/utils/vllm verl/workers/rollout/sglang_rollout verl/workers/rollout/trtllm_rollout
PATH="$PWD/.venv/bin:$PATH" pre-commit run --all-files --show-diff-on-failure --color=always
```

The full pre-commit suite passed, including Ruff, Ruff format, mypy,
generated-config verification, repository sanity checks, and compileall.

The targeted pytest suites and GPU end-to-end tests were not run in this
local environment because PyTorch, ModelOpt, and compressed-tensors were
unavailable. CI/GPU validation should cover Megatron with vLLM and
SGLang using FP8_BLOCK, plus MXFP8/MXFP4/NVFP4 on supported Blackwell
hardware.

### API and Usage Example

The previous API:

```bash
actor_rollout_ref.rollout.quantization=fp8
```

is replaced by:

```bash
actor_rollout_ref.rollout.quantization.linear=FP8_BLOCK \
actor_rollout_ref.rollout.quantization.moe=FP8_BLOCK
```

Enable matching Megatron QAT with:

```bash
actor_rollout_ref.actor.megatron.qat.enable=true \
actor_rollout_ref.actor.megatron.qat.mode=fp8
```

Dense and routed-expert precision can be configured independently. For
example, use BF16 dense layers and MXFP4 experts with:

```bash
actor_rollout_ref.rollout.quantization.linear=null \
actor_rollout_ref.rollout.quantization.moe=MXFP4 \
actor_rollout_ref.actor.megatron.qat.enable=true \
actor_rollout_ref.actor.megatron.qat.mode=mxfp4
```

TRT-LLM no longer accepts `rollout.quantization`; use vLLM or SGLang for
supported low-precision rollout configurations.

### Design & Code Changes

- Introduce `WeightQuantizer` dispatch based on rollout overrides or the
model's native Hugging Face quantization config.
- Export canonical HF-named tensors from Megatron, quantize them on the
trainer side, and transfer the final checkpoint layout through the
checkpoint engine.
- Build compressed-tensors configs from per-layer presets and enforce
QAT/rollout precision compatibility at startup.
- Add vLLM staging and post-load processing for FP8, MXFP8, MXFP4, and
NVFP4 refits.
- Pass compressed-tensors model overrides to SGLang for FP8_BLOCK
rollout.
- Keep default high-precision layers unquantized for both generated
compressed-tensors configs and native FP8 configs.
- Consolidate low-precision documentation and provide
`scripts/rollout_logprob_diff.py` for numerical comparison.
- Remove superseded ModelOpt/vLLM monkey-patch utilities and the TRT-LLM
FP8 E2E job.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [ ] Add unit or end-to-end tests to CI to cover all code. New CPU
tests and updated FP8 E2E jobs are included, but
ModelOpt/compressed-tensors test dependency coverage and MXFP4/NVFP4 E2E
coverage still need confirmation.
- [ ] Once this PR is ready for CI, send a message in the [`ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1).
- [x] Recipe submodule update is not applicable; this PR does not change
the recipe submodule.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…ect#8306)

### What does this PR do?

Follow up verl-project#8287, add SGLang
online weight refit for MXFP8, routed-expert MXFP4, and routed-expert
NVFP4 rollouts. The trainer exports pre-quantized checkpoint tensors,
while a bucket-aware SGLang custom loader stages canonical checkpoint
layouts and rebuilds backend-specific kernel layouts after the final
sync bucket without replacing CUDA-graph-captured storage.

The implementation:

- translates an all-MXFP8 compressed-tensors rollout configuration to
SGLang's native `quant_method: mxfp8` schema and exports
`weight_scale_inv`;
- supports BF16 dense layers with MXFP4×MXFP8 or NVFP4 W4A4 routed
experts;
- stages MXFP8 scales and MXFP4/NVFP4 expert tensors across all
weight-sync buckets, then reprocesses them once after the final bucket;
- restores live storage if loading or post-processing fails;
- pins validated FP8/MXFP8/MXFP4 SGLang paths to DeepGEMM while leaving
NVFP4 backend selection to SGLang;
- rejects the unsupported SGLang combination `linear=MXFP8, moe=MXFP4`;
- rejects `delta_flush` for custom quantized refit because finalization
requires the complete bucket sequence;
- documents configuration, backend constraints, support status, and
measured trainer–rollout mismatch.

This PR overlaps with
[verl-project#8130](verl-project#8130) on SGLang MXFP8
export and refit. The additional scope here is a unified refit lifecycle
shared by MXFP8, MXFP4, and NVFP4, including FP4 checkpoint-layout
staging, backend-specific in-place reconstruction, and Blackwell
validation for all three formats. It also overlaps conceptually with
[verl-project#7971](verl-project#7971), but that PR
implements native NVFP4 training and vLLM rollout; this PR adds the
SGLang NVFP4 refit path.

AI assistance was used for implementation review, debugging, and
preparation of this PR description. The human submitter must review
every changed line and confirm the validation results before submission.

### Checklist Before Starting

- [x] Search for similar PRs: [SGLang
QAT](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+sglang+qat),
[SGLang
MXFP4](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+sglang+mxfp4),
and [SGLang weight
refit](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+sglang+weight+refit).
- [x] Format the PR title as `[{modules}] {type}: {description}`.

### Test

The mismatch measurements below use Qwen3.5-35B-A3B with Megatron EP=8,
SGLang rollout TP=2, Blackwell, and the default CUDA Graph
configuration:

| Rollout | Backend | `k3_kl` without QAT | `k3_kl` with matching QAT |
|---|---|---:|---:|
| MXFP8 linear + MoE | DeepGEMM linear + MoE | 0.00324 | 0.00327 |
| BF16 linear + MXFP4 MoE | DeepGEMM MoE | 0.00571 | 0.00200 |
| BF16 linear + NVFP4 MoE | FlashInfer CUTLASS MoE | 0.00946 | 0.00653 |

Local checks performed while preparing the branch:

```bash
python -m compileall -q verl/utils/sglang/sglang_quant_utils.py
git diff --check
```

The SGLang CPU test suite could not be completed in the current
environment because importing `torchcodec` failed to resolve
`libc10_cuda.so`. The final branch currently does not add committed unit
tests; reviewers should treat the Blackwell measurements above as the
available validation and require focused CPU coverage if needed.

### API and Usage Example

All-MXFP8 rollout:

```yaml
actor_rollout_ref:
  rollout:
    name: sglang
    quantization:
      linear: MXFP8
      moe: MXFP8
      ignore:
        - "re:.*linear_attn\\.in_proj_[ab]$"
  actor:
    megatron:
      qat:
        enable: true
        mode: mxfp8
        ignore_patterns:
          - "*linear_attn.in_proj_a"
          - "*linear_attn.in_proj_b"
```

MXFP4 routed experts with BF16 dense layers:

```yaml
actor_rollout_ref:
  rollout:
    name: sglang
    quantization:
      linear: null
      moe: MXFP4
  actor:
    megatron:
      qat:
        enable: true
        mode: mxfp4
```

NVFP4 routed experts with BF16 dense layers:

```yaml
actor_rollout_ref:
  rollout:
    name: sglang
    quantization:
      linear: null
      moe: NVFP4
  actor:
    megatron:
      qat:
        enable: true
        mode: nvfp4
```

SGLang intentionally rejects `linear: MXFP8` together with `moe: MXFP4`.
Use BF16 dense layers for SGLang MXFP4 rollout, or use vLLM for the
mixed MXFP8-linear + MXFP4-MoE configuration.

### Design & Code Changes

- `verl/utils/sglang/sglang_quant_utils.py`
  - resolves native and override quantization configurations;
- translates all-MXFP8 compressed-tensors configurations to native
SGLang MXFP8;
  - selects the required refit kind and registers one custom loader;
- stages each bucket and sends a final marker to trigger post-load
reconstruction;
  - fails fast for unsupported mixed MXFP8-linear + MXFP4-MoE rollout.
- `verl/utils/sglang/sglang_fp8_utils.py`
  - rebuilds dense MXFP8 derived scale layouts;
- stages canonical MoE scales and folds rebuilt layouts into captured
storage.
- `verl/utils/sglang/sglang_fp4_utils.py`
  - stages MXFP4 and NVFP4 expert checkpoint tensors;
  - publishes packed-weight aliases expected by SGLang loaders;
  - rebuilds backend layouts and preserves CUDA Graph storage.
- `verl/utils/qat/compressed_tensors.py`
  - detects all-MXFP8 compressed-tensors configurations;
- selects an SGLang-specific MXFP8 exporter that emits native scale
names.
- `verl/workers/rollout/sglang_rollout/async_sglang_server.py`
  - prepares the effective SGLang quantization configuration;
  - registers the refit loader and validated GEMM backends.
- `verl/workers/rollout/sglang_rollout/sglang_rollout.py`
  - uses the custom loader for every bucket;
  - sends finalization after the last bucket;
  - rejects the incompatible `delta_flush` wire format.
- `docs/low_precision/qat.md`
- documents the support matrix, configuration, engine constraints,
mismatch measurements, and limitations.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [ ] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting):
`pre-commit install && pre-commit run --all-files --show-diff-on-failure
--color=always`.
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to the CI workflow. Focused tests
were developed during the branch history but are not present in the
final diff; the current evidence is the Blackwell mismatch validation
above.
- [ ] Once the PR is ready for CI, send a message in the `ci-request`
channel.
- [x] This PR is not related to the `recipe` submodule.

---------

Co-authored-by: root <root@maas-gpu-gb200-devbox-1.byted.org>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…roject#8127)

### What does this PR do?

Adds the **MXFP8 training recipe** (`fp8_recipe: mxfp8`, Blackwell) to
verl's Megatron path, an end-to-end example and its documentation,
`docs/low_precision/mxfp8.md`. Together with what main already has,
verl-project#8287 (the trainer quantizes the rollout weights with
compressed-tensors) and verl-project#8306 (SGLang MXFP8 refit), this gives an MXFP8
RL recipe: MXFP8 GEMMs in training through Transformer Engine (TE), and
an MXFP8 rollout on vLLM or SGLang.

- **Training.** `fp8_recipe: mxfp8` already flows through
`override_transformer_config`. The one functional gap was THD sequence
padding:
- TE's MXFP8 quantizer needs both GEMM dimensions divisible by 32, and
the wgrad scale blocks run along the token dimension.
- So packed sequences are padded to `lcm(32, align)` per sequence under
MXFP8, using Megatron-Core's own `get_fp8_align_size` (the mapping it
applies to its MoE FP8 padding).
  - Every other recipe keeps its previous alignment (`lcm(16, align)`).
- **QAT combined with a TE FP8 recipe is rejected** at engine init: QAT
fake-quantizes weights inside BF16 GEMMs, while an FP8 recipe switches
the GEMMs themselves to FP8.
- **Examples**, each with `PRECISION=mxfp8` (MXFP8 end to end) or
`PRECISION=bf16` (the same job as the BF16 baseline):
- `examples/mxfp8/run_qwen3_8b_megatron.sh`: GRPO on Qwen3-8B with
GSM8K; the defaults fit one B200.
- `examples/mxfp8/run_qwen3_30b_a3b_megatron.sh`: GRPO on Qwen3-30B-A3B
(MoE) with DAPO-Math-17k on one 4×B200 node, experts included.
- **Docs.** `docs/low_precision/mxfp8.md` (new, in the Low Precision
toctree): the configuration, which kernels the vLLM and SGLang rollouts
use, the examples with their results, and earlier runs on DAPO-Math-17k.

Nothing changes unless an FP8 recipe is configured.

This was part 1 of verl-project#7519's split; as the maintainers asked, it now
stands alone (the consolidated verl-project#8314 was closed in favor of it). Making
the THD packing helpers recipe-agnostic, as suggested in the review,
follows in a separate PR.

### Checklist Before Starting

- [x] Search for similar PRs:
https://github.com/verl-project/verl/pulls?q=is%3Apr+mxfp8. No other
open PR adds MXFP8 training on the Megatron path. verl-project#7971 (NVFP4 training,
open) changes the same THD padding code for FP4; whichever lands second
rebases.
- [x] PR title formatted as `[megatron] feat: ...`

### Test

```bash
python -m pytest tests/utils/test_megatron_fp8_padding_on_cpu.py tests/utils/test_mxfp8_ignored_layers_on_cpu.py -q
# 11 passed (with compressed-tensors 0.17.0; without it 2 are skipped)
# existing tests of the paths this PR touches:
python -m pytest tests/utils/test_megatron_bshd_preprocess.py tests/utils/test_config_router_replay_on_cpu.py \
  tests/utils/test_modelopt_mxfp8_on_cpu.py tests/workers/engine/megatron/test_router_replay_utils_on_cpu.py \
  tests/utils/megatron/test_router_replay_model_walk_on_cpu.py -q
# 94 passed, 1 skipped, 1 failed
python -m pytest tests/models/test_model_forward_fused.py -q
# 6 passed, 3 failed
```

- **The failure in the second set**,
`test_modelopt_mxfp8_on_cpu.py::test_nvfp4_qat_uses_experts_only_export_compatible_backends`,
fails the same way on main (954bd28) in the same environment (ModelOpt
0.46.0).
- **The three failures in `test_model_forward_fused.py`** check the
installed Megatron-Core's forward signature and fail the same way on
main in this environment.
`test_engine_hook_context_and_current_thd_arguments` passes; it now
expects the `fp8_recipe` THD argument.
- **Collection order:** `test_modelopt_mxfp8_on_cpu.py` has to be
collected before `test_router_replay_model_walk_on_cpu.py`, which
installs a `MagicMock` in `sys.modules` for Megatron modules at import.

`pre-commit run` passes on all changed files.

### End-to-end results on B200

**The example itself**, one B200, verl's locked environment (vLLM
0.29.0, TE 2.16.1, Megatron-Core 0.19.2, Megatron-Bridge 0.6.2,
compressed-tensors 0.17.0), `TOTAL_TRAINING_STEPS=10`, the GSM8K test
set (1,319 problems, greedy) at steps 0, 5 and 10:

| | GSM8K test accuracy (step 0 / 5 / 10) | training reward (step 1 →
10) | `rollout_corr/kl` |
|---|---|---|---|
| BF16 | 0.329 / 0.610 / 0.819 | 0.24 → 0.82 | 0.0006–0.0008 |
| MXFP8 | 0.372 / 0.616 / 0.803 | 0.25 → 0.80 | 0.0063–0.0071 |

![Qwen3-8B on GSM8K, BF16 vs
MXFP8](https://github.com/user-attachments/assets/58bc248c-91cf-4f3f-bff5-64f3052230d1)
*Blue: BF16, vermillion: MXFP8 end to end.*

wandb:
[BF16](https://forge.coreweave.com/wandb/wengeezhang-bytedance/verl_grpo_gsm8k_mxfp8/runs/zodbg1pr),
[MXFP8](https://forge.coreweave.com/wandb/wengeezhang-bytedance/verl_grpo_gsm8k_mxfp8/runs/cr5dk8fs).

- No non-finite metric in either run.
- Both are single runs of 10 steps, too few to resolve the 1.6-point gap
at step 10 (21 of 1,319 problems).
- FlashInfer was available to vLLM's workers, so vLLM picked FlashInfer
CuTe-DSL for the MXFP8 dense GEMMs. The example run does not log the
choice (verl starts the workers at `VLLM_LOGGING_LEVEL=WARN`); its
step-0 accuracy equals, to every digit, that of the CuTe-DSL run below,
which logged it.

**The same hyper-parameters on each rollout kernel**, launched through
our own script with `VLLM_LOGGING_LEVEL=INFO` (validation at steps 0 and
10):

| rollout | dense GEMM kernel | GSM8K test accuracy (step 0 → 10) |
`rollout_corr/kl` |
|---|---|---|---|
| vLLM 0.29.0, FlashInfer available | FlashInfer CuTe-DSL (MXFP8 weights
and activations) | 0.372 → 0.806 | 0.0064–0.0073 |
| vLLM 0.29.0, FlashInfer unavailable | Marlin (MXFP8 weights, BF16
activations) | 0.362 → 0.805 | 0.0041–0.0050 |
| SGLang 0.5.20 | DeepGEMM, pinned by verl (MXFP8 weights and
activations) | 0.356 → 0.830 | 0.0065–0.0071 |

No non-finite metric in any run. Qwen3-8B is dense, so these runs
exercise no MoE kernel.

**Earlier runs on DAPO-Math-17k** (settings D and B of verl-project#7519, before its
rebase on verl-project#8287: vLLM 0.24 served MXFP8 through ModelOpt, and FP8
blockwise through its own FP8 method; the training side is this PR's).
wandb:
[verl_mxfp8_b200](https://forge.coreweave.com/wandb/wengeezhang-bytedance/verl_mxfp8_b200).

Qwen3-30B-A3B, the 30B example's hyper-parameters, one 4×B200 machine
for every arm, 10 steps (mean of steps 1–10; "X/Y" is training precision
X, rollout precision Y):

| | BF16/BF16 | FP8-TW/BF16 | FP8-BW/FP8-BW | MXFP8/MXFP8 |
|---|---|---|---|---|
| step time (s) | 506 | 487 | 489 | 452 |
| generation (s) | 134 | 133 | 115 | 102 |
| `update_actor` (s) | 244 | 225 | 241 | 221 |
| `rollout_corr/kl` | 0.0018 | 0.0058 | 0.0090 | 0.0045 |

![Qwen3-30B-A3B on DAPO-Math-17k, per
step](https://github.com/user-attachments/assets/569f6f0f-1b78-4e62-9401-6c02c5cad49e)

- MXFP8 cuts the step time by 11% (generation −24%, `update_actor` −9%),
with the lowest mismatch of the three low-precision arms. Ten steps do
not show convergence.
- The MXFP8 arm, rerun for 5 steps on the compressed-tensors path (verl-project#7519
rebuilt on verl-project#8287, vLLM 0.29, another 4×B200 machine), read
`rollout_corr/kl` 0.0037–0.0040. The 30B example itself has not run on
this branch.

Qwen3-8B, 2×B200, 32 steps, every arm with an FP8 KV cache (which raises
every arm's KL) and the MXFP8 rollout's `gate_up_proj` in BF16 (mean of
steps 23–32):

| | BF16/BF16 | FP8-TW/BF16 | MXFP8/MXFP8 |
|---|---|---|---|
| `rollout_corr/kl` | 0.0088 | 0.0297 | 0.0088 |
| entropy | 0.220 | 0.083 | 0.338 |
| step time (s) | 407 | 345 | 386 |

![Qwen3-8B on DAPO-Math-17k, per
step](https://github.com/user-attachments/assets/444018cd-2219-49fe-a907-8b147ef9b804)

- MXFP8's mismatch stays at the BF16 level, while FP8-TW with a BF16
rollout drifts from 0.009 to 0.037 and its entropy nearly collapses
(0.307 at step 1, 0.062 at step 32).

Also from verl-project#7519 (GSM8K, 60 steps): held-out accuracy 0.9393 with MXFP8
vs. 0.9363 with BF16, and the `update_actor` cost per token cut by 43%.

### API and Usage Example

No API change.

```yaml
# Blackwell (SM100+), Megatron-Core >= 0.13, TE >= 2.1, Megatron-Bridge model path (default)
actor_rollout_ref.actor.megatron.override_transformer_config:
  fp8: "e4m3"            # or "hybrid"
  fp8_recipe: "mxfp8"
# MXFP8 rollout on vLLM or SGLang
actor_rollout_ref.rollout.quantization:
  linear: MXFP8
  moe: MXFP8
```

```bash
PRECISION=mxfp8 bash examples/mxfp8/run_qwen3_8b_megatron.sh       # Qwen3-8B, GSM8K, one B200
PRECISION=mxfp8 bash examples/mxfp8/run_qwen3_30b_a3b_megatron.sh  # Qwen3-30B-A3B, DAPO-Math-17k, 4×B200
# PRECISION=bf16 for the baselines
```

### Design & Code Changes

- `verl/models/mcore/util.py`:
- `_compute_fp8_thd_align_size(align, fp8_recipe)` uses
`get_fp8_align_size`;
- `get_fp8_padding_options` replaces the padding decision repeated at
each call site.
- `model_forward.py`, `model_forward_fused.py`,
`verl/utils/megatron/router_replay_utils.py`: pass `fp8_recipe` to every
THD preprocess call.
- `verl/workers/engine/megatron/transformer_impl.py`: QAT + TE FP8
rejected.
- `examples/mxfp8/run_qwen3_8b_megatron.sh`,
`run_qwen3_30b_a3b_megatron.sh` (new): the examples above.
- `docs/low_precision/mxfp8.md` (new), listed in `docs/index.rst`.
- Tests:
- `test_megatron_fp8_padding_on_cpu.py`: alignment math and both THD
preprocess paths;
  - `test_mxfp8_ignored_layers_on_cpu.py`:
- the exporter's default ignore list keeps `lm_head`, the embedding
(also under a VL model's `language_model` prefix) and the routers in
BF16, but not `gate_up_proj`;
- an MXFP8 export with a user ignore list leaves `lm_head` and the
embedding in BF16;
- `test_model_forward_fused.py`: the expected THD arguments include
`fp8_recipe`.

### AI assistance disclosure

Developed with AI assistance. I reviewed every changed line, ran the
tests and the B200 runs above, and can defend the change end to end.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs):
`docs/low_precision/mxfp8.md`.
- [x] Add unit test(s): CPU tests above; B200 runs above.
- [ ] Once your PR is ready for CI, send a message in the `ci-request`
channel.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…of top-k distillation tensors (verl-project#8325)

### What does this PR do?

Follow-up to verl-project#8127, for two of its review threads.

1. **Recipe-agnostic THD padding**
([review](verl-project#8127 (comment))).
`preprocess_packed_seqs` and `preprocess_thd_engine` took two
FP8-specific arguments, `use_fp8_padding` and `fp8_recipe`. They now
take one integer, `quant_align_size`: the token alignment each packed
sequence needs for the model's low-precision GEMMs (`None`: no
low-precision padding). `get_thd_quant_align_size(config)` derives it
from the `TransformerConfig`, for FP8 recipes through Megatron-Core's
`get_fp8_align_size` (32 for MXFP8, 16 otherwise), so a new format such
as NVFP4 or MXFP4 (verl-project#7971) adds a branch there instead of another
parameter. The model forward, the fused forward and router replay all
call it. The padding itself is unchanged.
2. **Top-k distillation under FP8**
([review](verl-project#8127 (comment))).
`compute_forward_kl_topk` packs the teacher's top-k tensors itself and
asserts they match the student logits, but it packed them without the
student's FP8 padding, so Megatron top-k distillation failed that
assertion under any FP8 recipe; the gap predates MXFP8. The engine now
passes `quant_align_size` to the Megatron loss, which packs the teacher
tensors with it on THD and with `use_fp8_padding` derived from it on
BSHD.

Left out: on BSHD, `pad_bshd_to_minibatch_max` (on by default) pads the
student micro-batch to the mini-batch max (`forced_max_seqlen`) while
the teacher tensors are not padded to match. That fails the same
assertion even in BF16, and verl-project#8061 fixes it. The two PRs touch
neighbouring lines in `compute_forward_kl_topk`, so whichever lands
second needs a small rebase.

### Checklist Before Starting

- [x] Search for similar PRs:
[quant_align_size](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+quant_align_size),
[distillation
padding](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+distillation+padding),
[pad_bshd_to_minibatch_max](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+pad_bshd_to_minibatch_max).
- Not a duplicate: verl-project#8061 fixes only the BSHD `forced_max_seqlen`
mismatch, which this PR leaves to it; verl-project#7887 handles metrics of
empty-response micro-batches; verl-project#7971 (native NVFP4 training) adds its own
padding flag to the same helpers and could add an NVFP4 branch to
`get_thd_quant_align_size` instead.
- [x] Format the PR title as `[{modules}] {type}: {description}`.
`[BREAKING]` because two public helpers change keyword arguments (see
below).

### Test

CPU unit tests and pre-commit:

```bash
pytest -q tests/utils/test_megatron_fp8_padding_on_cpu.py tests/workers/test_megatron_distillation_only_on_cpu.py \
  tests/utils/test_megatron_bshd_preprocess.py tests/utils/test_config_router_replay_on_cpu.py \
  tests/workers/engine/megatron/test_router_replay_utils_on_cpu.py tests/utils/megatron/test_router_replay_model_walk_on_cpu.py
# 90 passed, 1 skipped

pytest -q tests/models/test_model_forward_fused.py
# 6 passed, 3 failed; the same 3 fail on main in the same environment

pre-commit run --files <changed files>
# passed
```

New tests cover the alignment helper, both THD preprocess paths with and
without the padding, the teacher's top-k packing on THD (MXFP8 and a
16-token FP8 recipe) and BSHD, and that the engine and
`compute_topk_loss` pass `quant_align_size` to the Megatron loss only.
On hosts without Megatron, the tests stub Megatron-Core's
`get_fp8_align_size` with the same 32/16 rule.

End to end on 2×B200:

- A tiny random Qwen3-MoE (2 layers, 8 experts, top-2) on GSM8K, 5 steps
per run, the trainer on one GPU (TP, EP and CP 1).
- MXFP8 training with Transformer Engine (`fp8_recipe=mxfp8`) and an
MXFP8 vLLM rollout, except the last row.
- The distillation runs put the teacher (the same model, BF16, served by
vLLM) on the second GPU, with `forward_kl_topk` and top-16.
- vLLM 0.29.0, Transformer Engine 2.16.1, Megatron-Core 0.19.2.
- "This PR" is this PR's files, byte for byte, on verl-project#8127's head before it
merged. That predates verl-project#8308, so the vLLM rollout used FlashInfer
CuTe-DSL for the dense layers and FlashInfer TRT-LLM for the experts.
"Before" is verl-project#8127's head without this PR.

| run | code | result |
|---|---|---|
| GRPO, THD | this PR | passes: 5/5 steps, no non-finite metric,
`rollout_corr/kl` 0.0019–0.0023 |
| GRPO + router replay R2, THD | this PR | passes: `rollout_corr/kl`
0.0022–0.0027 |
| top-k distillation, THD | this PR | passes: `rollout_corr/kl`
0.0018–0.0026, distillation metrics logged |
| top-k distillation, THD | before | fails the shape assertion in
`compute_forward_kl_topk` |
| GRPO, THD | before | passes: `rollout_corr/kl` 0.0020–0.0030 (control)
|
| top-k distillation, BSHD | this PR | fails the same assertion: the
`forced_max_seqlen` mismatch that verl-project#8061 fixes |
| top-k distillation, BSHD | this PR + the micro-batch's
`forced_max_seqlen` passed to the teacher packing, as in verl-project#8061 | passes:
`rollout_corr/kl` 0.0018–0.0027 |
| top-k distillation, BSHD, BF16 | before | fails the same assertion:
that mismatch does not depend on FP8 |

`rollout_corr/kl` is the KL between vLLM's and the trainer's log-probs.
The fused forward, FP8 recipes other than MXFP8, and TP or CP above 1
were not run on hardware; the unit tests above cover them.

### API and Usage Example

No config or CLI change. `preprocess_packed_seqs` and
`preprocess_thd_engine` take `quant_align_size` instead of
`use_fp8_padding` and `fp8_recipe`. `preprocess_bshd_engine` keeps
`use_fp8_padding`, which verl-omni passes. Within verl-project, nothing
else calls the THD helpers with the removed arguments.

```python
from verl.models.mcore.util import get_thd_quant_align_size, preprocess_thd_engine

quant_align_size = get_thd_quant_align_size(tf_config)  # 32 for MXFP8, 16 for other FP8 recipes, None without FP8
packed, *_ = preprocess_thd_engine(input_ids, pre_process=True, quant_align_size=quant_align_size)
```

### Design & Code Changes

- `verl/models/mcore/util.py`: `get_thd_quant_align_size(config)`; the
THD helpers take `quant_align_size`. Each sequence is padded to a
multiple of `lcm(quant_align_size, align_size)` and the total to
`align_size * 128`, as before.
- `verl/models/mcore/model_forward.py`, `model_forward_fused.py`: derive
`quant_align_size` from the helper; BSHD uses `use_fp8_padding =
quant_align_size is not None`.
- `verl/utils/megatron/router_replay_utils.py`: router replay records
and replays with the helper instead of inlining the rule.
- `verl/workers/engine/megatron/transformer_impl.py`:
`_lm_head_logits_processor` passes `quant_align_size` to the top-k loss.
- `verl/trainer/distillation/losses.py`: `compute_topk_loss` and
`distillation_ppo_loss` forward it to the Megatron loss only.
- `verl/trainer/distillation/megatron/losses.py`:
`compute_forward_kl_topk` packs the teacher's top-k tensors with it.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs). No
user-facing change to document.
- [x] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/verl-project/verl/tree/main/.github/workflows)
to cover all the code: the new tests are CPU unit tests (`*_on_cpu.py`)
that CI runs.
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1).
- [x] Not related to the `recipe` submodule.

AI assistance was used to write the code, the tests and this
description. The submitter reviewed the changes and ran the tests above.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…r QAT (verl-project#8361)

### What does this PR do?

Fixes two ways Megatron QAT breaks because of the layer spec the engine
builds the model from, and corrects the Hopper note in the QAT guide.
Both came out of the MXFP4 recipe checks reported in verl-project#8186.

**1. Models whose Megatron-Bridge bridge sets its own layer spec fail
under QAT.** With QAT enabled, the engine replaced every provider's
`transformer_layer_spec` with `modelopt_transformer_layer_spec`.
Megatron-Bridge maps the HF weights onto the modules of the spec the
bridge chose, and more than twenty bridges set their own (OLMoE,
Qwen3-Next, text-only Qwen3.5, DeepSeek-V2/V3/V4, GLM, Kimi, MiniMax,
...). Under the ModelOpt spec those models lose modules. OLMoE fails at
load:

```
ValueError: Shape mismatch for megatron param decoder.layers.0.self_attention.k_layernorm.weight
```

By the same code path, Qwen3-Next and text-only Qwen3.5 would lose their
linear-attention layers (from reading the bridges; not run). Qwen3.5
checkpoints (`Qwen3_5MoeForConditionalGeneration`) only escape because
the Qwen3.5 VL provider builds its language block without reading
`transformer_layer_spec`, so their QAT already runs on the TE spec.

The engine now switches to the ModelOpt spec only when the provider uses
Bridge's default GPT spec (`default_layer_spec`). Any other spec is
kept, and ModelOpt quantizes its Transformer Engine modules in place
(`TELayerNormColumnParallelLinear`, `TERowParallelLinear`, the grouped
expert linears, ...). Models on the default spec are unchanged.

**2. Packed THD sequences fail at the first forward when context
parallelism is 1** (`use_remove_padding=True`, the default):

```
AssertionError: Attention mask type must be padding or padding_causal for qkv_format=thd!
```

When `use_arbitrary_attention_mask` is unset,
`modelopt_transformer_layer_spec` turns it on for CP=1; the arbitrary
mask is meant for speculative-decoding training. Transformer Engine then
rejects verl's THD inputs. This PR defaults the flag to `False` under
the ModelOpt spec, which keeps the causal mask that the non-QAT TE spec
uses. An explicit
`override_transformer_config.use_arbitrary_attention_mask` still wins.

**3. Docs.** `docs/low_precision/qat.md`:
- The `MXFP4` preset has no Marlin fallback: its MXFP4-weight ×
FP8-activation experts are served only by vLLM's DeepGEMM FP4 MoE kernel
(SM100/SM120), so on Hopper the rollout fails at engine start.
- New limitation: the ModelOpt spec's expert module (`SequentialMLP`)
rejects tensor and expert parallelism together, so default-spec models
need TP=1 or EP=1 under QAT. Models with their own spec keep it.

### Checklist Before Starting

- [x] Search for similar PRs: no open PR or issue mentions
`use_arbitrary_attention_mask`, `modelopt_transformer_layer_spec`, the
THD assertion, or QAT with a model-specific layer spec (searched
`use_arbitrary_attention_mask`, `qat thd`,
`modelopt_transformer_layer_spec`, `QAT layer spec`, `qat
transformer_layer_spec`, `QAT olmoe`, `QAT custom spec`, and the error
text). verl-project#7971 changes the same file but not the QAT layer spec.
- [x] Format the PR title as `[{modules}] {type}: {description}`.

### Test

```bash
python -m pytest -q tests/workers/test_megatron_qat_layer_spec_on_cpu.py tests/workers/test_megatron_*_on_cpu.py \
  tests/workers/test_engine_*_on_cpu.py tests/workers/test_router_replay_engine_helpers_on_cpu.py \
  tests/utils/test_mxfp8_ignored_layers_on_cpu.py
# 80 passed
bash scripts/generate_trainer_config.sh
# All good
pre-commit run --files <changed files>
# ruff, ruff format, mypy, docs time, docstring, license and structure checks pass
```

The CPU test covers the default-spec switch, an explicit mask override,
and keeping a bridge-specific spec.

Spec chosen by `_use_qat_layer_spec` on real Megatron-Bridge 0.6.2
providers (CPU, built from the HF configs as the engine does, no weights
loaded):

| Model | Bridge spec | Spec under QAT |
|---|---|---|
| OLMoE-1B-7B-0125-Instruct | `olmoe_layer_spec` | kept (was replaced) |
| Qwen3-MoE | `default_layer_spec` | `modelopt_transformer_layer_spec`
(unchanged) |
| Qwen3.5-35B-A3B | experimental attention variant | kept (the VL
provider ignores the field anyway) |
| Qwen3.5 MoE, text-only config | experimental attention variant | kept
(was replaced) |

GPU, 1× GH200 with the stock wheelhouse TE,
`scripts/rollout_logprob_diff.py --n-gpus 1 --tp 1 --gen-tp 1 --ep 1
--quant-linear FP8_BLOCK --quant-moe FP8_BLOCK`,
OLMoE-1B-7B-0125-Instruct:

| Code | QAT | Result | k3 KL |
|---|---|---|---|
| THD fix only (first commit) | `--qat-mode fp8` | fails at load with
the `k_layernorm` shape mismatch above | — |
| this PR | `--qat-mode fp8` | runs; 128 enabled quantizers, all on TE
modules (layernorm-column, row, grouped fc1 and fc2 linears) | 6.3e-3 |
| this PR | off | runs | 9.1e-3 |

- The rollout starts from dummy weights and receives the trainer's
export through `update_weights()`. The QAT run therefore also exercises
exporting the quantized TE modules, grouped experts included, through
the OLMoE bridge.
- The quantizer count came from a temporary debug print that is not part
of this PR.
- OLMoE is the only model with its own spec that I ran on GPU. The other
bridge-specific models now keep their own spec by the same code path; I
did not run them.

THD part, `scripts/rollout_logprob_diff.py` with Megatron QAT, CP=1,
`use_remove_padding=True`, on a small Qwen3-MoE (random init, 4 layers,
32 experts):

- **Before the fix**, on GH200 with the stock wheelhouse TE
(`--quant-linear FP8_BLOCK --quant-moe FP8_BLOCK --qat-mode fp8`) and on
an RTX 5090 (`--quant-moe MXFP4 --qat-mode mxfp4`): the assertion above
fires at the first actor forward.
- **With the equivalent override**
`+actor_rollout_ref.actor.megatron.override_transformer_config.use_arbitrary_attention_mask=False`:
both run to completion. On the 5090, MXFP4 QAT gives k3 KL 2.3e-4,
against 1.7e-3 for the same MXFP4 rollout without QAT.
- These THD runs used the override, not this branch. The code path in
this PR sets the same provider field through `provider_overrides`.

Hopper doc claim: on GH200 with vLLM 0.29, loading the trainer-exported
MXFP4 preset fails for every backend:
- `auto` raises `NotImplementedError: No MXFP4 MoE backend supports the
deployment configuration`;
- `deep_gemm` reports an unsupported device;
- `marlin` and `triton` reject the FP8 1x128 activation scheme;
- `rollout_logprob_diff.py --quant-moe MXFP4` fails at engine start.

### Design & Code Changes

- `verl/workers/engine/megatron/transformer_impl.py`: QAT spec selection
moves into `_use_qat_layer_spec(provider, provider_overrides)`. It
returns early unless the provider uses `default_layer_spec`; otherwise
it sets `modelopt_transformer_layer_spec` and
`provider_overrides.setdefault("use_arbitrary_attention_mask", False)`.
- `tests/workers/test_megatron_qat_layer_spec_on_cpu.py`: new CPU test.
- `docs/low_precision/qat.md`: Hopper note for the `MXFP4` preset and
the QAT layer-spec limitation.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply pre-commit checks.
- [x] Add / Update the documentation.
- [x] Add unit tests (CPU).

AI assistance was used for this change. I reviewed every changed line
and ran the tests above.

---------

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: wuxibin <wuxibin@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants