Skip to content

[megatron]feat: Deepseek v4 RL MXFP4 experts qat support - #7895

Draft
sophiayyya wants to merge 28 commits into
verl-project:mainfrom
sophiayyya:mxfp4-experts-qat
Draft

sophiayyya wants to merge 28 commits into
verl-project:mainfrom
sophiayyya:mxfp4-experts-qat

Conversation

@sophiayyya

@sophiayyya sophiayyya commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

MXFP4 QAT precision and data flow

This change adds routed-expert weight-only MXFP4 fake quantization on top of Transformer Engine (TE) FP8 training for DeepSeek-V4-Flash-0731. Checkpoint storage, trainable parameter dtype, and GEMM precision are separate:

Module / tensors Official 0731 checkpoint storage Parameters after Bridge import Added ModelOpt QAT
Routed experts: w1, w2, w3 MXFP4 E2M1, two values per I8 byte; one E8M0 scale per 32 contiguous input-channel elements BF16 MXFP4 weight fake-quant on mlp.experts.linear_fc1/linear_fc2
Shared experts; MLA wq_a, wq_b, wkv, wo_a, wo_b; indexer wq_b FP8 E4M3 with E8M0 scales over 128 × 128 weight blocks BF16 Disabled
Embeddings, prediction head, router, norms; compressor projections and indexer weights_proj BF16 BF16 Disabled
Native FP32 tensors, including mHC hc_attn_fn/hc_ffn_fn, attn_sink, and compressor ape FP32 FP32 retained Disabled

I8 is the byte container for packed FP4 values, not INT8 quantization. The checkpoint precision map establishes tensor storage and scale geometry; it does not establish runtime kernel precision. Bridge's E2M1 unpacking and the QAT configuration provide the implementation-level interpretation of the packed expert weights. Only indexer wq_b is FP8 in this table; the entire indexer is not uniformly FP8.

flowchart TD
    C["0731 checkpoint: MXFP4 experts / FP8 projections / BF16 / FP32"]
    I["Bridge import: unpack and dequantize quantized weights"]
    P["Trainable model parameters: BF16 plus native FP32 tensors"]
    Q["Routed expert weights: dynamic MXFP4 Q/DQ; BF16 temporary output"]
    T["TE FP8 quantization and GEMM: hybrid / blockwise recipe"]
    O["Adam update: FP32 main parameters and optimizer state; HDO CPU offload"]
    E["Bridge export: requantize updated parameters to MXFP4 / FP8 weight-scale pairs"]
    V["vLLM rollout / validation: quantization=fp8; KV cache=fp8"]
    C --> I --> P
    P -->|routed expert weights| Q
    Q --> T
    P -->|other TE FP8 linear paths| T
    T --> O
    O -->|update model parameters| P
    P -->|weight synchronization| E
    E --> V
Loading

QAT scope and kernel. qat.mode=mxfp4_experts disables all ModelOpt quantizers first, then enables only *mlp.experts*weight_quantizer, with num_bits=(2, 1) and block_sizes={-1: 32, "type": "dynamic", "scale_bits": (8, 0)}. Scales are computed on each call; algorithm=None requires no calibration pass. Shared experts, attention, router, and all ModelOpt input/output quantizers remain disabled. This change does not implement FP4 indexer QK computation or additional KV/output-projection activation fake-quantization.

In the validated ModelOpt 0.44 implementation, MXFP4 fake-quant uses modelopt_cuda_ext_mx.fused_amax_convert → CUDA amax_and_convert_kernel. It returns BF16 values on the MXFP4 grid without permanently packing the trainable parameters. TE then quantizes weights and activations for its FP8 GEMMs. On the validated H100 setup, this simulates MXFP4 weight error while computing the supported GEMMs in FP8; it is not native FP4 GEMM. bypass_te_fp8_assert=True allows this explicit composition. Direct FP8 parameter loading and fp8_param are not enabled.

Scale and rollout handling. Checkpoint E8M0 scales are distinct from runtime compute scales: the recipe sets NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1 for TE and VLLM_USE_DEEP_GEMM_E8M0=0 for the vLLM DeepGEMM path. Bridge exports the updated model parameters and independently recreates packed MXFP4 expert weights and FP8 projection weights with their scales. It does not export a previous forward pass's temporary fake-quant result. The generic NVFP4 QATWeightExporter is skipped for this mode, and rollout.qat.enable=False still retains Bridge's quantized weight synchronization. The diagram describes export formats and rollout configuration, not the compute precision of every vLLM kernel.

Checkpoint compatibility. Dynamic MXFP4 quantizers have no persistent calibration tensors. preserve_mxfp4_checkpoint_methods therefore preserves native TE/Megatron checkpoint methods while retaining the QAT forward path, with a guard rejecting persistent quantizer state. HDO resume restores optimizer main shards to the GPU before loading optimizer state. Neither fix changes the quantization recipe. Native FP32 parameters with HDO additionally require the detach fix in NVIDIA/Megatron-LM#6982, as documented in the example script.

Implementation: quantization configuration, checkpoint compatibility, and 40-step QAT example.

Quantization experiment evidence

The completed H100 experiment ran 40 steps with a 4096-token response limit, validation at steps 10/20/30/40, checkpoints at steps 20/40, and optimizer-state resume between the two 20-step segments. Step-40 AIME validation accuracy was 64.21% with MXFP4 QAT, versus 64.59% for the TE FP8 control without ModelOpt QAT. These are single-run measurements, not evidence of a statistically significant difference. See the QAT run and control run.

This experiment predates the final rebase. It used the implementation before publication cleanup and Megatron 1ff25ca7e with the equivalent local FP32-shard detach patch; the runtime patch helper is not included in this PR. These results validate that experiment, not a fresh 40-step run of the rebased head.

image

Checklist Before Starting

  • Search for similar PRs. Paste at least one query link here: ...
  • Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
    • {modules} include fsdp, megatron, veomni, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data, cfg, reward, fully_async, one_step_off
    • If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
    • {type} is in feat, fix, refactor, chore, test
    • If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
    • Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

sopyang and others added 6 commits September 16, 2026 07:53
The DeepSeek-V4 report describes FP4 quantization-aware training applied to MoE expert weights
during post-training. This adds `qat.mode=mxfp4_experts`: a weight-only ModelOpt fake-quant onto
the MXFP4 grid (E2M1 values, one E8M0 power-of-two scale per 32 contiguous K elements) scoped to
routed experts via `*mlp.experts*weight_quantizer`, with `algorithm=None` because dynamic block
quantization derives its scale per call and ModelOpt asserts it is never calibrated.

The GEMMs keep running in real FP8 through TE's blockwise autocast rather than falling back to a
simulated BF16 matmul. ModelOpt refuses to run any quantizer under `fp8_autocast` -- a guard that
keeps pure simulation faithful, but here the stacking is the point (FP4 weights feeding a real FP8
GEMM, which is what the checkpoint's own mixed MXFP4/FP8 layout implies). `qat.bypass_te_fp8_assert`
lifts it at runtime without patching ModelOpt. Activations are quantized once, by TE; ModelOpt's
input_quantizer stays disabled so they are not quantized twice.

Two adjacent fixes were required to make QAT usable on such a model:

- The ModelOpt layer spec is no longer forced on providers that declare an
  `experimental_attention_variant` (DeepSeek-V4 "dsv4_hybrid", GLM5 "dsa", Qwen3-Next
  "gated_delta_net"). Those install their own heterogeneous spec in `provider_bridge()`, and the
  generic ModelOpt GPT spec would silently replace it and build a different model. ModelOpt swaps
  QuantModules by class, so the spec change is not needed for quantization to apply.
- `export_qat_weights` is gated to the NVFP4 modes it actually implements. For `mxfp4_experts` the
  bridge already re-quantizes routed experts to MXFP4 on every weight sync, so the training forward
  and the rollout weights land on one grid without a second exporter.

`apply_qat_to_modules` now fails fast if the experts-only config did not land where intended: it
requires at least one enabled routed-expert weight quantizer and no other enabled quantizer (a
mis-ordered cfg would leave every remaining linear on ModelOpt's default per-tensor INT8), and logs
whether the fake-quant is the identity on the loaded weights -- it is, exactly, when the checkpoint
already stores those experts on the MXFP4 grid.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-authored-by: OpenAI Codex <noreply@openai.com>
Mirrors run_deepseek_v4_flash_megatron.sh with TE fp8 blockwise training, fp8 vLLM rollout and the
optional `mxfp4_experts` QAT (off by default). Documents why the FP4 step is simulated on Hopper
while the FP8 GEMM is not, and why filter_overlong_prompts must stay off for DeepSeek-V4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Preserve native Megatron checkpoint methods around stateless expert MXFP4
quantization, and restore offloaded optimizer shards before loading HDO state.

Add independent length-only and MXFP4 expert QAT recipes with a 4096-token
response budget, 40 total steps and validation every 10 steps. Require model
and dataset locations from the environment and use portable output defaults.

Validation: checkpoint regressions passed (6 and 7 tests); the original
length-only and QAT runs each completed 40 steps including save/resume.
Published launch arguments match both completed experiments at 20/40 steps.
Shell syntax, required-input checks and credential/path scans passed.
Two pre-existing repository checks are excluded: stale generated reference
configuration and naming of an older example. New example names pass.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Remove the two checkpoint regression test files from this branch as requested.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Record the HDO FP32-shard detach prerequisite, upstream fix and validation
provenance in the standalone MXFP4 QAT script. Remove the length-only recipe
and its stale reference from the QAT header.

Validation: bash -n and git diff --check pass; the executable QAT script body
is byte-for-byte unchanged.

Co-authored-by: OpenAI Codex <noreply@openai.com>
Rename the original DeepSeek-V4-Flash QAT example to use the required
Megatron backend suffix, preserving its contents. Refresh the generated
Megatron reference config with the existing bypass_te_fp8_assert defaults.

Co-authored-by: Codex
@CLAassistant

CLAassistant commented Sep 17, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ sophiayyya
❌ Sophia Yang


Sophia Yang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

wuxibin89 pushed a commit that referenced this pull request Sep 29, 2026
…8062)

### What does this PR do?

The FP8 end-to-end section of `docs/low_precision/fp8.md` lists
`NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1` as a general requirement. It only
applies on Hopper.

On Blackwell (SM100+), Transformer Engine has no native block-scaled
GEMM. It runs the block-wise recipe through MXFP8 GEMMs, whose E8M0
scales hold only an exponent. So both of its block-quantize kernels
assert power-of-two scales on SM100+, and a run with the variable set
stops at the first FP8 quantization:

```
transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:534 in function quantize_transpose_vector_blockwise:
Assertion failed: pow2_scale. On Blackwell and newer, the FP8 block scaling recipe is emulated with MXFP8, which requires using power of two scaling factors.
```

This is the failure reported in #6172 (8×B200) when following this doc
and the verl-recipe FP8 E2E script. This PR marks the variable as
Hopper-only and says that on Blackwell it must stay unset; power-of-two
scales are the Transformer Engine default. Docs only.

Refs #6172. This does not fix #6172: that issue also reports garbled FP8
rollouts on Blackwell once the variable is removed, and this PR does not
address that.

### Checklist Before Starting

- [x] Search for similar PRs:
-
[NVTE_FP8_BLOCK_SCALING_FP32_SCALES](https://github.com/verl-project/verl/pulls?q=NVTE_FP8_BLOCK_SCALING_FP32_SCALES)
- [blockwise
Blackwell](https://github.com/verl-project/verl/pulls?q=blockwise+Blackwell)
- No PR changes this requirement. Related: #6172 (the issue above);
#7895 (sets the variable in its DeepSeek-V4 example scripts, does not
touch this doc); #7519 (MXFP8 training, does not change this line).
- [x] Format the PR title as `[{modules}] {type}: {description}`

### Test

Documentation-only change.

- `pre-commit run --files docs/low_precision/fp8.md`: all hooks pass,
including "Check docs Last updated info".
- Reproduced on 1×B200 (SM 10.0, torch 2.11.0+cu130, CUDA 13.0,
Transformer Engine 2.16.1). `te.Linear` (2048→2048) and
`te.GroupedLinear` (4 × 2048→768) ran forward and backward under
`Float8BlockScaling(fp8_format=Format.E4M3)` and were compared with a
bf16 run of the same module:

| `NVTE_FP8_BLOCK_SCALING_FP32_SCALES` | `te.Linear` out / dgrad / wgrad
rel. err | `te.GroupedLinear` out / dgrad / wgrad rel. err |
|---|---|---|
| unset (power-of-two scales) | 0.0377 / 0.0423 / 0.0462 | 0.0376 /
0.0475 / 0.0510 |
| `1` | `RuntimeError`: the `pow2_scale` assertion above | same |

The same modules under `MXFP8BlockScaling` give the same errors as the
unset row, with or without the variable.

Source references (Transformer Engine 2.16.1, commit c9877beb):
-
`transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:533-536`
and
`transformer_engine/common/transpose/quantize_transpose_square_blockwise.cu:490-493`:
`if (sm_arch() >= 100) NVTE_CHECK(pow2_scale, ...)`.
- `transformer_engine/pytorch/csrc/extensions/gemm.cpp:289-300`:
"Emulate the FP8 block scaling recipe with MXFP8 on Blackwell and newer
as it is not natively supported by cublasLt".
- `transformer_engine/common/swizzle/swizzle_block_scaling.cu:116-118,
205-206`: each FP32 block scale becomes an E8M0 scale by keeping only
its exponent byte, `(sf >> 23) & 0xFF`. This is why the scales must be
powers of two.

### API and Usage Example

None.

### Design & Code Changes

`docs/low_precision/fp8.md`:
- The requirement line now says that the variable is Hopper-only, why it
fails on Blackwell, and what to do there.
- "Last updated" is bumped.

The verl-recipe script
`low_precision/run_dapo_qwen3_moe_30b_megatron_fp8e2e.sh` also exports
the variable unconditionally. That lives in the recipe repository and
can get a matching note separately.

### Checklist Before Submitting

- [ ] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/verl-project/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: documentation-only
change; the behaviour it documents was checked on B200 as described
under Test.
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1).
- [ ] If your PR is related to the `recipe` submodule, please also
update the reference to the submodule commit: not applicable.

AI assistance: this PR was prepared with AI assistance (Claude Code). I
reviewed the change and the evidence above, and the checks listed under
Test were run for this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wuxibin89 added a commit that referenced this pull request Nov 2, 2026
…h live refit (#7971)

### What does this PR do?

Add end-to-end native NVFP4 W4A4 training and rollout on NVIDIA
Blackwell GPUs, demonstrated with Qwen3-30B-A3B-Base DAPO using
Megatron/Transformer Engine and vLLM.

This contributes to Sophia's [low-precision roadmap
#5972](#5972), specifically
**NVFP4 real quant** and **NVFP4 training in Megatron**. In brief:

- **Training:** Transformer Engine runs selected routed-expert MLPs in
NVFP4 with per-token activation scaling and dequantized backward.
Persistent policy weights remain BF16, with full optimizer state.
- **Weight synchronization:** following the trainer-side quantization
boundary of #8287, the trainer exports the routed experts as NVFP4 with
`moe: NVFP4` and `moe_activation: nvfp4_per_token` after each policy
update.
- **Rollout:** vLLM loads the NVFP4 expert weights and quantizes
activations per token, matching training. Refit preserves the storage
referenced by CUDA graphs.

Attention and the configured boundary-layer MLPs remain BF16. The Qwen3
example keeps the first two and last four MLP layers in BF16 and applies
NVFP4 to the remaining 42 routed-expert MLP layers.

### Checklist Before Starting

- [x] Search for similar PRs:
[NVFP4](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+NVFP4),
[roadmap
#5972](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+5972+in%3Abody).
- [x] Format the PR title as `[{modules}] {type}: {description}`.

Related work: #6455 targets ModelOpt/TRT-LLM synchronization; #6992
provides FSDP W4A8 simulation; #7895 targets DeepSeek MXFP4 on TE FP8;
#7519 adds MXFP8 training and rollout. #8287 adds ModelOpt NVFP4 QAT
with static activation scales; this PR adds native TE NVFP4 training
with per-token activation scaling, on the same export path.

### Test

Unit tests (CPU and GB200) pass; scoped Ruff, formatting and lockfile
checks pass. The trainer-side export is bit-identical to vLLM's online
`nvfp4_per_token` quantization, and 6-step runs on Qwen3-30B-A3B (8 × 4
GB200) and a small DeepSeek-V3-layout model pass.

#### Long run

BF16 and W4A4 runs of the example script below (Qwen3-30B-A3B-Base,
DAPO, R3, token-level TIS) on 8 nodes × 4 GB200 GPUs, 700 steps with
checkpoint resume across jobs. **W4A4 vs BF16:**

**Training score (`critic/score/mean`)**

<img width="1813" height="957" alt="image"
src="https://github.com/user-attachments/assets/d0693c0b-6789-4285-8658-44a885d72a72"
/>

**Entropy (`actor/entropy`)**

<img width="1814" height="952" alt="image"
src="https://github.com/user-attachments/assets/e63b552c-fe9b-4b23-9751-62c38976732c"
/>

**Train–rollout mismatch (`rollout_corr/kl`)**

<img width="1811" height="952" alt="image"
src="https://github.com/user-attachments/assets/13131fe5-c670-41ea-91e1-68e5fff123ee"
/>


**Gradient norm (`actor/grad_norm`)**

<img width="1813" height="953" alt="image"
src="https://github.com/user-attachments/assets/b121c562-b6f7-4432-8a7e-ed5768d78e69"
/>

**Performance:** at matched response length, NVFP4 rollout generation is
about 1.3× faster than BF16 once the mean response exceeds ~2.5k tokens,
giving roughly 1.2× shorter end-to-end steps; NVFP4 training itself is
still slower than BF16 in Transformer Engine.

### API and Usage Example

With a Ray cluster running on eight nodes, provide model and dataset
paths shared by all workers:

```bash
MODEL_PATH=/shared/models/Qwen3-30B-A3B-Base \
TRAIN_FILE=/shared/data/train.parquet \
TEST_FILE=/shared/data/validation.parquet \
EXP_NAME=qwen3-native-nvfp4 \
PRECISION_MODE=real_nvfp4 \
bash examples/real_nvfp4/run_qwen3_30b_megatron.sh
```

Use `PRECISION_MODE=bf16` for the baseline. NVFP4 training is enabled
like FP8 training, through
`actor_rollout_ref.actor.megatron.override_transformer_config` (`fp4:
e2m1`, `fp4_recipe: nvfp4`, and `first_last_layers_bf16` with
`num_layers_at_start_in_bf16` / `num_layers_at_end_in_bf16`), and
rollout through `actor_rollout_ref.rollout.quantization.moe=NVFP4` with
`moe_activation=nvfp4_per_token`, whose `ignore` list keeps the same
layers in BF16; the trainer checks that the two agree. The example uses
the standard verl PPO entry point, parquet datasets and built-in reward
manager, with the DAPO recipe (clip-higher, token-level loss, dynamic
sampling and the overlong penalty), R3 routing replay and token-level
TIS. See [the example
guide](https://github.com/zhangyimi/verl/blob/feat/native-nvfp4/examples/real_nvfp4/README.md)
for configuration and runtime setup.

### Design & Code Changes

- **Training precision:** the Megatron per-module recipe (attention
BF16, routed-expert MLPs NVFP4, carved-out layers BF16) is generated
from the configuration rather than maintained as a file. After model
construction, the effective precision of every attention and expert
linear layer is checked against it. Packed THD sequences use the TE
alignment for FP4 as for FP8.
- **Rollout config and refit:** `moe_activation=nvfp4_per_token` selects
a compressed-tensors scheme (NVFP4 weights, fully dynamic NVFP4 inputs).
The trainer quantizes the experts with its own Triton kernel, sharing
one global scale across `gate_proj`/`up_proj`, bit-identical to vLLM's
online `nvfp4_per_token` quantization. In vLLM the scheme selects
FlashInfer TRT-LLM with per-token activations; refits load each bucket
in place into the expert storage captured by CUDA graphs and
post-process each MoE layer, so the extra memory is one MoE layer.
- **Routing replay:** use vLLM’s native fused-MoE routing capture and
align replay data with FP4 training padding.
- **Runtime compatibility:** Transformer Engine 2.20.2 from the verl
wheelhouse. No vLLM source patch is needed.

The validated Megatron + vLLM stack uses:

| Component | Version |
|---|---|
| PyTorch | 2.13 / CUDA 13 |
| vLLM | 0.32.0 |
| Transformers | 5.16.1 |
| Megatron-Core | 0.19.2 |
| Megatron-Bridge | 0.6.2 |
| Transformer Engine | 2.20.2 |
| FlashInfer | 0.7.0.post1 |
| CUTLASS DSL / QuACK | 4.8.0 / 0.6.5 |

SGLang end-to-end validation remains follow-up work.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [ ] Run the complete all-files pre-commit suite. Focused lint,
formatting, shell and lockfile checks have passed.
- [x] Add usage, design and runtime compatibility documentation.
- [x] Add focused unit and end-to-end coverage. Project CI integration
remains to be completed.
- [x] Request project CI and review.
- [x] Check recipe coordination: the recipe submodule reference is
unchanged; no companion recipe PR is required.

AI assistance: OpenAI Codex and Anthropic Claude assisted with
implementation, testing and this PR description.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: wuxibin <wuxibin@bytedance.com>
wuxibin89 added a commit that referenced this pull request Nov 2, 2026
### What does this PR do?

Add shared TileLang quantization kernels for FP8_BLOCK, MXFP8, and MXFP4
QAT and parameter export. QAT uses fused quantize-dequantize with
straight-through gradients, while export uses the same kernels without
dequantization so values and scales follow the same numerical recipe.
NVFP4 QAT uses triton kernel in
#8494.
This overlaps with
[#7895](#7895), which adds
DeepSeek-specific routed-expert MXFP4 weight-only QAT using ModelOpt's
CUDA extension. This PR instead provides reusable TileLang kernels for
FP8_BLOCK, MXFP8, and MXFP4, applies them to both weight/activation fake
quantization and checkpoint export, and keeps the existing QAT mode
interface.

AI assistance was used for implementation, numerical comparison,
testing, debugging, and preparation of this PR description. The human
submitter directed the numerical design and must review every changed
line and validation result before requesting maintainer review.

### Checklist Before Starting

- [x] Search for similar PRs: [QAT quant kernel
search](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+QAT+quant+kernel),
[MXFP4 QAT
search](https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+MXFP4+QAT).
- [x] Format the PR title as `[{modules}] {type}: {description}`.

### Test

```bash
# FP8, MXFP8, MXFP4 kernels, export integration, and ModelOpt integration
pytest -q tests/utils/test_qat_fp8_on_gpu.py tests/utils/test_qat_fp4_on_gpu.py tests/utils/test_qat_export_kernels_on_gpu.py tests/utils/test_modelopt_mxfp8_on_gpu.py
# 51 passed

# CPU CI-equivalent environment
uv run --frozen --all-packages --extra cpu pytest -s -x --asyncio-mode=auto tests/utils/real_nvfp4
# 24 passed, 1 skipped

uvx --with hydra-core pre-commit run --all-files --show-diff-on-failure --color=always
# all hooks passed

git diff --check origin/main...HEAD
# passed
```

FP8 and MXFP4 quantized values, scales, and fused QDQ outputs were also
compared bitwise against the DeepSeek-V4.1 inference reference kernels.

### API and Usage Example

```python
from verl.utils.qat.fp4 import fp4_quant
from verl.utils.qat.fp8 import fp8_quant

fp8_weight, fp8_scale = fp8_quant(weight, (128, 128))
mxfp8_weight, mxfp8_scale = fp8_quant(weight, (1, 32), scale_fmt="e8m0")
mxfp4_weight, mxfp4_scale = fp4_quant(weight)

# Fused fake quantization used by QAT:
dequantized_activation = fp8_quant(activation, (1, 32), scale_fmt="e8m0", dequant=True)
```

### Design & Code Changes

- Add 2D FP8_BLOCK and 1D MXFP8 TileLang kernels with optional fused
dequantization and FP32/E8M0 scale output.
- Add a packed 1x32 MXFP4 TileLang kernel with optional fused
dequantization.
- Route ModelOpt FP8_BLOCK, MXFP8, and MXFP4 fake quantizers through the
new kernels while retaining straight-through gradients.
- Route compressed-tensors FP8_BLOCK, MXFP8, and MXFP4 CUDA weight
export through the same kernels; retain CPU and NVFP4 fallbacks.
- Add numerical and export integration tests and document the shared
QAT/export behavior.
- Upgrade ModelOpt to 0.47.0 and update the lockfile.
- Isolate Megatron stubs in the Muon CPU tests so they do not pollute
later tests through `sys.modules`.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [x] Add unit tests to cover the kernels, QAT integration, and export
integration.
- [ ] Once the PR is ready for CI, send a message in the `ci-request`
channel.
- [x] This PR is not related to the `recipe` submodule.
sophiayyya and others added 11 commits November 4, 2026 11:28
…eam/main)

- mxfp4_experts moves from verl/utils/modelopt to verl/utils/qat/modelopt.py as one more QAT mode
- bypass_te_fp8_assert config knob removed: the mode itself lifts modelopt's TE-FP8 guard, and the
  QAT x TE-FP8 mutual-exclusion check exempts it
- DeepSeek-V4 native fp8 quantization_config accepts qat.mode=mxfp4_experts in dispatch_rollout_weight_quantizer
- drop rollout.quantization=fp8 and the length-only recipe; HDO resume fix already upstream (verl-project#8235)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… in the DSv4 QAT recipe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in the DSv4 QAT recipe

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Covers the quantize config (routed-expert weight_quantizers only), the rollout dispatch pairing with
DeepSeek-V4's native fp8 quantization_config, the on-GPU fake-quant (idempotent on the MXFP4 grid, other
linears untouched) and the fail-fast layout check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…; test imports the mcore patch first

_check_mxfp4_experts matched ".mlp.experts." with a leading dot, so a routed-expert quantizer on a module
rooted at mlp.experts (a bare MoE block, as in the unit test) was reported as unexpected. The test also imports
verl.models.mcore before modelopt, as the Megatron engine does, so its megatron plugin does not trip over
flash_attn.cute on the mcore dev branch, and builds the qat namespace directly instead of importing the
worker config package.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Dev Megatron-Core maps apply_dsa_kernel_fusion=True to dsa_kernel_backend=cudnn, which hard-requires
flash_mla (nv_dev) and fails fast when it is not installed. Expose the switch so the recipe can run
on the PyTorch fallback.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…antizers

modelopt 0.47 quantizes a TE grouped linear with a GroupedQuantizer whose per-expert quantizers are
named `...linear_fc1.weight_quantizer.{i}`; _check_mxfp4_experts only accepted names ending in
`weight_quantizer`, so on DeepSeek-V4 it saw 0 expert quantizers and raised. Add a single-GPU test
on a real Megatron-Core MoE (TE grouped GEMM) that pins this layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n it applies

Megatron-Core dev removed dist_checkpointing.strategies.async_utils (NVIDIA/Megatron-LM#7052), so the
unconditional import made `import verl.models.mcore` fail on any newer mcore.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…CPU offload

Megatron-Core dev asserts that optimizer_cpu_offload (HybridDeviceOptimizer) runs on the precision-aware
optimizer path. Main grads and Adam moments stay fp32 by default, so numerics are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eek-V4) models

- RouterReplay accepts is_mtp_layer (NVIDIA/Megatron-LM#7310 passes it from TopKRouter) and every
  router is registered once: drop the replay object Megatron-Core builds itself before adding ours.
- The hash-routing wrapper passes new arguments (dense_output, verl-project#7450) through.
- is_moe_layer reads hybrid_layer_pattern when present: Megatron-Bridge >= 0.7 builds DeepSeek-V4 as
  attention + MoE hybrid sublayers, so only the 'E' positions own a router.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sophiayyya and others added 5 commits November 9, 2026 22:45
On Megatron-Core dev, mcore_fsdp_adapter.FullyShardedDataParallel is a factory dispatching to
FullyShardedDataParallelV1/V2, so using it in isinstance raised TypeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… that has it

HybridModel (DeepSeek-V4 via Megatron-Bridge >= 0.7) exposes Megatron's output_processor hook, so it
needs no GPT-specific forward patch. When a model can be neither hooked nor patched, the engine now
disables fused kernels instead of calling the unpatched forward with fused-only kwargs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stale rollout.qat

- PIPELINE_MODEL_PARALLEL_LAYOUT may be empty (PP=1 / provider-built layout).
- ROLLOUT_MOE_BACKEND: verl defaults quantized models to DeepGEMM, whose MXFP4 MoE kernel is
  Blackwell-only; use marlin on Hopper.
- actor_rollout_ref.rollout.qat no longer exists on RolloutConfig.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…atron-Bridge's helper

Megatron-Bridge >= 0.7 builds DeepSeek-V4 as attention + MoE hybrid sublayers. With PP > 1 the
hybrid_layer_pattern has to be split at stage boundaries as well, which a logical-layer
pipeline_model_parallel_layout string cannot express. When no layout is given, call Bridge's
set_deepseek_v4_pipeline_model_parallel_layout (what its own recipes use).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stage

Megatron-Core requires hash-routed MoE layers on the stage that owns input_ids. Bridge's helper splits
logical layers evenly, which breaks this when the first stage gets fewer blocks than num_hash_layers
(e.g. 16 layers over PP8). Grow the first stage when needed (only then; the full 43-layer Flash model keeps
the even split) and pass the counts to the helper. CPU unit test included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wuxibin89 pushed a commit that referenced this pull request Nov 10, 2026
### What does this PR do?

Adds the MXFP4 recipe from the low-precision roadmap (#8186): the MXFP4
counterpart of the MXFP8 recipe in #8127, for the routed experts of MoE
models.

- `examples/mxfp4/run_qwen3_30b_a3b_megatron.sh`: GRPO on Qwen3-30B-A3B
with DAPO-Math-17k on one 4-GPU Blackwell node (tested on 4×B300; 4×B200
untested). It uses the data, hyper-parameters and layout of
`examples/mxfp8/run_qwen3_30b_a3b_megatron.sh` (training TP2 × EP2, full
recompute, optimizer on the CPU; rollout vLLM TP2). `PRECISION` selects
the arm:
- `mxfp4_qat`: MXFP4 rollout experts (`rollout.quantization.moe=MXFP4`)
and ModelOpt MXFP4 QAT on the routed experts (`qat.mode=mxfp4`)
  - `mxfp4`: MXFP4 rollout experts only
  - `bf16`: the baseline
- `docs/low_precision/mxfp4_qat.md`: requirements, configuration, notes
and the example; linked from `docs/index.rst`. The page is named after
QAT because Transformer Engine has no MXFP4 training: the routed experts
are fake-quantized, everything else trains in BF16.
- The example sets `data.seed` (default 1), so every arm sees the same
data order.

Results, ten GRPO steps per arm on one 4×B300 machine (details in the
doc page and under Test): QAT cuts the rollout–trainer mismatch of the
MXFP4 rollout by 80 % (`rollout_corr/kl` 0.0150 → 0.0030; BF16 0.0018)
and makes a step 6 % longer than the MXFP4 rollout alone on an idle
machine (900 s vs 848 s; BF16 827 s).

The QAT cost assumes the grouped QAT and export kernels of #8498 and
#8500, both merged; this PR no longer depends on an open PR.

### Checklist Before Starting

- [x] Search for similar PRs: no open PR adds an MXFP4 recipe or example
(searched `mxfp4`; #7895 is DeepSeek-V4 MXFP4 expert QAT, #8398 the
Hopper MoE backend default).
- [x] Format the PR title as `[{modules}] {type}: {description}`.

### Test

```bash
python3 tests/special_sanity/check_docs_time_info.py            # ✅
python3 tests/special_sanity/check_example_naming.py --root examples   # ✅ 115 scripts
python3 tests/special_sanity/check_uv_gpu_only.py               # ✅
python3 tests/special_sanity/validate_structure.py              # ✅
```

All three arms of the new script compose (`--cfg job`, `VERL_USE_UV=0`):
the QAT arm resolves to `qat.enable=true`, `qat.mode=mxfp4`,
`rollout.quantization.moe=MXFP4`, TP2 × EP2 for actor and ref.

GPU, B300 (SM103). The versions verl locks: vLLM 0.32.0, Transformer
Engine 2.20.2, Megatron-Core 0.19.2, Megatron-Bridge 0.6.2, ModelOpt
0.47.0, TileLang 0.1.15.

1. verl main 174888f, one GPU, `scripts/rollout_logprob_diff.py
--r3-router-replay` on a small random Qwen3-MoE (4 layers, 32 experts,
top-4), run before the QAT path moved to the TileLang kernels:

   | Arm | vLLM MoE backend | k3 KL |
   |---|---|---|
   | BF16 | FlashInfer TRT-LLM (unquantized) | 1.0e-5 |
   | MXFP4 rollout experts | `DEEPGEMM_MXFP4` | 1.0e-3 |
   | MXFP4 rollout experts + MXFP4 QAT | `DEEPGEMM_MXFP4` | 4.0e-5 |

   QAT brings the training forward 26× closer to the MXFP4 rollout.

2. Four GPUs, the configuration of this script with
`PRECISION=mxfp4_qat` (host-specific cache paths added), one GRPO step
on Qwen3-30B-A3B (TP2 × EP2): QAT builds on the Transformer Engine spec
with tensor and expert parallelism combined, weight sync and the step
complete with `rollout_corr/kl` 0.0030. With the grouped kernels this
first step takes 1096 s (generation 121 s, `old_log_prob` 170 s, `ref`
110 s, `update_actor` 635 s, `update_weights` 18 s).

3. Four GPUs, the three arms of this script for 10 steps each, one after
the other on the same machine, verl main e41dd69 plus this PR.
Host-specific cache paths, Ray `num_cpus=64`, the venv's python started
directly instead of through `uv run`, the console, file and wandb
loggers, DAPO-Math-17k deduplicated to its 17,398 unique prompts as in
#8127.

   | mean of steps 1–10 | BF16/BF16 | BF16/MXFP4 | BF16+QAT/MXFP4 |
   |---|---|---|---|
   | step time (s) | 827 | 848 | 1000 |
   | generation (s) | 107 | 114 | 115 |
   | `old_log_prob` (s) | 118 | 118 | 149 |
   | `ref` (s) | 113 | 117 | 130 |
   | `update_actor` (s) | 476 | 477 | 582 |
   | `update_weights` (s) | 8 | 18 | 19 |
   | `rollout_corr/kl` | 0.0018 | 0.0150 | 0.0030 |
   | score | −0.195 | −0.161 | −0.183 |
   | response length | 6517 | 6411 | 6438 |

The QAT arm's steps 1–4 overlapped another tenant on the same node (the
machine's other four GPUs at 100 %); its steps 5–10 ran on an idle
machine: step time 900 s, `old_log_prob` 135 s, `update_actor` 511 s,
`rollout_corr/kl` 0.0030.

wandb:
[QAT](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/z52e8qr2),
[MXFP4](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/mzx5r4ys),
[BF16](https://wandb.ai/train0z5a/verl_grpo_dapo_mxfp4/runs/xd7yxpnw).

### Design & Code Changes

- New example script; shares its parameter arrays and launch block with
the MXFP8 example, with `PRECISION` arms for MXFP4.
- New doc page with the results; no code changes.
- The example sets `data.seed=${SEED:-1}`. With `data.seed` unset, as in
the MXFP8 example, the shuffle is unseeded and each run sees a different
data order.

### Checklist Before Submitting

- [x] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply pre-commit checks.
- [x] Add / Update the documentation.
- [ ] Add unit or end-to-end test(s): example script only, no CI test;
the three-arm E2E run is under Test.

AI assistance was used for this change. I reviewed every changed line
and ran the checks above.

---------

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
thecourageyu pushed a commit to thecourageyu/verl that referenced this pull request Nov 11, 2026
…erl-project#8062)

### What does this PR do?

The FP8 end-to-end section of `docs/low_precision/fp8.md` lists
`NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1` as a general requirement. It only
applies on Hopper.

On Blackwell (SM100+), Transformer Engine has no native block-scaled
GEMM. It runs the block-wise recipe through MXFP8 GEMMs, whose E8M0
scales hold only an exponent. So both of its block-quantize kernels
assert power-of-two scales on SM100+, and a run with the variable set
stops at the first FP8 quantization:

```
transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:534 in function quantize_transpose_vector_blockwise:
Assertion failed: pow2_scale. On Blackwell and newer, the FP8 block scaling recipe is emulated with MXFP8, which requires using power of two scaling factors.
```

This is the failure reported in verl-project#6172 (8×B200) when following this doc
and the verl-recipe FP8 E2E script. This PR marks the variable as
Hopper-only and says that on Blackwell it must stay unset; power-of-two
scales are the Transformer Engine default. Docs only.

Refs verl-project#6172. This does not fix verl-project#6172: that issue also reports garbled FP8
rollouts on Blackwell once the variable is removed, and this PR does not
address that.

### Checklist Before Starting

- [x] Search for similar PRs:
-
[NVTE_FP8_BLOCK_SCALING_FP32_SCALES](https://github.com/verl-project/verl/pulls?q=NVTE_FP8_BLOCK_SCALING_FP32_SCALES)
- [blockwise
Blackwell](https://github.com/verl-project/verl/pulls?q=blockwise+Blackwell)
- No PR changes this requirement. Related: verl-project#6172 (the issue above);
verl-project#7895 (sets the variable in its DeepSeek-V4 example scripts, does not
touch this doc); verl-project#7519 (MXFP8 training, does not change this line).
- [x] Format the PR title as `[{modules}] {type}: {description}`

### Test

Documentation-only change.

- `pre-commit run --files docs/low_precision/fp8.md`: all hooks pass,
including "Check docs Last updated info".
- Reproduced on 1×B200 (SM 10.0, torch 2.11.0+cu130, CUDA 13.0,
Transformer Engine 2.16.1). `te.Linear` (2048→2048) and
`te.GroupedLinear` (4 × 2048→768) ran forward and backward under
`Float8BlockScaling(fp8_format=Format.E4M3)` and were compared with a
bf16 run of the same module:

| `NVTE_FP8_BLOCK_SCALING_FP32_SCALES` | `te.Linear` out / dgrad / wgrad
rel. err | `te.GroupedLinear` out / dgrad / wgrad rel. err |
|---|---|---|
| unset (power-of-two scales) | 0.0377 / 0.0423 / 0.0462 | 0.0376 /
0.0475 / 0.0510 |
| `1` | `RuntimeError`: the `pow2_scale` assertion above | same |

The same modules under `MXFP8BlockScaling` give the same errors as the
unset row, with or without the variable.

Source references (Transformer Engine 2.16.1, commit c9877beb):
-
`transformer_engine/common/transpose/quantize_transpose_vector_blockwise.cu:533-536`
and
`transformer_engine/common/transpose/quantize_transpose_square_blockwise.cu:490-493`:
`if (sm_arch() >= 100) NVTE_CHECK(pow2_scale, ...)`.
- `transformer_engine/pytorch/csrc/extensions/gemm.cpp:289-300`:
"Emulate the FP8 block scaling recipe with MXFP8 on Blackwell and newer
as it is not natively supported by cublasLt".
- `transformer_engine/common/swizzle/swizzle_block_scaling.cu:116-118,
205-206`: each FP32 block scale becomes an E8M0 scale by keeping only
its exponent byte, `(sf >> 23) & 0xFF`. This is why the scales must be
powers of two.

### API and Usage Example

None.

### Design & Code Changes

`docs/low_precision/fp8.md`:
- The requirement line now says that the variable is Hopper-only, why it
fails on Blackwell, and what to do there.
- "Last updated" is bumped.

The verl-recipe script
`low_precision/run_dapo_qwen3_moe_30b_megatron_fp8e2e.sh` also exports
the variable unconditionally. That lives in the recipe repository and
can get a matching note separately.

### Checklist Before Submitting

- [ ] Read the [Contribute
Guide](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md).
- [x] Apply [pre-commit
checks](https://github.com/verl-project/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting).
- [x] Add / Update [the
documentation](https://github.com/verl-project/verl/tree/main/docs).
- [ ] Add unit or end-to-end test(s) to [the CI
workflow](https://github.com/verl-project/verl/tree/main/.github/workflows)
to cover all the code. If not feasible, explain why: documentation-only
change; the behaviour it documents was checked on B200 as described
under Test.
- [ ] Once your PR is ready for CI, send a message in [the `ci-request`
channel](https://verl-project.slack.com/archives/C091TCESWB1).
- [ ] If your PR is related to the `recipe` submodule, please also
update the reference to the submodule commit: not applicable.

AI assistance: this PR was prepared with AI assistance (Claude Code). I
reviewed the change and the evidence above, and the checks listed under
Test were run for this PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sophia Yang and others added 5 commits November 17, 2026 07:43
…e); TileLang weight QDQ

DeepSeek-V4's routed-expert FP4 QAT is now the regular `mxfp4` mode instead of a separate mode:

- New `qat.activation` ("mxfp8" default, or "none"). "none" drops the routed-expert MXFP8 input quantizer, so
  the experts are weight-only -- for an MXFP4 W4A16 rollout (BF16 training) or for TE FP8 training.
- `mxfp4` + `activation=none` is the one QAT configuration allowed together with
  override_transformer_config.fp8 (TE's FP8 GEMM quantizes the activations once); only then is ModelOpt's
  TE-FP8 guard lifted. Every other QAT + TE FP8 combination is still rejected.
- Expert weights use verl's TileLang MXFP4 QDQ (`verl_mxfp4` backend, DeepSeek-V4 UE8M0 scale recipe), batched
  over all experts of a TE grouped linear; on real DeepSeek-V4-Flash weights it matches ModelOpt's native MX
  fake-quant bit for bit.
- DeepSeek-V4 native fp8 checkpoints pair with `mxfp4` in dispatch_rollout_weight_quantizer.
- The stateless-checkpoint guard and the routed-expert quantizer check apply to `mxfp4`.
- Tests renamed to test_mxfp4_weight_only*; recipe, qat.md and megatron.yaml updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Hopper

With FP32 block scales (NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1) the TE FP8 cast re-rounds 48% of the
MXFP4-grid routed-expert weights of DeepSeek-V4-Flash; with TE's default power-of-two scales it changes none
(measured on 12 expert matrices of the 0731 checkpoint), matching DeepSeek-V4's lossless FP4->FP8 dequant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d re-registered params in place

- Humming joins the refittable MXFP4 backends. With VLLM_HUMMING_INPUT_QUANT_CONFIG selecting FP8 activations
  (1x128 dynamic groups) it serves MXFP4 weights x FP8 activations on Hopper via WGMMA dequant.
- Humming's conversion deletes and re-creates every w13_*/w2_* parameter with setattr and publishes only some of
  them through replace_parameter. A refit therefore left the CUDA graph reading stale values of the others:
  on a 6-layer DeepSeek-V4-Flash slice the rollout/trainer probs diff was 0.16 with Humming W4A8 vs 0.028 with
  Marlin W4A16. Every live parameter of a staged MoE module is now recorded before staging, and any parameter
  re-registered outside replace_parameter is copied back into the captured storage afterwards (shape/dtype must
  match). With this the Humming W4A8 probs diff is 0.027-0.029.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rollout

ROLLOUT_HUMMING_W4A8=True selects vLLM's Humming MoE backend and passes VLLM_HUMMING_INPUT_QUANT_CONFIG (FP8
activations, dynamic 1x128 groups, FP32 scales) to the rollout workers through Ray's runtime_env, so the MXFP4
experts run as W4A8 on Hopper instead of Marlin's W4A16. Checked on a 6-layer DeepSeek-V4-Flash slice: the
knob alone (no extra overrides) gives the Humming backend and a probs diff of 0.028-0.029.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mxfp4 now builds its quantizer config from the upstream ModelOpt presets, so the extra startup check (enabled
quantizer census and on-grid identity log) no longer guards anything the unit tests do not already cover. The
mcore test asserts the per-expert quantizer count and that the QDQ equals the TileLang kernel directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants