Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
180 commits
Select commit Hold shift + click to select a range
533ba79
feat(megatron): add real NVFP4 end-to-end training
zhangyimi Aug 31, 2026
b288876
build: bind NVFP4 image to source state
zhangyimi Aug 31, 2026
915b53d
build: allow NVFP4 jobs alongside other workloads
zhangyimi Sep 1, 2026
02e56f9
fix: avoid host uv in scheduled NVFP4 jobs
zhangyimi Sep 1, 2026
b4391af
fix: make runtime probe architecture neutral
zhangyimi Sep 1, 2026
d4d8fa8
build: bootstrap pinned uv in runtime image
zhangyimi Sep 1, 2026
a4b44bc
build: attest megatron source version
zhangyimi Sep 1, 2026
66d3014
test: align NVFP4 grouped preflight shapes
zhangyimi Sep 1, 2026
9be1a1c
fix: preserve legacy vllm weight updates
zhangyimi Sep 1, 2026
547979c
fix: guard optional flash attention 4 import
zhangyimi Sep 1, 2026
e777730
fix: backport writable nvfp4 moe scales
zhangyimi Sep 1, 2026
e077e99
fix: align R3 replay rows for FP4
zhangyimi Sep 1, 2026
8b9f928
fix: backport stateless TE checkpoint state
zhangyimi Sep 1, 2026
18bed46
fix: audit completed FP4 smoke teardown
zhangyimi Sep 1, 2026
f75eb7c
feat: add guarded real NVFP4 long chain
zhangyimi Sep 1, 2026
ab8a8b1
fix: backport post-v0.26 online NVFP4 fixes
zhangyimi Sep 1, 2026
e59008c
test: isolate FlashInfer NVFP4 MoE PDL hang
zhangyimi Sep 1, 2026
4f3f212
fix: initialize diagnostic bundle path before readonly
zhangyimi Sep 1, 2026
729ff9b
fix: invoke reused diagnostic submit harness with bash
zhangyimi Sep 1, 2026
9ea5879
docs: record real NVFP4 submission incidents
zhangyimi Sep 1, 2026
0b629e3
docs: record successful PDL-off validation
zhangyimi Sep 1, 2026
b5b100e
test: isolate PDL-off on the validated v8 image
zhangyimi Sep 1, 2026
b0cc910
docs: record the v11 rollout collapse root cause
zhangyimi Sep 1, 2026
c26f232
feat: gate the real NVFP4 long chain on audited rollout metrics
zhangyimi Sep 1, 2026
1d7d4ea
fix: shrink long-chain chunks to fit the 5h wall clock
zhangyimi Sep 1, 2026
71de61d
docs: record the stitched BF16 reference curve
zhangyimi Sep 1, 2026
23c1b2f
fix: keep long-chain chunk targets consistent
zhangyimi Sep 1, 2026
334f2c3
docs: note the attestation blind spot v11 slipped through
zhangyimi Sep 1, 2026
945c038
feat: compare a real NVFP4 arm to BF16 at matched steps
zhangyimi Sep 1, 2026
c900be8
docs: close out the PDL-off isolation and release the long chain
zhangyimi Sep 1, 2026
e040262
docs: record that a stalled .out does not mean a stalled chunk
zhangyimi Sep 1, 2026
f021218
fix: do not call a flat length slope a failure before BF16 takes off
zhangyimi Sep 1, 2026
63e2c46
test: add a single-variable BF16 control on the v12 stack
zhangyimi Sep 2, 2026
9ecd4e3
fix: point the BF16 control's build gates at the bundle that built th…
zhangyimi Sep 2, 2026
8cd7aa3
docs: record the W4A4 response-length plateau and its control
zhangyimi Sep 2, 2026
d30109a
docs: record two submission traps hit on the BF16 control
zhangyimi Sep 2, 2026
60f9403
docs: add the final 50 steps and quantify TIS out of contention
zhangyimi Sep 2, 2026
dd58a4c
docs: point the control section at the job that actually runs
zhangyimi Sep 2, 2026
c665b6f
test: prove the BF16 control on one node before spending eight
zhangyimi Sep 2, 2026
c9a78e3
docs: record the BF16 control timeout and the process lesson
zhangyimi Sep 2, 2026
9d25692
fix: never strand the weight sender when a receiver bucket fails
zhangyimi Sep 2, 2026
617a845
test: rebuild the BF16 control on the receiver-drain fix
zhangyimi Sep 2, 2026
778c356
fix: define the v15 base image before it is used
zhangyimi Sep 2, 2026
39c6bc4
fix: stop clobbering the MoE expert weight_loader on modern vLLM
zhangyimi Sep 2, 2026
aaedad8
test: rebuild the BF16 control on the MoE weight_loader fix
zhangyimi Sep 2, 2026
b60e4cf
feat: allow non-quantized rollouts to opt into vLLM native reload
zhangyimi Sep 2, 2026
8a8775f
test: run the BF16 control through vLLM native reload
zhangyimi Sep 2, 2026
e382098
fix: propagate the native-reload flag through Ray runtime_env
zhangyimi Sep 2, 2026
c8b19be
docs: record the three bugs that blocked the BF16 control
zhangyimi Sep 3, 2026
0e8da1a
feat: make truncated importance sampling a single-variable knob
zhangyimi Sep 3, 2026
af468ee
test: add a W4A4 arm with truncated importance sampling off
zhangyimi Sep 3, 2026
42b0eea
docs: retract the TIS dismissal and record the recipe comparison
zhangyimi Sep 3, 2026
a3c75fd
feat: allow restoring strict Minerva scoring for math_dapo
zhangyimi Sep 3, 2026
ae80b08
test: add a W4A4 arm using strict Minerva scoring
zhangyimi Sep 3, 2026
7d8690e
docs: root-cause the length plateau to an upstream verl scoring change
zhangyimi Sep 3, 2026
becfdf3
feat: make DAPO dynamic sampling a knob alongside strict scoring
zhangyimi Sep 3, 2026
cb8912f
test: add a W4A4 arm matching NeMo's strict scoring plus dynamic samp…
zhangyimi Sep 3, 2026
e21c458
test: fully match NeMo - strict scoring, dynamic sampling, and no TIS
zhangyimi Sep 3, 2026
a27dd79
debug: instrument vLLM 0.26 RoutedExperts weight matching
zhangyimi Sep 4, 2026
6a38ba7
test: 1-node diagnostic arm for the BF16 MoE reload mismatch
zhangyimi Sep 4, 2026
5d9c57f
feat: make the first/last-layer BF16 carve-out a recipe knob
zhangyimi Sep 4, 2026
d734bbf
test: rebuild the BF16 MoE diagnostic on the carve-out knob commit
zhangyimi Sep 4, 2026
f701f3f
examples/real_nvfp4: add the BF16 reload repro and the v25 long-run b…
zhangyimi Sep 4, 2026
b0f317a
examples/real_nvfp4: scope the v25 verifier guard so it cannot match …
zhangyimi Sep 4, 2026
f518b24
real_nvfp4: install the R3 capture hook for any routed-expert rollout…
zhangyimi Sep 4, 2026
a66977e
examples/real_nvfp4: match the pretty-printed filter_groups block in …
zhangyimi Sep 4, 2026
fdbd3af
examples/real_nvfp4: extend the v25 chains to step 1000 with per-arm …
zhangyimi Sep 5, 2026
9e6132c
examples/real_nvfp4: stop the v25 numeric gate from refusing a health…
zhangyimi Sep 6, 2026
efaea3a
real_nvfp4: move FlashInfer past the SM100 MoE hang instead of workin…
zhangyimi Sep 7, 2026
e276a92
real_nvfp4: pin FlashInfer autotune per precision instead of inheriti…
zhangyimi Sep 7, 2026
d6fec6c
real_nvfp4: rebuild the runtime from the lock so the FlashInfer fix i…
zhangyimi Sep 7, 2026
2020455
examples/real_nvfp4: refresh the v28 carve-out assertion for the knob…
zhangyimi Sep 7, 2026
c4737b6
examples/real_nvfp4: point the v28 jobs at their own manifest
zhangyimi Sep 7, 2026
8a98912
real_nvfp4: turn FlashInfer autotune off for both arms
zhangyimi Sep 7, 2026
5fd401b
examples/real_nvfp4: production bundle with the overlong penalty alig…
zhangyimi Sep 7, 2026
a327b2c
real_nvfp4: derive the MoE layout from the config instead of allowlis…
zhangyimi Sep 7, 2026
46d6bee
real_nvfp4: count carved-out MLP modules against the sparse layers, n…
zhangyimi Sep 7, 2026
9bde7b6
examples/real_nvfp4: give v29 a clean uniform chunk schedule
zhangyimi Sep 7, 2026
3c261a2
tests: assert the structural real_nvfp4 model contract, not the Qwen3…
zhangyimi Sep 8, 2026
97764b7
examples/real_nvfp4: rebuild v29 as v29b so the image matches the cor…
zhangyimi Sep 8, 2026
e235a53
real_nvfp4: reapply the site-packages backports the rebuilt image dro…
zhangyimi Sep 8, 2026
95730cd
examples/real_nvfp4: v30 restores the PDL workaround and gates on 8-n…
zhangyimi Sep 8, 2026
b8a28ec
examples/real_nvfp4: keep the Ray head port out of Ray's worker range
zhangyimi Sep 8, 2026
2cff96a
real_nvfp4: pin FlashInfer to 0.6.15.post1, the last good SM100 MoE a…
zhangyimi Sep 8, 2026
24396b4
examples/real_nvfp4: make the numeric gate see NaN, and parse negativ…
zhangyimi Sep 8, 2026
f6a0227
fix(dapo): restore dynamic sampling and NVFP4 rollout parity
zhangyimi Sep 9, 2026
4f85578
test(real-nvfp4): assert rollout layer partition
zhangyimi Sep 9, 2026
3e6e960
chore(real-nvfp4): add exact-source v33 rerun
zhangyimi Sep 9, 2026
0611c9a
test(real-nvfp4): report missing preflight config field
zhangyimi Sep 9, 2026
4082d60
test(real-nvfp4): expect enabled trainer carve-out
zhangyimi Sep 9, 2026
30626a4
fix(real-nvfp4): scale dynamic smoke gate by profile
zhangyimi Sep 9, 2026
c3e0d8b
fix(real-nvfp4): pass online ignore via model config
zhangyimi Sep 9, 2026
d589e2a
chore(real-nvfp4): add v34 runtime rerun
zhangyimi Sep 9, 2026
5e77750
test(real-nvfp4): isolate rollout carve-out environment
zhangyimi Sep 9, 2026
28a27ef
chore(real-nvfp4): add exact-source v35 rerun
zhangyimi Sep 9, 2026
21e5e1e
chore(real-nvfp4): right-size production smoke gate
zhangyimi Sep 9, 2026
0104c84
chore(real-nvfp4): reuse unchanged v35 smoke evidence
zhangyimi Sep 9, 2026
c927cf7
Fix dynamic sampling reward alignment and audit optimizer updates
zhangyimi Sep 10, 2026
7934dbe
Handle missing AIME predictions in validation metrics
zhangyimi Sep 10, 2026
d3022b2
Validate TE 2.18 release with paired rollout evidence and workload ac…
zhangyimi Sep 15, 2026
551d68d
Harden NVFP4 runtime contracts and dynamic sampling lifecycle
zhangyimi Sep 15, 2026
e1e86de
Fix stale native NVFP4 derived scales across refits
zhangyimi Sep 15, 2026
26d2c20
fix: restore retained NVFP4 activation scales after level-2 sleep
zhangyimi Sep 15, 2026
7504e20
fix: release refit IPC mappings before final acknowledgment
zhangyimi Sep 15, 2026
c2f735e
Merge upstream and package TE 2.18 NVFP4 delivery candidate
zhangyimi Sep 16, 2026
06b37cb
fix: preserve false for vLLM boolean fields with delayed defaults
zhangyimi Sep 16, 2026
c9afd3c
fix: package Megatron FA4 distribution guard for clean runtimes
zhangyimi Sep 16, 2026
5d5c4c7
[rollout, perf] fix: preserve uint8 expert routes in agent storage
zhangyimi Sep 16, 2026
91f4b39
test: cover NVFP4 padding in integrated uint8 route replay
zhangyimi Sep 16, 2026
f73dbee
fix: materialize NVFP4 activation scales before native refit
zhangyimi Sep 16, 2026
dfbac03
test: replace metadata-only sleep fixture with real NVFP4 layout
zhangyimi Sep 16, 2026
81f6fc3
fix: backport stateless grouped checkpoint support for TE 2.18
zhangyimi Sep 16, 2026
6e7494b
test: scope legacy FP8 pickle opt-in to the trusted checkpoint fixture
zhangyimi Sep 16, 2026
5833b8c
validation: audit native runtime guards for vLLM 0.27.1 and Core 0.19
zhangyimi Sep 18, 2026
35ffef9
fix: accept audited vLLM 0.27.1 R3 capture entry
zhangyimi Sep 20, 2026
3f704ca
Update the official uv runtime for NVFP4 release validation
zhangyimi Sep 20, 2026
0538e16
Preserve declared extras in version-scoped uv metadata
zhangyimi Sep 20, 2026
a90d48f
Retain validated Bridge runtime requirements during exact sync
zhangyimi Sep 20, 2026
d4b6608
Integrate upstream updates and finalize real NVFP4 training support
zhangyimi Sep 20, 2026
76fc4ef
Merge upstream wheelhouse and CI updates
zhangyimi Sep 21, 2026
9faeb17
Remove previously declined compact route storage from NVFP4 scope
zhangyimi Sep 21, 2026
81978c9
Narrow NVFP4 contribution to runtime support and regression coverage
zhangyimi Sep 21, 2026
bb5558e
Separate NVFP4 integration from experiment-specific training logic
zhangyimi Sep 21, 2026
1e58067
Use vLLM 0.29 native routing capture and released NVFP4 fixes
zhangyimi Sep 21, 2026
de89f29
Remove experiment-only R3 success marker
zhangyimi Sep 22, 2026
cc98085
Merge current main and align NVFP4 with released Megatron dependencies
zhangyimi Oct 8, 2026
60082e8
Restore official Transformer Engine build isolation settings
zhangyimi Oct 8, 2026
9b05a52
Remove redundant runtime checks and Megatron FA4 backport
zhangyimi Oct 8, 2026
93a2c42
Simplify native NVFP4 integration and use upstream BF16 refit
zhangyimi Oct 8, 2026
1b37e1d
Align evaluation precision regression fixture with example layer sele…
zhangyimi Oct 8, 2026
4e9eed7
Simplify native NVFP4 integration
zhangyimi Oct 9, 2026
8508188
Return early from the real_nvfp4 vLLM quantization branch
zhangyimi Oct 9, 2026
65482ff
Use the DAPO recipe in the native NVFP4 example
zhangyimi Oct 9, 2026
c413e64
Refit native NVFP4 through layerwise reload over the regular bucket r…
zhangyimi Oct 9, 2026
ee4f51b
Rename the NVFP4 fake receiver to avoid shadowing an existing test he…
zhangyimi Oct 9, 2026
c60d483
Keep the regular MoE weight-loader patch call on the native NVFP4 path
zhangyimi Oct 9, 2026
7d6c4a4
Move the BF16 refit check into vllm_runtime.py, its only user
zhangyimi Oct 9, 2026
509bc3e
Leave the unrelated Megatron-Core version comment in the Dockerfile u…
zhangyimi Oct 9, 2026
61f5c46
Give the NVFP4 example a larger training micro-batch token budget
zhangyimi Oct 10, 2026
fe1f9b2
Trim NVFP4 tests to the core training, refit and rollout checks
zhangyimi Oct 12, 2026
34c838d
Merge current main into native NVFP4 support
zhangyimi Oct 12, 2026
85e9322
Gate the NVFP4 MTP guard on rollout MTP and pin the example working dir
zhangyimi Oct 13, 2026
0961bf2
Merge main and export native NVFP4 rollout weights from the trainer
zhangyimi Oct 19, 2026
da8152d
Keep non-NVFP4 packed experts packed in the NVFP4_PER_TOKEN export
zhangyimi Oct 19, 2026
8a8052d
Make the example's rollout ignore list safe for Ray's /bin/sh entrypoint
zhangyimi Oct 19, 2026
768678e
fix(rollout): clear NaN left in the KV cache by NVFP4_PER_TOKEN start-up
zhangyimi Oct 20, 2026
23acdb7
Merge main (#8306) into native NVFP4 PR
zhangyimi Oct 21, 2026
432f150
Check NVFP4_PER_TOKEN rollouts from the built quantization config
zhangyimi Oct 21, 2026
d4265b6
Generalize NVFP4_PER_TOKEN checks beyond the Qwen MoE layout
zhangyimi Oct 21, 2026
0f9723b
Update the per-token worker refit test to the name-based checks
zhangyimi Oct 21, 2026
9d6e8a6
Document the layouts native NVFP4 supports
zhangyimi Oct 21, 2026
b3e7390
Merge main into native NVFP4 PR
zhangyimi Oct 22, 2026
fe16d91
Update the NVFP4 R3 padding test to quant_align_size
zhangyimi Oct 22, 2026
5804c67
Read the real_nvfp4 backward mode from Transformer Engine only
zhangyimi Oct 22, 2026
5f2e5a6
Accept TE modules of a non-FP4 submodel in the real_nvfp4 attestation
zhangyimi Oct 22, 2026
28256f5
Re-apply the real_nvfp4 per-module recipe after the model is built
zhangyimi Oct 22, 2026
bd05189
Run real_nvfp4 experts in NVFP4 without an enclosing FP4 autocast
zhangyimi Oct 22, 2026
630aa26
Harden the NVFP4_PER_TOKEN rollout paths from review
zhangyimi Oct 22, 2026
37e8fe8
Add the NVFP4_PER_TOKEN log-prob mismatch to the QAT docs
zhangyimi Oct 22, 2026
a0c511b
Format the NVFP4_PER_TOKEN mismatch table like the table above
zhangyimi Oct 22, 2026
d16adb5
Move the real NVFP4 Megatron setup out of MegatronEngine
zhangyimi Oct 22, 2026
8e4501b
Enable NVFP4 training through override_transformer_config.fp4
zhangyimi Oct 23, 2026
7f9f533
Allow the NVFP4_PER_TOKEN rollout with EP=TP
zhangyimi Oct 23, 2026
f4695bf
Load only the local experts of an expert-parallel NVFP4_PER_TOKEN rol…
zhangyimi Oct 22, 2026
6ccedb6
Allow tensor-parallel experts in the NVFP4_PER_TOKEN rollout
zhangyimi Oct 23, 2026
1c2e4d3
Tighten NVFP4_PER_TOKEN checks from review
zhangyimi Oct 23, 2026
46b66bc
Export NVFP4_PER_TOKEN weights without vLLM; stop serving after a par…
zhangyimi Oct 23, 2026
32b5303
Merge main (vLLM 0.32) and use the wheelhouse Transformer Engine 2.20.2
zhangyimi Oct 27, 2026
bb95cd4
Default the THD alignment to Megatron-Core's NVFP4 recipe; free vLLM …
zhangyimi Oct 27, 2026
cb6ad9e
Describe the NVFP4_PER_TOKEN rollout scope as verl's, not vLLM's
zhangyimi Oct 27, 2026
cb01d30
Refit NVFP4_PER_TOKEN in place and address review comments
zhangyimi Oct 28, 2026
f7b8003
Split the NVFP4 training switch from the shared BF16 layer selection
zhangyimi Oct 28, 2026
1d4d593
Add versions, the example command and the long-run curves to the NVFP…
zhangyimi Oct 28, 2026
c2cc7f1
Explain why the NVFP4_PER_TOKEN refit copies results back itself
zhangyimi Oct 28, 2026
9ef174a
Spell out the tensor-parallel per-token scale note in the NVFP4 docs
zhangyimi Oct 28, 2026
70917ae
Refit NVFP4_PER_TOKEN through the shared quantized refit
zhangyimi Oct 29, 2026
a1677fe
Drop the per-token code from the rollout worker refit
zhangyimi Oct 29, 2026
1893adc
Select per-token NVFP4 with moe_activation instead of a separate preset
zhangyimi Oct 29, 2026
e87adec
fix ruff
wuxibin89 Oct 30, 2026
3b78ec3
Merge branch 'main' into feat/native-nvfp4
wuxibin89 Nov 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ verl is fast with:

low_precision/fp8.md
low_precision/mxfp8.md
low_precision/nvfp4.md
low_precision/nvfp4_qat.md
low_precision/qat.md

Expand Down
111 changes: 111 additions & 0 deletions docs/low_precision/nvfp4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# NVFP4 RL on Blackwell (experimental)

Last updated: 10/28/2026

This recipe trains the routed experts of an MoE model in real NVFP4 with Megatron and Transformer Engine,
and serves the rollout in the same precision. Activations are quantized per token on both sides.
Parameters stay BF16; attention, dense MLPs, shared experts, router, norms, embeddings and the LM head run
in BF16.

For NVFP4 fake-quantization training, see [NVFP4 QAT](nvfp4_qat.md). The rollout quantization interface
shared by all formats is described in [Low-Precision Rollout and Quantization-Aware Training](qat.md).

## Requirements

- Blackwell (SM100) GPUs
- Validated with PyTorch 2.13 / CUDA 13, Transformer Engine 2.20.2, Megatron-Core 0.19.2, Megatron-Bridge 0.6.2
and vLLM 0.32.0 (`uv sync --frozen --extra megatron --extra vllm`)
- Megatron backend, PP=1 without virtual pipeline parallelism
- vLLM rollout with TP and EP=1 or EP=TP, the colocated `naive` checkpoint engine and `FULL_DECODE_ONLY`
CUDA graphs. Data-parallel EP, EPLB, context parallelism (DCP/PCP) and speculative decoding are not
supported.

## Configuration

```yaml
actor_rollout_ref:
actor:
megatron:
override_transformer_config:
fp4: e2m1
fp4_recipe: nvfp4
first_last_layers_bf16: true
num_layers_at_start_in_bf16: 2 # whole decoder layers kept in BF16
num_layers_at_end_in_bf16: 4
rollout:
quantization:
moe: NVFP4
moe_activation: nvfp4_per_token # activations quantized per token, as in training
ignore: ["re:.*layers\\.(0|1|44|45|46|47)\\.mlp\\.experts\\..*"] # same layers, 48-layer model
```

- `fp4` applies only to the routed experts and cannot be combined with `qat`. Training quantizes their
activations per token in the forward pass and runs the backward GEMMs on the dequantized forward operands;
`NVTE_BACKWARD_OVERRIDE=high_precision` uses the BF16 operands instead.
- The rollout must keep exactly the training BF16 layers; the trainer checks every exported expert weight
against them.
- Transformer Engine reads its NVFP4 recipe from `NVTE_*` environment variables; the trainer rejects a
recipe that differs. See `examples/real_nvfp4/runtime_env.yaml`.
- Routed experts are found by name, so the recipe is not tied to one architecture. It is validated on
Qwen3-30B-A3B, Qwen3.5-35B-A3B and a small model with the DeepSeek-V3 layout (dense first layer, MLA,
shared expert).

## Rollout

`moe_activation: nvfp4_per_token` (vLLM's `quantization_config.moe.activation`) exports NVFP4 expert weights on the trainer side, bit-identical to vLLM's
online `nvfp4_per_token` quantization, and runs them on FlashInfer's TRT-LLM MoE kernel with per-token
activation scales. Refits load each bucket in place into the storage the CUDA graphs captured, so the extra
memory is one MoE layer's post-processing. With rollout TP and EP off, each rank holds a slice of every
expert and computes the per-token scale of the `down_proj` input over its slice, so results differ slightly
from TP=1.

## Mismatch

Qwen3.5-35B-A3B, EP=4 on 4 GPUs, vLLM rollout TP=1, measured with the script in
[qat.md](qat.md#measuring-the-mismatch):

| Rollout | Training | Backend | k3_kl |
|---|---|---|---|
| NVFP4 MoE | NVFP4 QAT | FlashInfer CUTLASS (MoE; BF16 Linear) | 0.00832 |
| NVFP4 MoE, `nvfp4_per_token` | BF16 | FlashInfer TRT-LLM (MoE; BF16 Linear) | 0.01025 |
| NVFP4 MoE, `nvfp4_per_token` | native NVFP4 | FlashInfer TRT-LLM (MoE; BF16 Linear) | 0.00740 |

## Example: Qwen3-30B-A3B-Base with DAPO

[`examples/real_nvfp4/run_qwen3_30b_megatron.sh`](https://github.com/verl-project/verl/blob/main/examples/real_nvfp4/run_qwen3_30b_megatron.sh)
runs DAPO with routing replay (R3) and token-level TIS on 8 nodes × 4 GB200, keeping the first two and last four
of the 48 layers in BF16; `PRECISION_MODE=bf16` runs the same job in BF16. With a Ray cluster running:

```bash
MODEL_PATH=/shared/models/Qwen3-30B-A3B-Base \
TRAIN_FILE=/shared/data/train.parquet \
TEST_FILE=/shared/data/validation.parquet \
PRECISION_MODE=real_nvfp4 EXP_NAME=qwen3-nvfp4 NNODES=8 \
bash examples/real_nvfp4/run_qwen3_30b_megatron.sh
```

See the [example README](https://github.com/verl-project/verl/blob/main/examples/real_nvfp4/README.md) for the
dataset format, runtime environment and tests.

### Results

700 steps of NVFP4 and BF16 with the example script, on an earlier version of this integration (vLLM 0.29).

![Training score](https://github.com/user-attachments/assets/d0693c0b-6789-4285-8658-44a885d72a72)
*`critic/score/mean`. Orange: NVFP4, red: BF16*

![Entropy](https://github.com/user-attachments/assets/e63b552c-fe9b-4b23-9751-62c38976732c)
*`actor/entropy`. Orange: NVFP4, red: BF16*

![Rollout-training mismatch](https://github.com/user-attachments/assets/13131fe5-c670-41ea-91e1-68e5fff123ee)
*`rollout_corr/kl`. Orange: NVFP4, red: BF16*

![Gradient norm](https://github.com/user-attachments/assets/b121c562-b6f7-4432-8a7e-ed5768d78e69)
*`actor/grad_norm`. Orange: NVFP4, red: BF16*

Results and observations:

- The training score follows BF16 for the first ~450 steps and is lower afterwards.
- The rollout–training mismatch is 0.003–0.01 with NVFP4, against about 0.0003 with BF16.
- NVFP4 rollout generation is about 1.3× faster than BF16 once the mean response exceeds ~2.5k tokens, giving
roughly 1.2× shorter steps; NVFP4 training itself is still slower than BF16 in Transformer Engine.
99 changes: 99 additions & 0 deletions examples/real_nvfp4/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Native NVFP4 training and rollout (experimental)

This example trains Qwen3-30B-A3B-Base with Megatron/Transformer Engine NVFP4
and runs vLLM rollout in the matching NVFP4 precision. Training parameters stay
BF16; each refit exports the routed experts as NVFP4 on the trainer side
(`rollout.quantization.moe=NVFP4` with `moe_activation=nvfp4_per_token`), and vLLM reloads them into the
storage its CUDA graphs captured.

## Precision and supported configuration

`actor_rollout_ref.actor.megatron.override_transformer_config.fp4=e2m1` (with
`fp4_recipe=nvfp4`) runs the routed-expert MLPs in NVFP4 with per-token activation scaling, dequantized backward, and
adaptive 4-over-6 disabled. The `nvfp4_per_token` rollout uses the same
precision: NVFP4 expert weights and activations quantized per token by
FlashInfer's TRT-LLM MoE kernel. Attention, router, norms, embeddings and the
LM head remain BF16 on both sides.

Megatron-Core's `first_last_layers_bf16` with `num_layers_at_start_in_bf16` and
`num_layers_at_end_in_bf16` keeps whole decoder layers in BF16 during training; `rollout.quantization.ignore` must list the same
routed-expert layers, which the trainer checks when it exports weights. This example keeps
the first two and last four of 48 layers in BF16.

Routed experts are found by their names (`mlp.experts`, unpacked or packed),
as the trainer-side export does, so the integration is not tied to one
architecture: dense MLP layers, shared experts and other attention types such
as MLA stay BF16 on both sides. It has been validated on Qwen3-30B-A3B and,
with a small random model, on the DeepSeek-V3 layout (dense first layer, MLA,
shared expert). Training uses PP=1 without virtual pipeline parallelism, and
rollout TP (`ROLLOUT_TP`, default 1) with EP=1 or EP=TP (`ROLLOUT_EP`), the
colocated `naive` checkpoint engine, BF16 KV cache, `FULL_DECODE_ONLY` CUDA
graphs and no speculative decoding.

## Installation

The runtime uses PyTorch 2.13.0/CUDA 13, Transformer Engine 2.20.2,
Megatron-Core 0.19.2, Megatron-Bridge 0.6.2 and vLLM 0.32.0:

```bash
uv sync --frozen --extra megatron --extra vllm
```

## Run

Use preprocessed datasets in verl's standard `RLHFDataset` format. Each row
contains a `prompt` chat-message list, a `data_source` identifier and a
`reward_model` mapping with `style: rule` and the answer in `ground_truth`.
The built-in reward dispatcher selects scoring from `data_source`; use
`math_dapo` for DAPO math or an `aime`-prefixed identifier for AIME. Optional
`extra_info` is passed through by the dataset. Dataset preprocessing belongs
outside this example; see the existing [preprocessing scripts](../data_preprocess).

Start Ray on the allocated nodes, then provide shared model and parquet paths:

```bash
MODEL_PATH=/shared/models/Qwen3-30B-A3B-Base \
TRAIN_FILE=/shared/data/train.parquet \
TEST_FILE=/shared/data/validation.parquet \
CKPTS_DIR=/shared/checkpoints/my-run \
PRECISION_MODE=real_nvfp4 EXP_NAME=my-run NNODES=8 \
bash examples/real_nvfp4/run_qwen3_30b_megatron.sh
```

Use `PRECISION_MODE=bf16` for a control with the same example settings. The
launcher uses verl's standard PPO entry point with the built-in DAPO reward
manager and the default rule-based scorer for each `data_source`. The default
configuration is 8 nodes × 4 GPUs, EP=4, DAPO (GRPO advantages, clip-higher,
token-level loss, dynamic sampling on accuracy and a 512-token overlong
penalty), R3 routing replay and token-level TIS, training on 32 prompts × 16
responses per update. Adam uses learning rate 1e-6 and betas (0.9, 0.999).

Rollout memory utilization defaults to 0.8, with 256 sequences and 32768 batched
tokens. Set `TOTAL_TRAINING_STEPS`, `RESUME_MODE` and checkpoint paths for the
run. Checkpoints include optimizer, scheduler and dataloader progress.
Logging uses verl's standard console and W&B integration.

Transformer Engine reads its NVFP4 settings from `NVTE_*` environment variables
in each worker; `runtime_env.yaml` sets them and the trainer rejects a recipe
that differs. Keep these variables when passing another file via `RUNTIME_ENV`.
Only `NVTE_BACKWARD_OVERRIDE` may change: `dequantized` (the default here) runs
the backward GEMMs on the dequantized forward operands, `high_precision` on the
original BF16 ones.

R3 uses vLLM's native routing capture inside the fused MoE path. Runtime checks
validate the exported expert tensors, the NVFP4 layer selection and the scale
values after each reload.

## Tests

Run GPU tests in a Blackwell allocation using the prepared runtime:

```bash
export CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1
export FLASHINFER_DISABLE_FP4_QUANT_FAST_MATH=1 TRTLLM_DISABLE_FP4_QUANT_FAST_MATH=1
python -m pytest -q tests/utils/real_nvfp4
python -m pytest -q tests/utils/test_bucketed_weight_transfer.py
```

Coverage includes recipe compatibility, the rollout preset, the in-place refit, and
bit-exact agreement of the trainer-side export with vLLM's online NVFP4 quantization.
Loading
Loading