Repository navigation
Description
Model Engine
Low Precision Training
Background
Goals in RL
- Cheaper rollout and inference-only forwards (ref, teacher, old_log_prob): less memory, less memory traffic, faster GEMM.
- Lower train/rollout mismatch once rollout is quantized: training should see the same quantization as sampling.
Two routes
| Route | Training compute | Training speedup | Key issues |
|---|---|---|---|
| BF16 + QAT | BF16, fake quant (quantize-dequantize + STE) on weight / act / KV in the rollout format | no | quantizer parity with rollout kernels; QAT noise raises gradient variance |
| Real low-precision compute | FP8 / MXFP8 / MXFP4 / NVFP4 GEMMs (e.g. TE), FP32 master weights | yes | rowwise/columnwise copies per format; low-precision FSDP all-gather; train/rollout format alignment |
Where FP32 master weights live decides the backend design:
- In the optimizer (Megatron): model holds BF16 or low-precision compute weights, cast once per optimizer step on the
local shard; easy to keep low-precision params persistent. - In the model (FSDP2 / VeOmni): model holds sharded FP32 params, cast to compute dtype at every all-gather; any optimizer
works, high-precision weights always available (checkpoint, weight sync), cast cost repeats per micro-batch.
Formats
| Format | Block | Scale | fprop / dgrad see the same W | HW |
|---|---|---|---|---|
| FP8 per-tensor | tensor | FP32 | yes | Hopper+ |
| FP8 blockwise | A 1x128, W 128x128 | FP32 or pow2 | yes for W (2D block) | Hopper |
| MXFP8 | 1x32 | E8M0 | no (rowwise and columnwise quantized separately) | Blackwell |
| MXFP4 | 1x32 | E8M0 | no | Blackwell (Hopper via FP4->FP8 dequant) |
| NVFP4 | 1x16 + per-tensor FP32 | E4M3 | yes when weight uses 16x16 2D (TE default) | Blackwell |
DeepSeek V4 / V4.1 reference
- MXFP4 QAT for routed experts and the indexer QK path; FP32 master -> FP4 -> lossless dequant to FP8 128x128, so
training reuses the FP8 framework and dgrad uses the same W as fprop (V4 §3.4). - Real FP4 weights for rollout, ref and teacher forwards (V4 §5.2.1).
- KV cache: FP8 + BF16 RoPE dims (V4 §2.3.4); FP4 main KV cache QAT, SWA KV stays FP8 (V4.1 §2.4.4).
- Comm: BF16 + stochastic rounding for MoE grad sync, all-to-all + local FP32 sum (V4 §3.5.1).
- Batch-invariant / deterministic kernels for bitwise train/infer alignment (V4 §3.3).
Existing
BF16 + QAT
NVFP4 QAT (Quantization-Aware Training) in verl
| Backend | QAT format (training) | Rollout format | Implementation | Verified |
|---|---|---|---|---|
| FSDP | NVFP4 W4A16 (group 16) | NVFP4 W4A16, vLLM compressed-tensors |
verl/utils/qat/ |
Qwen3-8B-Base, Qwen3-30B-A3B-Base |
| Megatron | NVFP4 W4A16 (group 16) | NVFP4 W4A16, vLLM modelopt |
verl/utils/modelopt/ |
Qwen3-8B-Base, Qwen3-30B-A3B-Base |
| VeOmni | FP8 blockwise (W 128x128, A 1x128; linear / KV NoPE / indexer QK / MoE) + FP4 1x32 routed experts | FP8 blockwise + MXFP4 experts, vLLM Mxfp4MoEMethod refit |
veomni/ops/qat/, verl/utils/vllm/vllm_fp4_utils.py |
DeepSeek-V4-Flash |
Real low-precision compute
| Backend | Mode | Training compute | Rollout compute | Implementation | Verified |
|---|---|---|---|---|---|
| FSDP | rollout-only | BF16 | FP8 blockwise, vLLM / SGLang / TRT-LLM | {vllm,sglang}/*fp8_utils.py |
Qwen3-8B-Base, Qwen3-30B-A3B-Base |
| Megatron | rollout-only | BF16 | FP8 blockwise, vLLM / SGLang / TRT-LLM | same as above | Qwen3-30B-A3B |
| Megatron | E2E | FP8 blockwise via TE (fp8: hybrid, fp8_recipe: blockwise) + FP8 optimizer |
FP8 blockwise, vLLM | Megatron-Bridge + TE | Qwen3-8B, Qwen3-30B-A3B (Hopper); on Blackwell TE runs blockwise as MXFP8 (#6172) |
| VeOmni | - | - | - | - | not supported |
Q4 Plan
| Priority | Common | Megatron | VeOmni |
|---|---|---|---|
| P0 | train/rollout quantizer parity | DeepSeek-V4 FP4->FP8 lossless path | DeepSeek-V4/V4.1 QAT E2E |
| P1 | unified recipe config; low-precision inference-only forwards; low-precision comm | Blackwell MXFP8 E2E; Blackwell NVFP4 training | torchao low-precision compute + FSDP2 all-gather; torchao NVFP4/MXFP4 QAT |
| P2 | batch-invariant / deterministic kernels | FP8 optimizer states; MoE W4A8 | Muon + BF16 Newton-Schulz |
Common @wuxibin89
- P0 Train/rollout quantizer parity: same quantizer + scale fmt (ue8m0/e4m3), report mismatch KL & rollout throughput [BREAKING][megatron, vllm, sglang, rollout, doc] refactor: unify QAT and low-precision rollout #8287 [2/3][sglang] feat: support mxfp8/mxfp4/nvfp4 weight refit #8306 [3/3][vllm] feat: standardize low-precision kernels on DeepGEMM #8308 [megatron] feat: add TileLang FP8 and FP4 QAT kernels #8498 [megatron, rollout, perf] feat: add grouped TileLang quantization for QAT and export #8500
- P1 Unified low-precision recipe config across FSDP/VeOmni/Megatron/Automodel
- today:
rollout.quantization,megatron.override_transformer_config.fp8*,fsdp_config.qat,megatron.qat, veomniqat_implementation, automodelenable_fp8 - recipe = train compute fmt + QAT (weight/act/KV fmt, modules) + rollout fmt; fail fast on train/rollout fmt mismatch
- today:
- P1 Batch-invariant / deterministic train+rollout kernels for bitwise alignment https://github.com/Inferact/zero-mismatch-rl [fsdp, trainer, cfg, doc, tests] feat: batch-invariant actor forward and sampler log-prob formula #8026
Megatron @ISEEKYAN @wuxibin89
Route: Megatron-LM + TransformerEngine, master weights in the optimizer.
-
Compute weights are BF16, or FP8/MXFP8 with
fp8_param/fp8_param_gather; DistributedOptimizer / Megatron-FSDP
keep sharded FP32 main params and cast back after each step. -
TE provides the low-precision GEMMs and recipes (per-tensor delayed/current, blockwise, MXFP8, NVFP4) and the
rowwise/columnwise copies; under MXFP8, Megatron-FSDP gathers rowwise for fprop and columnwise for dgrad. -
QAT goes through modelopt; HF <-> mcore weights and FP8 export go through Megatron-Bridge.
-
Fit: large-scale MoE (TP/PP/EP/CP), mature FP8 stack. Cost: new models need mcore / Bridge support.
-
P0 Modern Megatron integration: [RFC] Modern Megatron integration #8324
-
P0 DeepSeek-V4 FP4->FP8 lossless path: FP4 fake quant in FP32 master -> FP8 (128x128) cast, reuse TE blockwise +
fp8_param_gather(V4 §3.4)- 2D 128x128 block => fprop/dgrad use the same W, no re-quant of W^T
- monitor max/min FP4 sub-block scale ratio per 128x128 block, fallback when lossless condition breaks
-
P1 Blackwell MXFP8 E2E: TE MXFP8 +
fp8_param_gather+reuse_grad_buf_for_mxfp8_param_ag- resolve train MXFP8 (1x32, fprop/dgrad W differ) vs rollout blockwise 128x128 (DeepGEMM) format mismatch [megatron] feat: MXFP8 training recipe and end-to-end example #8127
-
P1 Blackwell NVFP4 training: TE NVFP4 recipe (16x16 2D weight, RHT, SR) + vLLM NVFP4 W4A4 rollout [megatron, vllm, env] feat: add native NVFP4 training and rollout with live refit #7971 [BREAKING][megatron, vllm] feat: per-token activations for NVFP4 QAT #8494
-
P2 FP8 optimizer states, MoE W4A8 (FP4 x FP8 GEMM) [megatron, doc] feat: MXFP4 MoE recipe and Qwen3-30B-A3B example #8426
VeOmni @wuxibin89 @FoolPlayer @amenomori
Route: FSDP2 + torchao, master weights in the model.
-
Model holds sharded FP32 params; FSDP2
MixedPrecisionPolicycasts each shard to BF16 at all-gather; any standard
optimizer works unchanged. -
QAT is VeOmni's own fake-quant ops (
veomni/ops/qat/, STE) wired into patchgen-generated HF modeling, which makes
new models (e.g. DeepSeek-V4) quick to bring up. -
Real low-precision compute is to come from torchao: float8 / MX linear, plus float8 FSDP2 all-gather that quantizes
the high-precision shard before gather (low-precision comm without persistent low-precision weights). -
Fit: HF-native models, fast model bring-up, flexible recipes. Cost: no low-precision compute yet; current TileLang
QAT kernels are SM90 + BF16 only. -
P0 DeepSeek-V4/V4.1 QAT E2E (blocks Model Support: DeepSeek-V4.1-Flash)
-
P1 torchao low-precision compute: float8 (tensorwise/rowwise), MXFP8 on Blackwell
- low-precision FSDP2 all-gather: quantize sharded high-precision weights before gather (float8 / MXFP8)
-
P1 torchao fake-quant QAT for NVFP4 / MXFP4
Model Support
- DeepSeek-V4.1-Flash
- Qwen3.8-Flash-Next
- GLM-5.3-Flash
Score Centering
- vllm+FSDP: [rollout, vllm, algo, fsdp, trainer, doc] feat: score centering for bypass-mode REINFORCE #8010 [algo, rollout, cfg, doc] feat: automatically align score centering with sampler support #8196
- Megatron: [algo, megatron, cfg, doc, tests] feat: score centering on the Megatron actor #8199
- sglang support
- fuse CE kernel: [perf, fsdp, megatron, veomni] feat: add TileLang fused linear cross entropy #8654
- recipe&doc: [recipe, veomni, data, cfg, doc, tests] feat: Qwen3-30B-A3B 32K Score Centering recipe & data guides #8644
Rollout
Async rollout
- P0 Token-level interrupt + persist KV cache / routing across weight switch, no re-prefill on resume
- P1 Sample-level dispatch: dispatch next prompt once #completed samples >= group size [BREAKING][trainer, cfg] feat: stream V1 rollout dispatch through a dataloader actor with session-watermark refill #8203
Weight refit optimization
- Delta shard support for quantized
(weight, scale) - NCCL/NIXL M2N reshard [ckpt, megatron] feat: export local shards through NCCL M2N #8088 [ckpt, megatron, rollout, vllm, ci, doc] feat: sharded_nixl checkpoint engine with no trainer-side all-gather #8395
Performance optimization
- Centralized kv-cache pooling [rollout, vllm, sglang, trtllm, cfg] feat: salt external KV caches by weight version instead of resetting them #8323
- kv-cache offload
Fault tolerance
Target on v1 separate_async mode: @HaoWang113 @ETOgaosion @tongyx361
- RFC: [RFC] FT proposal for separate_async: independent trainer-side and rollout-side restarts #8380
- Rollout replica restart without interrupting trainer
- Trainer restart without interrupting rollout
- Sandbox recover on restart
- Rollout trajectory data persistent, e.g 3fs, oss, see discussion in #8738
Uni-Agent
Accelerator support
We will keep enhancing multi-accelerator support: built-in for NVIDIA / Ascend / AMD, and out-of-tree for others via
verl-hardware-plugin.
Huawei Ascend
Huawei Ascend NPU Q4 roadmap: Huawei Ascend verl Q4 roadmap
AMD ROCm
- AMD Q4 roadmap: [roadmap][ROCm] veRL on AMD roadmap 26Q4 #8537
- ROCm platform backend #6702, MI300 e2e CI #6668
- DeepSeek-V4-Flash GRPO on AMD GPUs #7050, on MI355X with QAT / FP8 / BF16 #8146
Google TPU
- Ray TPU resource discovery #7849, SFT with TorchTitan engine #8098
- Plugin: TPU platform plugin#18, TorchTitan TPU engine plugin#31, vLLM rollout on multi-host TPU plugin#34, Raiden P2P weight sync plugin#37
- GRPO with TorchTitan + vLLM via plugin #8155
Trainer
- Deprecate
verl/experimental/fully_async_policy|one_step_off_policyto verl-recipe [BREAKING][trainer, recipe, ci, doc] refactor: remove V0 trainer and migrate legacy async trainers to recipes #8625 - Remove V0 trainer
verl/trainer/ppo/ray_trainer.py
verl-omni
verl-omni Q4 roadmap: verl-project/verl-omni#700
rl-insight
rl-insight Q4 roadmap:verl-project/rl-insight#191
Activity
cr-gao commentedon Oct 11, 2026
Hi @wuxibin89 , I'd like to work on "Token-level interrupt + persist KV cache / routing across weight switch, no re-prefill on resume" under the async rollout part.
Btw for batch-invariant rollout, #8025 and #8026 might be related. As they are initially opend for qwen3-omni true on-policy training on verl-omni side, I can later align it with a verl native model. Thanks for any feedback.
SamitHuang commentedon Oct 12, 2026
It'd be better to link VeRL-Omni Q4 roadmap as well verl-project/verl-omni#700
0z5a commentedon Oct 17, 2026
Hi @wuxibin89 @ISEEKYAN, I'd like to take two items under Model Engine → Low Precision Training → Common.
1. [P0] Train/rollout quantizer parity
scale_fmt, with a fail-fast check when train and rollout formats disagree.Starting points I found on main:
verl/utils/kernel/fp8_kernel.pycompute scales differently (amax/448 vs. 1/(448/amax)), so they are not bitwise equal to each other.verl/utils/qat/linear.py) and the exporter (verl/utils/qat/quantizer.py) compute the e4m3 block scale in a different op order and handle zero scales differently.2. [P1] Batch-invariant / deterministic train+rollout kernels
Two questions before I start:
I'll open a draft PR with the parity harness and the FP8 blockwise tests first.
VikingDeng commentedon Oct 18, 2026
Hi @wuxibin89 @ISEEKYAN, I'd like to pick up the P1 "unified low-precision recipe config" item under Common.
Right now the same thing is configured in six places:
rollout.quantization,fsdp_config.qat,megatron.qat,megatron.override_transformer_config.fp8*, VeOmni'sqat_implementationand Automodel'senable_fp8. The FSDP and Megatron YAMLs even giveQATEngineConfigdifferent defaults (ignore_patterns,activation_observer), and nothing checks that train and rollout formats agree. I ran into this with Qwen3-0.6B GRPO on GSM8K (one RTX PRO 6000, main @ 8ced5d2). With BF16 training androllout.quantization=fp8,training/rollout_probs_diff_meangoes from ~0.0058 to ~0.031, and there's no warning.What I have in mind is one typed
low_precisionblock (train format, QAT, rollout format) that just expands into the existing backend fields. It would fail fast on combinations that don't make sense, and the old keys would keep working behind a deprecation warning. I'd start with FSDP and Megatron. For tests: unit tests that each recipe expands to exactly the fields you'd write by hand, negative tests for bad combos, and BF16/FP8 rollout runs. I can only run BF16 and FP8 on my card, so MXFP8/NVFP4 would be unit tests only.@0z5a's parity work adds a
scale_fmtcheck, so I'd keep this to the config side and reuse those format definitions instead of adding a second set. #8132's runtime layer audit is separate from this.A few things I'd like your take on before starting: where the block should live (
actor_rollout_ref.low_precisionor top level), whether format names should follow TE/Megatron recipe names, and whether something like BF16 train + FP8 rollout should only warn or need an explicit opt-in. Happy to write a short RFC first if that's easier.wuxibin89 commentedon Oct 19, 2026
@0z5a Thanks for your contribution. For the item 1, I have submit a PR #8287 to cover weight refit on fp8/mxfp8/mxfp4/nvfp4. For following work, we're going to provide more e2e verified training recipe to users. There're two ongoing works on mxfp8 #7519 and nvfp4 #7971, you can start with either of these recipes below, maybe mxfp4 is good start:
cr-gao commentedon Oct 19, 2026
@0z5a For batch-invariant kernels and true on-policy I have opened PR #8025 and #8026, welcome any comments on them and any follow-ups.
VikingDeng commentedon Oct 21, 2026
Looks like #8287 already covers most of what I proposed here (structured
rollout.quantization, one QAT path), so I'll drop this item. Thanks @wuxibin89, nice refactor. If there's a follow-up you'd want help with, like adding the BF16-train / FP8-rollout mismatch numbers to the QAT guide or a CPU test for invalid combos, I'm happy to pick it up.amenomori commentedon Oct 22, 2026
Hi @wuxibin89 @ISEEKYAN, I'd like to pick up P1 low-precision comm, starting with VeOmni’s node-local ReduceScatter.
VeOmni #1167 is already included in verl’s pinned dependency, but the configuration isn’t wired through yet. I’d add opt-in support and validate correctness and performance on two GPUs.
Is this scope available for me to take?
[veomni] feat: expose low-precision ReduceScatter communication (#8330)
7 remaining items
wuxibin89 commentedon Nov 11, 2026
@LOGO127 Sure, as typical rollout setting is temperature=1.0, top_p=1, top_k=-1.
Megatron support has been merged in #8199, so we should verify on both fsdp and megatron.
[veomni] feat: expose low-precision ReduceScatter communication (verl…
[megatron, ci] fix: honor module ignore patterns in ModelOpt QAT (ver…
[megatron, doc] fix: keep bridge layer specs and the causal mask unde…
WentaoXu03 commentedon Nov 11, 2026
Hi @wuxibin89, I'd like to pick up the P1 "low-precision inference-only forwards (ref / teacher / old_log_prob)" item under Low Precision Training. I ran a first microbenchmark to ground the design discussion — happy to reshape the scope based on your feedback.
Setup: single RTX 4090 (Ada, SM89), torch 2.7.1 + torchao 0.12, tensorwise float8 (
torch._scaled_mmverified working on SM89; rowwise is SM90-only, so left as an H100 follow-up). Models: Qwen3-0.6B / 1.7B / 8B-Base, seq 1024, logprob-style full-sequence forward without KV cache (matching the shape of verl'sinfer_batchpath), on gsm8k prompts + model greedy continuations.Results at teacher scale (Qwen3-8B-Base, batch 4 × 1024 tokens):
Fidelity vs BF16 on non-pad tokens: mean |Δlogprob| 0.02–0.10, KL 0.011–0.038 at 8B (larger models are quantized more gracefully than small ones here). Two caveats from the same data: small models are overhead-bound (0.6B is 1.6–3× slower under FP8), so I'd gate this by model size / memory pressure rather than enabling unconditionally; and weight-only FP8 (dequant at compute) saves resident bytes but not peak, and is uniformly slower — only interesting for sleep/offload-style residency.
Proposed integration point (mirroring the existing
rollout.quantizationmachinery from #8287): aref.quantizationoption that quantizes the ref worker's engine with torchao before the FSDP wrap, so the FSDP2 all-gather produces FP8 shards directly — which I understand aligns with the "torchao low-precision compute + FSDP2 all-gather" note in the P1 row. I checked the open PRs (#7186, #4435 are rollout-side; #8610 is teacher fleet scheduling) — I believe this trainer-side item is still unclaimed, but please correct me if I missed someone.Three design questions before I write an RFC / draft PR:
old_log_prob(the actor's own eval-mode forward) share the same path, or ref/teacher only?Next step on my side: a Qwen3-0.6B GRPO e2e run on the same 4090 with an FP8 ref forward (local prototype of the hook above) to validate training-loop integration and end-to-end memory savings.
WentaoXu03 commentedon Nov 11, 2026
Update to my comment above — ran the e2e validation in the meantime (Qwen3-0.6B GRPO, gsm8k, 8 steps, single RTX 4090, verl main + a prototype
ref.quantizationhook; BF16 vs FP8-dynamic tensorwise ref forward):CPUOffloadPolicyfor forward-only engines (transformer_impl.py:457), so a quantized ref should be measured against CPU offload (wake/PCIe cost, teachers too big to offload cleanly), not a GPU-resident BF16 ref._init_sharded_param(aten::new_zerosunimplemented), quantize-after-wrap loses the DTensor interface the all-gather needs. The prototype skips the wrap for the forward-only ref engine (validated on 1 GPU); multi-GPU FP8 shards (torchtitan-style float8 all-gather) is the main open question for the RFC.wuxibin89 commentedon Nov 11, 2026
@WentaoXu03 We're not going to support low precision on fsdp backend, it will be mainly support in megatron and veomni.
WentaoXu03 commentedon Nov 11, 2026
Thanks for the pointer
Two follow-ups so I aim this correctly:
veomni/ops/qatfp8 blockwise, or TE fp8) for the trainer-side ref / old_log_prob forward, or is there a separate plan? Happy to move the prototype to whichever backend this should live in.aten::new_zerosunimplemented in_init_sharded_param; after: the DTensor interface is lost for the all-gather). Not sure whetherveomni/ops/qatalready handles this — if not, worth knowing.If the item is already covered by the megatron/veomni plans — I'll drop the claim.
wuxibin89 commentedon Nov 12, 2026
WentaoXu03 commentedon Nov 12, 2026
Thanks — dropping the claim.
Zhikaiiii commentedon Nov 16, 2026
@wuxibin89 Hello, it looks like replaying the rollout sampling mask is currently tied to score centering #8199. Would verl consider making sampling-mask replay available independently, so standard GRPO can also compute log-probs over the same token set used by the rollout sampler? This would be useful when rollouts use truncation such as top_p or top_k.
wuxibin89 commentedon Nov 16, 2026
@Zhikaiiii Score centering with top_p/top_k truncation is already supported in #8196.
Zhikaiiii commentedon Nov 16, 2026
Yes, #8196 already supports this when score centering is enabled. But standard GRPO still lacks this option. Are there any plans to make it available for GRPO with score centering disabled?
tongyx361 commentedon Nov 16, 2026
Thanks for clarifying. Supporting this independently of score centering is a reasonable request. There are a few points to distinguish:
The optimization target matters. If we intend to optimize the policy with a constant top-p/top-k truncation, the trainer should account for masking and renormalization when computing log-probs. Using the original, untruncated distribution gives a different objective.
Stale data complicates masking. The trainer’s current policy can produce a different top-p/top-k mask from the behavior policy. Replaying the old mask therefore does not generally represent the current truncated policy. To optimize that policy, we should apply the masking and renormalization operator to the current logits, analogous to applying the quantization operator during quantization-aware training (QAT).
Mixed temperatures and masks leave the objective open. When different temperatures or masking configurations are used across tokens or samples, there is no established standard for which policy the trainer should optimize. The intended objective needs to be specified explicitly; replaying the sampler’s transformations is not automatically the correct choice.
SC has a specific motivation for reusing the behavior mask. SC already uses a top-k approximation. When the sampler itself truncates the distribution, its mask allows exact computation over that sampling support. This provides a rationale within SC, but does not automatically justify the same treatment in standard GRPO.
For standard GRPO, our usual setup uses untruncated sampling (
top_p=1,top_k=-1), with temperature scaling already handled by the trainer. Supporting truncation consistently outside SC would be a useful extension, but we currently consider it a lower priority and do not have a concrete timeline. Could you share the more details motivating top-p/top-k truncation in your training setup?Zhikaiiii commentedon Nov 17, 2026
Thanks for the detailed explanation. I had the first approach in mind: replaying the exact token set retained at each rollout step and renormalizing the current actor’s log-probs over that same set. The goal is to compute the GRPO ratio on the same token support as the behavior policy when rollouts use top-p or top-k truncation. DeepSeek‑V3.2 describes this as “Keep Sampling Mask,” and MiMo‑V2.6 reports using the same approach in its GRPO training. DeepSeek‑V3.2 §3.1 · MiMo‑V2.6 §6.4