Skip to content

[roadmap] verl 26Q4 roadmap #8186

Open
Open

Description

@wuxibin89

Model Engine

Low Precision Training

Background

Goals in RL

  • Cheaper rollout and inference-only forwards (ref, teacher, old_log_prob): less memory, less memory traffic, faster GEMM.
  • Lower train/rollout mismatch once rollout is quantized: training should see the same quantization as sampling.

Two routes

Route Training compute Training speedup Key issues
BF16 + QAT BF16, fake quant (quantize-dequantize + STE) on weight / act / KV in the rollout format no quantizer parity with rollout kernels; QAT noise raises gradient variance
Real low-precision compute FP8 / MXFP8 / MXFP4 / NVFP4 GEMMs (e.g. TE), FP32 master weights yes rowwise/columnwise copies per format; low-precision FSDP all-gather; train/rollout format alignment

Where FP32 master weights live decides the backend design:

  • In the optimizer (Megatron): model holds BF16 or low-precision compute weights, cast once per optimizer step on the
    local shard; easy to keep low-precision params persistent.
  • In the model (FSDP2 / VeOmni): model holds sharded FP32 params, cast to compute dtype at every all-gather; any optimizer
    works, high-precision weights always available (checkpoint, weight sync), cast cost repeats per micro-batch.

Formats

Format Block Scale fprop / dgrad see the same W HW
FP8 per-tensor tensor FP32 yes Hopper+
FP8 blockwise A 1x128, W 128x128 FP32 or pow2 yes for W (2D block) Hopper
MXFP8 1x32 E8M0 no (rowwise and columnwise quantized separately) Blackwell
MXFP4 1x32 E8M0 no Blackwell (Hopper via FP4->FP8 dequant)
NVFP4 1x16 + per-tensor FP32 E4M3 yes when weight uses 16x16 2D (TE default) Blackwell

DeepSeek V4 / V4.1 reference

  • MXFP4 QAT for routed experts and the indexer QK path; FP32 master -> FP4 -> lossless dequant to FP8 128x128, so
    training reuses the FP8 framework and dgrad uses the same W as fprop (V4 §3.4).
  • Real FP4 weights for rollout, ref and teacher forwards (V4 §5.2.1).
  • KV cache: FP8 + BF16 RoPE dims (V4 §2.3.4); FP4 main KV cache QAT, SWA KV stays FP8 (V4.1 §2.4.4).
  • Comm: BF16 + stochastic rounding for MoE grad sync, all-to-all + local FP32 sum (V4 §3.5.1).
  • Batch-invariant / deterministic kernels for bitwise train/infer alignment (V4 §3.3).

Existing

BF16 + QAT

NVFP4 QAT (Quantization-Aware Training) in verl

Backend QAT format (training) Rollout format Implementation Verified
FSDP NVFP4 W4A16 (group 16) NVFP4 W4A16, vLLM compressed-tensors verl/utils/qat/ Qwen3-8B-Base, Qwen3-30B-A3B-Base
Megatron NVFP4 W4A16 (group 16) NVFP4 W4A16, vLLM modelopt verl/utils/modelopt/ Qwen3-8B-Base, Qwen3-30B-A3B-Base
VeOmni FP8 blockwise (W 128x128, A 1x128; linear / KV NoPE / indexer QK / MoE) + FP4 1x32 routed experts FP8 blockwise + MXFP4 experts, vLLM Mxfp4MoEMethod refit veomni/ops/qat/, verl/utils/vllm/vllm_fp4_utils.py DeepSeek-V4-Flash

Real low-precision compute

FP8 RL in verl

Backend Mode Training compute Rollout compute Implementation Verified
FSDP rollout-only BF16 FP8 blockwise, vLLM / SGLang / TRT-LLM {vllm,sglang}/*fp8_utils.py Qwen3-8B-Base, Qwen3-30B-A3B-Base
Megatron rollout-only BF16 FP8 blockwise, vLLM / SGLang / TRT-LLM same as above Qwen3-30B-A3B
Megatron E2E FP8 blockwise via TE (fp8: hybrid, fp8_recipe: blockwise) + FP8 optimizer FP8 blockwise, vLLM Megatron-Bridge + TE Qwen3-8B, Qwen3-30B-A3B (Hopper); on Blackwell TE runs blockwise as MXFP8 (#6172)
VeOmni - - - - not supported

Q4 Plan

Priority Common Megatron VeOmni
P0 train/rollout quantizer parity DeepSeek-V4 FP4->FP8 lossless path DeepSeek-V4/V4.1 QAT E2E
P1 unified recipe config; low-precision inference-only forwards; low-precision comm Blackwell MXFP8 E2E; Blackwell NVFP4 training torchao low-precision compute + FSDP2 all-gather; torchao NVFP4/MXFP4 QAT
P2 batch-invariant / deterministic kernels FP8 optimizer states; MoE W4A8 Muon + BF16 Newton-Schulz

Common @wuxibin89

Megatron @ISEEKYAN @wuxibin89

Route: Megatron-LM + TransformerEngine, master weights in the optimizer.

VeOmni @wuxibin89 @FoolPlayer @amenomori

Route: FSDP2 + torchao, master weights in the model.

  • Model holds sharded FP32 params; FSDP2 MixedPrecisionPolicy casts each shard to BF16 at all-gather; any standard
    optimizer works unchanged.

  • QAT is VeOmni's own fake-quant ops (veomni/ops/qat/, STE) wired into patchgen-generated HF modeling, which makes
    new models (e.g. DeepSeek-V4) quick to bring up.

  • Real low-precision compute is to come from torchao: float8 / MX linear, plus float8 FSDP2 all-gather that quantizes
    the high-precision shard before gather (low-precision comm without persistent low-precision weights).

  • Fit: HF-native models, fast model bring-up, flexible recipes. Cost: no low-precision compute yet; current TileLang
    QAT kernels are SM90 + BF16 only.

  • P0 DeepSeek-V4/V4.1 QAT E2E (blocks Model Support: DeepSeek-V4.1-Flash)

  • P1 torchao low-precision compute: float8 (tensorwise/rowwise), MXFP8 on Blackwell

    • low-precision FSDP2 all-gather: quantize sharded high-precision weights before gather (float8 / MXFP8)
  • P1 torchao fake-quant QAT for NVFP4 / MXFP4

Model Support

  • DeepSeek-V4.1-Flash
  • Qwen3.8-Flash-Next
  • GLM-5.3-Flash

Score Centering

Rollout

Async rollout

Weight refit optimization

Performance optimization

Fault tolerance

Target on v1 separate_async mode: @HaoWang113 @ETOgaosion @tongyx361

Uni-Agent

Uni-Agent Q4 roadmap

Accelerator support

We will keep enhancing multi-accelerator support: built-in for NVIDIA / Ascend / AMD, and out-of-tree for others via
verl-hardware-plugin.

Huawei Ascend

Huawei Ascend NPU Q4 roadmap: Huawei Ascend verl Q4 roadmap

AMD ROCm

Google TPU

Trainer

verl-omni

verl-omni Q4 roadmap: verl-project/verl-omni#700

rl-insight

rl-insight Q4 roadmap:verl-project/rl-insight#191

Activity

  1. pinned this issue on Oct 10, 2026
  2. cr-gao commented on Oct 11, 2026

    @cr-gao
    Contributor

    Hi @wuxibin89 , I'd like to work on "Token-level interrupt + persist KV cache / routing across weight switch, no re-prefill on resume" under the async rollout part.
    Btw for batch-invariant rollout, #8025 and #8026 might be related. As they are initially opend for qwen3-omni true on-policy training on verl-omni side, I can later align it with a verl native model. Thanks for any feedback.

  3. SamitHuang commented on Oct 12, 2026

    @SamitHuang
    Contributor

    It'd be better to link VeRL-Omni Q4 roadmap as well verl-project/verl-omni#700

  4. 0z5a commented on Oct 17, 2026

    @0z5a
    Contributor

    Hi @wuxibin89 @ISEEKYAN, I'd like to take two items under Model Engine → Low Precision Training → Common.

    1. [P0] Train/rollout quantizer parity

    • One reference quantizer per format (FP8 blockwise with fp32/ue8m0 scales, NVFP4 e4m3 + fp32 global, MXFP4/MXFP8 ue8m0), shared by training fake-quant and rollout weight export, with bitwise tests against the weights vLLM actually loads.
    • Explicit scale_fmt, with a fail-fast check when train and rollout formats disagree.
    • A per-format report of mismatch KL and rollout throughput.

    Starting points I found on main:

    • The FP8 blockwise Triton and PyTorch paths in verl/utils/kernel/fp8_kernel.py compute scales differently (amax/448 vs. 1/(448/amax)), so they are not bitwise equal to each other.
    • NVFP4 QAT fake-quant (verl/utils/qat/linear.py) and the exporter (verl/utils/qat/quantizer.py) compute the e4m3 block scale in a different op order and handle zero scales differently.
    • TE blockwise defaults to power-of-two scales, while the rollout-only export emits fp32 scales.

    2. [P1] Batch-invariant / deterministic train+rollout kernels

    Two questions before I start:

    1. Is anyone already working on the quantizer parity item?
    2. Should the consistency switch fold into the P1 unified recipe config, or stay separate?

    I'll open a draft PR with the parity harness and the FP8 blockwise tests first.

  5. VikingDeng commented on Oct 18, 2026

    @VikingDeng

    Hi @wuxibin89 @ISEEKYAN, I'd like to pick up the P1 "unified low-precision recipe config" item under Common.

    Right now the same thing is configured in six places: rollout.quantization, fsdp_config.qat, megatron.qat, megatron.override_transformer_config.fp8*, VeOmni's qat_implementation and Automodel's enable_fp8. The FSDP and Megatron YAMLs even give QATEngineConfig different defaults (ignore_patterns, activation_observer), and nothing checks that train and rollout formats agree. I ran into this with Qwen3-0.6B GRPO on GSM8K (one RTX PRO 6000, main @ 8ced5d2). With BF16 training and rollout.quantization=fp8, training/rollout_probs_diff_mean goes from ~0.0058 to ~0.031, and there's no warning.

    What I have in mind is one typed low_precision block (train format, QAT, rollout format) that just expands into the existing backend fields. It would fail fast on combinations that don't make sense, and the old keys would keep working behind a deprecation warning. I'd start with FSDP and Megatron. For tests: unit tests that each recipe expands to exactly the fields you'd write by hand, negative tests for bad combos, and BF16/FP8 rollout runs. I can only run BF16 and FP8 on my card, so MXFP8/NVFP4 would be unit tests only.

    @0z5a's parity work adds a scale_fmt check, so I'd keep this to the config side and reuse those format definitions instead of adding a second set. #8132's runtime layer audit is separate from this.

    A few things I'd like your take on before starting: where the block should live (actor_rollout_ref.low_precision or top level), whether format names should follow TE/Megatron recipe names, and whether something like BF16 train + FP8 rollout should only warn or need an explicit opt-in. Happy to write a short RFC first if that's easier.

  6. wuxibin89 commented on Oct 19, 2026

    @wuxibin89
    CollaboratorAuthor

    Hi @wuxibin89 @ISEEKYAN, I'd like to take two items under Model Engine → Low Precision Training → Common.

    1. [P0] Train/rollout quantizer parity

    • One reference quantizer per format (FP8 blockwise with fp32/ue8m0 scales, NVFP4 e4m3 + fp32 global, MXFP4/MXFP8 ue8m0), shared by training fake-quant and rollout weight export, with bitwise tests against the weights vLLM actually loads.
    • Explicit scale_fmt, with a fail-fast check when train and rollout formats disagree.
    • A per-format report of mismatch KL and rollout throughput.

    Starting points I found on main:

    • The FP8 blockwise Triton and PyTorch paths in verl/utils/kernel/fp8_kernel.py compute scales differently (amax/448 vs. 1/(448/amax)), so they are not bitwise equal to each other.
    • NVFP4 QAT fake-quant (verl/utils/qat/linear.py) and the exporter (verl/utils/qat/quantizer.py) compute the e4m3 block scale in a different op order and handle zero scales differently.
    • TE blockwise defaults to power-of-two scales, while the rollout-only export emits fp32 scales.

    2. [P1] Batch-invariant / deterministic train+rollout kernels

    Two questions before I start:

    1. Is anyone already working on the quantizer parity item?
    2. Should the consistency switch fold into the P1 unified recipe config, or stay separate?

    I'll open a draft PR with the parity harness and the FP8 blockwise tests first.

    @0z5a Thanks for your contribution. For the item 1, I have submit a PR #8287 to cover weight refit on fp8/mxfp8/mxfp4/nvfp4. For following work, we're going to provide more e2e verified training recipe to users. There're two ongoing works on mxfp8 #7519 and nvfp4 #7971, you can start with either of these recipes below, maybe mxfp4 is good start:

  7. cr-gao commented on Oct 19, 2026

    @cr-gao
    Contributor

    Hi @wuxibin89 @ISEEKYAN, I'd like to take two items under Model Engine → Low Precision Training → Common.

    1. [P0] Train/rollout quantizer parity

    • One reference quantizer per format (FP8 blockwise with fp32/ue8m0 scales, NVFP4 e4m3 + fp32 global, MXFP4/MXFP8 ue8m0), shared by training fake-quant and rollout weight export, with bitwise tests against the weights vLLM actually loads.
    • Explicit scale_fmt, with a fail-fast check when train and rollout formats disagree.
    • A per-format report of mismatch KL and rollout throughput.

    Starting points I found on main:

    • The FP8 blockwise Triton and PyTorch paths in verl/utils/kernel/fp8_kernel.py compute scales differently (amax/448 vs. 1/(448/amax)), so they are not bitwise equal to each other.
    • NVFP4 QAT fake-quant (verl/utils/qat/linear.py) and the exporter (verl/utils/qat/quantizer.py) compute the e4m3 block scale in a different op order and handle zero scales differently.
    • TE blockwise defaults to power-of-two scales, while the rollout-only export emits fp32 scales.

    2. [P1] Batch-invariant / deterministic train+rollout kernels

    Two questions before I start:

    1. Is anyone already working on the quantizer parity item?
    2. Should the consistency switch fold into the P1 unified recipe config, or stay separate?

    I'll open a draft PR with the parity harness and the FP8 blockwise tests first.

    @0z5a For batch-invariant kernels and true on-policy I have opened PR #8025 and #8026, welcome any comments on them and any follow-ups.

  8. VikingDeng commented on Oct 21, 2026

    @VikingDeng

    Looks like #8287 already covers most of what I proposed here (structured rollout.quantization, one QAT path), so I'll drop this item. Thanks @wuxibin89, nice refactor. If there's a follow-up you'd want help with, like adding the BF16-train / FP8-rollout mismatch numbers to the QAT guide or a CPU test for invalid combos, I'm happy to pick it up.

  9. amenomori commented on Oct 22, 2026

    @amenomori
    Contributor

    Hi @wuxibin89 @ISEEKYAN, I'd like to pick up P1 low-precision comm, starting with VeOmni’s node-local ReduceScatter.

    VeOmni #1167 is already included in verl’s pinned dependency, but the configuration isn’t wired through yet. I’d add opt-in support and validate correctness and performance on two GPUs.

    Is this scope available for me to take?

  10. 7 remaining items

  11. wuxibin89 commented on Nov 11, 2026

    @wuxibin89
    CollaboratorAuthor

    Would a first PR limited to untruncated stochastic sampling (temperature > 0, top_p=1, top_k=-1, min_p=0) be welcome?

    @LOGO127 Sure, as typical rollout setting is temperature=1.0, top_p=1, top_k=-1.

    Does this split fit the roadmap, and should the first PR also include Megatron validation?

    Megatron support has been merged in #8199, so we should verify on both fsdp and megatron.

  12. WentaoXu03 commented on Nov 11, 2026

    @WentaoXu03

    Hi @wuxibin89, I'd like to pick up the P1 "low-precision inference-only forwards (ref / teacher / old_log_prob)" item under Low Precision Training. I ran a first microbenchmark to ground the design discussion — happy to reshape the scope based on your feedback.

    Setup: single RTX 4090 (Ada, SM89), torch 2.7.1 + torchao 0.12, tensorwise float8 (torch._scaled_mm verified working on SM89; rowwise is SM90-only, so left as an H100 follow-up). Models: Qwen3-0.6B / 1.7B / 8B-Base, seq 1024, logprob-style full-sequence forward without KV cache (matching the shape of verl's infer_batch path), on gsm8k prompts + model greedy continuations.

    Results at teacher scale (Qwen3-8B-Base, batch 4 × 1024 tokens):

    BF16 FP8 dyn-act tensorwise Δ
    peak allocated 17.6 GiB 10.5 GiB −40%
    resident weights 15.3 GiB 8.2 GiB −46%
    median latency 514 ms 469 ms −8.8% (batch 2: −17%)

    Fidelity vs BF16 on non-pad tokens: mean |Δlogprob| 0.02–0.10, KL 0.011–0.038 at 8B (larger models are quantized more gracefully than small ones here). Two caveats from the same data: small models are overhead-bound (0.6B is 1.6–3× slower under FP8), so I'd gate this by model size / memory pressure rather than enabling unconditionally; and weight-only FP8 (dequant at compute) saves resident bytes but not peak, and is uniformly slower — only interesting for sleep/offload-style residency.

    Proposed integration point (mirroring the existing rollout.quantization machinery from #8287): a ref.quantization option that quantizes the ref worker's engine with torchao before the FSDP wrap, so the FSDP2 all-gather produces FP8 shards directly — which I understand aligns with the "torchao low-precision compute + FSDP2 all-gather" note in the P1 row. I checked the open PRs (#7186, #4435 are rollout-side; #8610 is teacher fleet scheduling) — I believe this trainer-side item is still unclaimed, but please correct me if I missed someone.

    Three design questions before I write an RFC / draft PR:

    1. Persistent quantized copy vs quantize-after-all-gather for the ref model?
    2. Should old_log_prob (the actor's own eval-mode forward) share the same path, or ref/teacher only?
    3. Is teacher-forward distillation in scope for a first PR, or better as a follow-up?

    Next step on my side: a Qwen3-0.6B GRPO e2e run on the same 4090 with an FP8 ref forward (local prototype of the hook above) to validate training-loop integration and end-to-end memory savings.

  13. WentaoXu03 commented on Nov 11, 2026

    @WentaoXu03

    Hi @wuxibin89, I'd like to pick up the P1 "low-precision inference-only forwards (ref / teacher / old_log_prob)" item under Low Precision Training. I ran a first microbenchmark to ground the design discussion — happy to reshape the scope based on your feedback.

    Setup: single RTX 4090 (Ada, SM89), torch 2.7.1 + torchao 0.12, tensorwise float8 (torch._scaled_mm verified working on SM89; rowwise is SM90-only, so left as an H100 follow-up). Models: Qwen3-0.6B / 1.7B / 8B-Base, seq 1024, logprob-style full-sequence forward without KV cache (matching the shape of verl's infer_batch path), on gsm8k prompts + model greedy continuations.

    Results at teacher scale (Qwen3-8B-Base, batch 4 × 1024 tokens):

    BF16 FP8 dyn-act tensorwise Δ
    peak allocated 17.6 GiB 10.5 GiB −40%
    resident weights 15.3 GiB 8.2 GiB −46%
    median latency 514 ms 469 ms −8.8% (batch 2: −17%)
    Fidelity vs BF16 on non-pad tokens: mean |Δlogprob| 0.02–0.10, KL 0.011–0.038 at 8B (larger models are quantized more gracefully than small ones here). Two caveats from the same data: small models are overhead-bound (0.6B is 1.6–3× slower under FP8), so I'd gate this by model size / memory pressure rather than enabling unconditionally; and weight-only FP8 (dequant at compute) saves resident bytes but not peak, and is uniformly slower — only interesting for sleep/offload-style residency.

    Proposed integration point (mirroring the existing rollout.quantization machinery from #8287): a ref.quantization option that quantizes the ref worker's engine with torchao before the FSDP wrap, so the FSDP2 all-gather produces FP8 shards directly — which I understand aligns with the "torchao low-precision compute + FSDP2 all-gather" note in the P1 row. I checked the open PRs (#7186, #4435 are rollout-side; #8610 is teacher fleet scheduling) — I believe this trainer-side item is still unclaimed, but please correct me if I missed someone.

    Three design questions before I write an RFC / draft PR:

    1. Persistent quantized copy vs quantize-after-all-gather for the ref model?
    2. Should old_log_prob (the actor's own eval-mode forward) share the same path, or ref/teacher only?
    3. Is teacher-forward distillation in scope for a first PR, or better as a follow-up?

    Next step on my side: a Qwen3-0.6B GRPO e2e run on the same 4090 with an FP8 ref forward (local prototype of the hook above) to validate training-loop integration and end-to-end memory savings.

    Update to my comment above — ran the e2e validation in the meantime (Qwen3-0.6B GRPO, gsm8k, 8 steps, single RTX 4090, verl main + a prototype ref.quantization hook; BF16 vs FP8-dynamic tensorwise ref forward):

    • Both arms run through without OOM, step latency parity (43.5s vs 44.3s); actor-vs-ref kl_loss rises 0.0008 → 0.016 from quantization noise, other training metrics stay healthy.
    • Worth knowing: verl forces CPUOffloadPolicy for forward-only engines (transformer_impl.py:457), so a quantized ref should be measured against CPU offload (wake/PCIe cost, teachers too big to offload cleanly), not a GPU-resident BF16 ref.
    • torchao Float8Tensor doesn't compose with FSDP2 in either order: quantize-before-wrap fails in _init_sharded_param (aten::new_zeros unimplemented), quantize-after-wrap loses the DTensor interface the all-gather needs. The prototype skips the wrap for the forward-only ref engine (validated on 1 GPU); multi-GPU FP8 shards (torchtitan-style float8 all-gather) is the main open question for the RFC.
  14. wuxibin89 commented on Nov 11, 2026

    @wuxibin89
    CollaboratorAuthor

    @WentaoXu03 We're not going to support low precision on fsdp backend, it will be mainly support in megatron and veomni.

  15. WentaoXu03 commented on Nov 11, 2026

    @WentaoXu03

    @wuxibin89 commented:

    We're not going to support low precision on fsdp backend, it will be mainly support in megatron and veomni.

    Thanks for the pointer

    Two follow-ups so I aim this correctly:

    1. For the inference-only forwards item on megatron/veomni — is the expected shape to reuse the rollout-side quantization paths (e.g. veomni/ops/qat fp8 blockwise, or TE fp8) for the trainer-side ref / old_log_prob forward, or is there a separate plan? Happy to move the prototype to whichever backend this should live in.
    2. In case it's useful for the veomni side: quantizing torchao Float8Tensor params around the FSDP2 wrap fails in both orders (before the wrap: aten::new_zeros unimplemented in _init_sharded_param; after: the DTensor interface is lost for the all-gather). Not sure whether veomni/ops/qat already handles this — if not, worth knowing.

    If the item is already covered by the megatron/veomni plans — I'll drop the claim.

  16. wuxibin89 commented on Nov 12, 2026

    @wuxibin89
    CollaboratorAuthor

    @wuxibin89 commented:

    We're not going to support low precision on fsdp backend, it will be mainly support in megatron and veomni.

    Thanks for the pointer

    Two follow-ups so I aim this correctly:

    1. For the inference-only forwards item on megatron/veomni — is the expected shape to reuse the rollout-side quantization paths (e.g. veomni/ops/qat fp8 blockwise, or TE fp8) for the trainer-side ref / old_log_prob forward, or is there a separate plan? Happy to move the prototype to whichever backend this should live in.
    2. In case it's useful for the veomni side: quantizing torchao Float8Tensor params around the FSDP2 wrap fails in both orders (before the wrap: aten::new_zeros unimplemented in _init_sharded_param; after: the DTensor interface is lost for the all-gather). Not sure whether veomni/ops/qat already handles this — if not, worth knowing.

    If the item is already covered by the megatron/veomni plans — I'll drop the claim.

    1. The inference-only forward on megatron/veomni should be same as actor train forward path, so there's no need to support inference-only forward separately.
    2. For veomni, please refer to [RFC][VeOmni] End-to-end low-precision training #8342
  17. WentaoXu03 commented on Nov 12, 2026

    @WentaoXu03

    @wuxibin89 commented:

    1. The inference-only forward on megatron/veomni should be same as actor train forward path, so there's no need to support inference-only forward separately.
    2. For veomni, please refer to [RFC][VeOmni] End-to-end low-precision training #8342

    Thanks — dropping the claim.

  18. Zhikaiiii commented on Nov 16, 2026

    @Zhikaiiii
    Contributor

    Would a first PR limited to untruncated stochastic sampling (temperature > 0, top_p=1, top_k=-1, min_p=0) be welcome?

    @LOGO127 Sure, as typical rollout setting is temperature=1.0, top_p=1, top_k=-1.

    Does this split fit the roadmap, and should the first PR also include Megatron validation?

    Megatron support has been merged in #8199, so we should verify on both fsdp and megatron.

    Would a first PR limited to untruncated stochastic sampling (temperature > 0, top_p=1, top_k=-1, min_p=0) be welcome?

    @LOGO127 Sure, as typical rollout setting is temperature=1.0, top_p=1, top_k=-1.

    Does this split fit the roadmap, and should the first PR also include Megatron validation?

    Megatron support has been merged in #8199, so we should verify on both fsdp and megatron.

    @wuxibin89 Hello, it looks like replaying the rollout sampling mask is currently tied to score centering #8199. Would verl consider making sampling-mask replay available independently, so standard GRPO can also compute log-probs over the same token set used by the rollout sampler? This would be useful when rollouts use truncation such as top_p or top_k.

  19. wuxibin89 commented on Nov 16, 2026

    @wuxibin89
    CollaboratorAuthor

    @Zhikaiiii Score centering with top_p/top_k truncation is already supported in #8196.

  20. Zhikaiiii commented on Nov 16, 2026

    @Zhikaiiii
    Contributor

    @Zhikaiiii Score centering with top_p/top_k truncation is already supported in #8196.

    Yes, #8196 already supports this when score centering is enabled. But standard GRPO still lacks this option. Are there any plans to make it available for GRPO with score centering disabled?

  21. tongyx361 commented on Nov 16, 2026

    @tongyx361
    Collaborator

    @Zhikaiiii Score centering with top_p/top_k truncation is already supported in #8196.

    Yes, #8196 already supports this when score centering is enabled. But standard GRPO still lacks this option. Are there any plans to make it available for GRPO with score centering disabled?

    Thanks for clarifying. Supporting this independently of score centering is a reasonable request. There are a few points to distinguish:

    1. The optimization target matters. If we intend to optimize the policy with a constant top-p/top-k truncation, the trainer should account for masking and renormalization when computing log-probs. Using the original, untruncated distribution gives a different objective.

    2. Stale data complicates masking. The trainer’s current policy can produce a different top-p/top-k mask from the behavior policy. Replaying the old mask therefore does not generally represent the current truncated policy. To optimize that policy, we should apply the masking and renormalization operator to the current logits, analogous to applying the quantization operator during quantization-aware training (QAT).

    3. Mixed temperatures and masks leave the objective open. When different temperatures or masking configurations are used across tokens or samples, there is no established standard for which policy the trainer should optimize. The intended objective needs to be specified explicitly; replaying the sampler’s transformations is not automatically the correct choice.

    4. SC has a specific motivation for reusing the behavior mask. SC already uses a top-k approximation. When the sampler itself truncates the distribution, its mask allows exact computation over that sampling support. This provides a rationale within SC, but does not automatically justify the same treatment in standard GRPO.

    For standard GRPO, our usual setup uses untruncated sampling (top_p=1, top_k=-1), with temperature scaling already handled by the trainer. Supporting truncation consistently outside SC would be a useful extension, but we currently consider it a lower priority and do not have a concrete timeline. Could you share the more details motivating top-p/top-k truncation in your training setup?

  22. Zhikaiiii commented on Nov 17, 2026

    @Zhikaiiii
    Contributor

    Replaying the old mask therefore does not generally represent the current truncated policy.

    — tongyx361

    Thanks for the detailed explanation. I had the first approach in mind: replaying the exact token set retained at each rollout step and renormalizing the current actor’s log-probs over that same set. The goal is to compute the GRPO ratio on the same token support as the behavior policy when rollouts use top-p or top-k truncation. DeepSeek‑V3.2 describes this as “Keep Sampling Mask,” and MiMo‑V2.6 reports using the same approach in its GRPO training. DeepSeek‑V3.2 §3.1 · MiMo‑V2.6 §6.4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions