[roadmap] verl Q1 roadmap

[wuxibin89]

Model Engine

• Switch default to new model engine
  • Mark legacy engine as deprecated
  • Feature parity new and legacy model engine: LoRA/PEFT, etc [megatron] feat: Share actor and ref in LoRA #4673, [worker] fix: new engine saves megatron LoRA adapters checkpoints #4866, [worker] feat: New engine share actor and ref for LoRA #4867

Megatron

• Perfomance optimization
  • Megatron dynamic CP [BREAKING][megatron] feat: support dynamic CP #5057
  • MoE multi-modal model training
  • Long context training: fine-grained activation recompuation/offload

VeOmni

• RL support: [veomni, trainer] feat: add rl support for veomni backend #4882

TorchTitan #5306

Rollout Engine

• Improve rollout server profiling: [perf] feat: verl profiler system support Agent Loop scenario and integrate torch.profiler #4320
• New rollout engine: TensorRT-LLM [ray,rollout,trtllm] feat: Adding tensorrt_llm as new rollout engine #4665
  • [roadmap] verl TensorRT-LLM rollout roadmap #5042
• Separate vllm worker from trainer, sync by cuda ipc [BREAKING][worker, rollout, vllm] feat: implement vLLM colocated training-inference rollout with process separation #4280
• Router reply
  • vllm: [algo] feat: support router replay #4101, [Feature] Support recording expert indices for rollout router replay vllm-project/vllm#28284
  • sglang: [algo, rollout, sglang] feat: Support router replay with sglang #4840
• AgentLoop
  • Refactor tool definition and registration
  • Support multiple AgentLoopOutput for one sample: prompt switch, context compression, multi-agent, etc.

Checkpoint Engine

• Add checkpoint engine abstract interface [ckpt] feat: add checkpoint-engine abstraction #4775
• Add NCCL, NIXL transport backend and more [ckpt] feat: add Hccl ckpt engine backend #4885 [ckpt] feat: add kimi ckpt engine backend #4954
• Add checkpoint engine manager [ckpt] feat: add CheckpointEngineManager #5031

Trainer

• Online policy distillation:
  • fsdp: [fsdp,vllm,trainer,algo] feat: On-Policy Distillation #4897
  • megatron: [megatron,trainer,algo] feat: On-Policy Distillation  #5041
• New sync trainer with TransferQueue
  • RFC: [RFC] PPOTrainer with TransferQueue Integration #5400
  • New sync trainer with TransferQueue: [trainer] feat: add new trainer with TranferQueue #5401
• Fully async trainer
  • Refactor one-step-off/fully async with model engine and checkpoint engine [fsdp, megatron] feat: refactor fully-async and one-step-off training to support multiple checkpoint engine backends #5029
  • Remove PartialAgentLoop [rollout] feat: support auto resume on abort in FullyAsyncLLMServerManager #5430
  • Standalone megatron worker group to recompute old_log_prob

Speculative Decoding

• Support MTP SFT/RL training [megatron] feat: Using MTP in RL Training and Inference #4936 [megatron] feat: Support MTP training in SFT  #4981
• [rfc]:add speculator training scripts and checkpoint support #4947

Ascend NPU

#4537

Model Support List

#5389

[moehanabi]

Online policy distillation

maybe http://github.com/volcengine/verl/pull/3975 ?

[wuxibin89]

Online policy distillation

maybe http://github.com/volcengine/verl/pull/3975 ?

Will support it officially in main.

[flishwang]

I noticed that Qwen3-Next-80B-A3B GRPO/DAPO training is supported by megatron backend in 0.7.0
Will it be supported by fsdp?

[JacobHelwig]

I would like to work on on-policy distillation #4897

[JacobHelwig]

I would like to work on on-policy distillation #4897

Plan for OPD is here.

[zhihaocheng]

hi @JacobHelwig, could you please share some official learning curves and results for reproduction of OPD? Do you have tested the algorithm on different models?

[JacobHelwig]

hi @JacobHelwig, could you please share some official learning curves and results for reproduction of OPD? Do you have tested the algorithm on different models?

Sure, please see the PR description.

[KurodaKanbei]

Hi, I am wondering what Support MTP RL training means. Does it refer to accelerating the rollout by speculative decoding, or post-training the MTP module?

[zw0610]

The current design and implementation of Transfer Queue does not fit into the main branch of veRL. More generally speaking, Transfer Queue should be something integrated with Ray instead of with veRL.

Let's postpone the related todo item util further discussion and decision.