🚀 [2026/03/09] Recent Updates Summary for ROLL Project

[PanAndy]

大家好！感谢大家对ROLL的关注。
ROLL近期更新了大量新功能，以下是近期更新的一些梳理，我们将持续对ROLL进行迭代更新，欢迎加入ROLL的社区。
#366

🚀亮点:

• rollout 重构为由router调度，支持sglang-router
• 新增[On-Policy Distillation](docs_roll/i18n/zh-Hans/docusaurus-plugin-content-docs/current/User Guides/Pipeline/on_policy_distill_pipeline_start.md)训练支持
• 支持Qwen3.5 Dense / MoE 系列模型

🚀主要新特性：

• Rollout
  • 重构router调度支持
    • sglang strategy重构，同时支持engine、server两种模式。
    • Scheduler重构(rlvr的DynamicScheduler/ agentic的Rolloutscheduler)，统一由Router提供调度
    • 迁移原LoadBalancer、RequestScheduler为PromptAffinityRouter、EnvAffinityRouter
    • 新增支持sglang-router
• pipeline recipe
  • 新增On-Policy Distillation训练支持
• Models
  • 支持Qwen3.5 Dense/MoE系列模型
• docker
  • torch2.10 、vllm 0.16.0 nightly、vllm0.15.1版本 、mcore 0.16.0
• bug fix:
  • 默认设置vllm VLLM_USE_FLASHINFER_SAMPLER=0 for torch 280，解决reponse重复度过高
  • fix sglang & vllm 偶现port conflict
  • fix sglang multi-nodes fail when infer_dp > 1
  • fix reward worker metrics 透出能力
  • fix model download get_node_ip cache，可能导致死锁timeout
  • fix CPU offload时FSDP2 DCE save
  • fix FSDP2 model initialization casting

[PanAndy]

Hello everyone! Thank you for your interest in ROLL.
ROLL has recently received a large set of new features. Below is a summary of the latest updates. We will continue iterating on ROLL—welcome to join the ROLL community.
#366

🚀 Highlights

• Rollout has been refactored to be scheduled by a router, with support for sglang-router.
• Added training support for [On-Policy Distillation](docs_roll/i18n/zh-Hans/docusaurus-plugin-content-docs/current/User Guides/Pipeline/on_policy_distill_pipeline_start.md).
• Added support for the Qwen3.5 model family: Dense / MoE.

🚀 Major New Features

• Rollout
  • Router scheduling refactor
    • Refactored the sglang strategy to support both engine and server modes.
    • Refactored schedulers (rlvr DynamicScheduler / agentic RolloutScheduler) so that scheduling is now uniformly provided by the Router.
    • Migrated the original LoadBalancer and RequestScheduler to PromptAffinityRouter and EnvAffinityRouter.
    • Added support for sglang-router.
• Pipeline recipes
  • Added On-Policy Distillation training support.
• Models
  • Added support for Qwen3.5 Dense/MoE series models.
• Docker
  • Updated images/environments: torch 2.10, vLLM 0.16.0 nightly, vLLM 0.15.1, mcore 0.16.0.
• Bug fixes
  • Set VLLM_USE_FLASHINFER_SAMPLER=0 by default for vLLM on Torch 2.8.0 to mitigate overly repetitive responses.
  • Fixed occasional port conflicts between sglang and vLLM.
  • Fixed sglang multi-node failures when infer_dp > 1.
  • Fixed reward worker metrics exposure.
  • Fixed a get_node_ip cache issue in model download that could cause deadlock/timeouts.
  • Fixed FSDP2 DCE save when CPU offloading is enabled.
  • Fixed casting during FSDP2 model initialization.

[zzfoutofspace]

Nice work of the OPD implementation!

I ran the experiment via the suggested script, where Qwen3-32B as the teacher and Qwen3-4B as the student. However, the reward curve does not increase. Is it due to that the two model's performances are close on the selected dataset?

Could you share the training curve of the OPD showcase experiment? Also, I wonder what is the most proper setting to evaluate the effectiveness of OPD?

[Schnabel-8]

Nice work of the OPD implementation!

I ran the experiment via the suggested script, where Qwen3-32B as the teacher and Qwen3-4B as the student. However, the reward curve does not increase. Is it due to that the two model's performances are close on the selected dataset?

Could you share the training curve of the OPD showcase experiment? Also, I wonder what is the most proper setting to evaluate the effectiveness of OPD?

Thank you for the feedback. Below is the configuration used in our previous experiments:

================================================================================

exp_name: qwen3-8B-onpolicy-distill-dapo
output_dir: ./output/
logging_dir: ./output/logs
rollout_dump_dir: ./output/rollout_dump
seed: 42
system_envs:
USE_MODELSCOPE: '1'
VLLM_USE_FLASHINFER_SAMPLER: '0'
checkpoint_config:
type: file_system
output_dir: /data/cpfs_0/rl_examples/models/qwen3-8B-onpolicy-distill-dapo
track_with: tensorboard
num_nodes: 1
num_gpus_per_node: 8
max_steps: 1024
save_steps: 100
logging_steps: 1
eval_steps: 10
resume_from_checkpoint: false
rollout_batch_size: 16
prompt_length: 2048
response_length: 8192
num_return_sequences_in_group: 4
ppo_epochs: 1
adv_estimator: reinforce
pretrain: Qwen/Qwen3-8B
reward_pretrain: Qwen/Qwen3-8B
global_template: qwen3
validation:
data_args:
file_name:
- data/math_benchmarks.jsonl
generating_args:
max_new_tokens: 8192
top_p: 0.6
top_k: 50
num_beams: 1
temperature: 0.6
num_return_sequences: 1
student_train:
model_args:
model_name_or_path: Qwen/Qwen3-8B
attn_implementation: fa2
disable_gradient_checkpointing: false
dtype: bf16
model_type: null
training_args:
learning_rate: 5.0e-07
weight_decay: 0.1
per_device_train_batch_size: 8
gradient_accumulation_steps: 1
warmup_steps: 20
num_train_epochs: 50
data_args:
file_name:
- data/dapo_math_17k_boxed.jsonl
domain_interleave_probs:
math_rule: 1.0
dataset_dir: data
messages: messages
interleave_probs: '1.0'
preprocessing_num_workers: 16
strategy_args:
strategy_name: megatron_train
strategy_config:
tensor_model_parallel_size: 1
context_parallel_size: 1
pipeline_model_parallel_size: 2
expert_model_parallel_size: 1
use_distributed_optimizer: true
recompute_granularity: full
use_sequence_packing: true
sequence_packing_args:
algorithm: load_balance
max_packed_sequence_length_forward: 10240
min_num_micro_batches_forward: 2
max_packed_sequence_length_train: 10240
min_num_micro_batches_train: 2
device_mapping: list(range(0,16))
infer_batch_size: 4
student_infer:
model_args:
model_name_or_path: Qwen/Qwen3-8B
disable_gradient_checkpointing: true
dtype: bf16
generating_args:
max_new_tokens: 8192
top_p: 0.99
top_k: 100
num_beams: 1
temperature: 0.99
num_return_sequences: 4
strategy_args:
strategy_name: vllm
strategy_config:
gpu_memory_utilization: 0.8
block_size: 16
load_format: auto
device_mapping: list(range(0,16))
infer_batch_size: 1
teacher:
model_args:
model_name_or_path: Qwen/Qwen3-32B
disable_gradient_checkpointing: true
dtype: bf16
model_type: null
strategy_args:
strategy_name: megatron_infer
strategy_config:
tensor_model_parallel_size: 2
pipeline_model_parallel_size: 4
context_parallel_size: 1
bf16: true
device_mapping: list(range(0,16))
infer_batch_size: 1
sequence_packing_args:
algorithm: load_balance
max_packed_sequence_length_forward: 10240
min_num_micro_batches_forward: 4
rewards:
math_rule:
worker_cls: roll.pipeline.rlvr.rewards.math_rule_reward_worker.MathRuleRewardWorker
model_args:
model_name_or_path: Qwen/Qwen3-8B
data_args:
template: qwen3
tag_included:
- math_dapo
world_size: 8
infer_batch_size: 1

================================================================================

The dataset follows the format below:

{"id": "9a9b6eb4-a1cb-49d1-8c1e-62eaf2f74079", "source": "math_dapo", "prompt": "In triangle $ABC$, $\\sin \\angle A = \\frac{4}{5}$ and $\\angle A < 90^\\circ$. Let $D$ be a point outside triangle $ABC$ such that $\\angle BAD = \\angle DAC$ and $\\angle BDC = 90^\\circ$. Suppose that $AD = 1$ and that $\\frac{BD}{CD} = \\frac{3}{2}$. If $AB + AC$ can be expressed in the form $\\frac{a\\sqrt{b}}{c}$ where $a, b, c$ are pairwise relatively prime integers, find $a + b + c$.", "messages": "[{\"role\": \"system\", \"content\": \"Please reason step by step, and put your final answer within \\\\boxed{}.\"}, {\"role\": \"user\", \"content\": \"In triangle $ABC$, $\\\\sin \\\\angle A = \\\\frac{4}{5}$ and $\\\\angle A < 90^\\\\circ$. Let $D$ be a point outside triangle $ABC$ such that $\\\\angle BAD = \\\\angle DAC$ and $\\\\angle BDC = 90^\\\\circ$. Suppose that $AD = 1$ and that $\\\\frac{BD}{CD} = \\\\frac{3}{2}$. If $AB + AC$ can be expressed in the form $\\\\frac{a\\\\sqrt{b}}{c}$ where $a, b, c$ are pairwise relatively prime integers, find $a + b + c$.\\n\\nRemember to put your answer within \\\\boxed{}.\"}]", "ground_truth": "34", "tag": "math_dapo"}

We processed the dataset based on open-r1/DAPO-Math-17k-Processed by prepending the instruction "Please reason step by step, and put your final answer within \\boxed{}." to the prompt and appending "Remember to put your answer within \\boxed{}." at the end. These modifications ensure that the outputs can be correctly parsed by the MathRewardWorker.

The validation curves observed during the experiment are as follows:

The advantages curves are as follows: