Skip to content

Releases: NVIDIA-NeMo/RL

Release v0.3.1

01 Aug 22:16
@terrykong terrykong
v0.3.1
27d0af9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Release v0.3.1 Latest
Latest

🚀 Release v0.3.1

This is a patch release to fix a bug with sequence-packing and a memory optimization in the softmax.

Please read about the major features in v0.3 on the release notes.

What's Changed

  • cp: fix: Enforce minimum packed sequence bin count and multiple of bin count (748) into r0.3.1 by @chtruong814 in #759
  • cp: feat: avoid softmax deepcopy in logprobs (tpoisonooo) (761) into r0.3.1 by @chtruong814 in #763
  • chore: 0.3.0 -> 0.3.1 by @terrykong in #825

Full Changelog: v0.3.0...v0.3.1

Contributors

terrykong and chtruong814
Assets 2
Loading

Release 0.3.0

25 Jul 17:31
@terrykong terrykong
v0.3.0
b1c86a8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Release 0.3.0

🚀 Release v0.3.0

📝 Blog

Our latest blog post shares highlights and progress from recent work—take a look!

✨ Highlights

🏗️ Improved Training Throughput and Scalability via Megatron-Core Backend

In addition to PyT DTensor backend to seamlessly support 🤗HuggingFace models, this release has added Megatron-Core backend("Megatron backend") to enable large scale dense and MoE model training. It includes efficient parallelisms (data, tensor, pipeline, context, expert and sequence) and distributed optimizers for efficient training, and is our recommendation for RL on large model sizes and compute scale.

To use the Megatron backend, ensure you have initialized the submodules of NeMo RL:

git submodule update --init --recursive

You can try out the Megatron backend using predefined configs:

# Example 1 GPU 
uv run examples/run_grpo_math.py --config=examples/configs/grpo_math_1B_megatron.yaml

Or by enabling it from the command line:

# Example 1 GPU
uv run examples/run_sft.py policy.megatron_cfg.enabled=True

To learn more about the different backends and their configuration, visit our documentation on Traning Backends.

For FAQ using the Megatron backend, see this section.

⚡ Context Parallelism and Sequence Packing

Users can now train with longer sequences at enhanced GPU utilization via Context Parallelism ("CP") and Sequence Packing support for both Megatron-Core and PyT DTensor backends.

For the Megatron backend, both Context Parallelism and sequence packing can be enabled together:

policy:
  megatron_cfg:
    context_parallel_size: 2
  sequence_packing:
    enabled: True

DTensor backend also supports CP and Sequence Packing, but cannot be used together yet. Progress on this feature is being tracked here #520. In addition, there is also a known issue of CP with sequence parallelism, tracked here #659. For more information about CP and some of its current limitations in the Dtensor backend, visit our documentation.

policy:
  dtensor_cfg:
    context_parallel_size: 2
  # CP and sequence packing cannot be used together (To enable sequence packing, set context_parallel_size=1)
  sequence_packing:
     enabled: False

We recommend sequence packing to avoid extra padding and accelerate your training run, but if your model cannot use sequence packing (e.g., due to unsupported attention kernel), we recommend using dynamic_batching instead (see config). Dynamic batching is mutually exclusive with sequence packing, so it should be enabled on its own.

For more details on sequence packing and dynamic batching and how to use it, refer to our design documentation.

💎 Expanded Model Support

🔹 Qwen3 Support

Full support for Qwen3 model family with optimized configurations is available on the Megatron backend.

Qwen3 dense variants and the smallest MoE variant (Qwen/Qwen3-30B-A3B) is also available in the DTensor backend. If you need full N-d parallelism and the largest scale, we recommend the Megatron backend.

🔹 DeepSeekV3 Support

DeepSeekV3 (671B) is now supported on the Megatron backend. See #591 for more details on how to launch. We are continuing to optimize performance on DeepSeekV3 and other large MoE models, which we hope to land in our next release.

🚀 Async VLLM Engine

We have added Async VLLM Engine (v1) support in v0.3 which enables two important features not possible before:

  • Multi-node VLLM rollouts (for large MoEs like DSV3)
  • Pipeline Parallelism

Async engine can be enabled with the following config change:

  generation:
    backend: "vllm"
    vllm_cfg:
      async_engine: true

With Async VLLM Engine enabled, multi-turn rollouts are now much faster since we no longer block at each turn to wait for all the batch elements to complete.

📍 Non-colocated Generation ("Split Placement")

NeMo RL now supports placing the training backend on a different set of GPUs than the generation backend. This is currently supported in the DTensor backend, with support in the Megatron backend coming soon #613

This feature can be useful if:

  • training and generation have incompatible parallelism/world sizes
  • the memory after offloading for training or generation is still not low enough

Non-colcated generation can be enabled with the following config changes:

  generation:
    backend: "vllm"
    colocated:
      # true: generation shares training GPUs
      # false: uses dedicated generation resources
      enabled: false
      # only relevant when enabled is false
      resources:
        gpus_per_node: null # Decides num gpus to be dedicated to generation when there is one node in the cluster i.e cluster.num_nodes == 1
        num_nodes: 1 # Decides number of nodes to be dedicated to generation

An example multi-node command is:

# 5 nodes with 8GPUs, 4 nodes for train and 1 node for inference
uv run python examples/run_grpo_math.py \
    policy.generation.colocated.enabled=false \
    policy.generation.colocated.resources.num_nodes=1 \
    cluster.num_nodes=5 \
    cluster.gpus_per_node=8

Non-colocated generation is also an important prerequisite for our continued work on Async RL.

📊 MLFlow Integration for Experiment Tracking

NeMo RL now supports MLFlow integration for comprehensive experiment tracking and management. This extends our suite of loggers which included Tensorboard and Wandb.

Enable MLFlow tracking in your configuration:

logger:
  mlflow_enabled: true
  mlflow:
    experiment_name: "grpo-dev"
    run_name: "grpo-dev-logger"

⚡ Performance Optimizations

🚀 Refit Optimizations

Multiple improvements to the refit process (weight updates from training to generation backend) lead to a several fold speedup. In large MoE models this has a significant effect on E2E step time. We have measured these optimizations on DeepSeekV3 which has brought down refit time from 850 seconds to 51 seconds (16x improvement). The improvements are particularly beneficial for extra-large models with large TP sizes in vLLM.

The core engineering team is planning on sharing some of the insights and optimization techniques used to achieve this in a blog; so stay tuned!

📊 VLLM CUDA Graphs

In v0.3 we now enable CUDA Graphs in VLLM by default

🚫 FSDP1 Deprecation

NeMo RL has officially removed the original FSDP1 path used for multi-gpu multi-node training in pure Pytorch. For training in pure Pytorch without the Megatron backend, we now recommend using the DTensor path which uses FSDP2 by default and is strictly better in terms of functionality and performance.

For more information on the deprecation and the burn testing done before its removal see #614.

🛠️ Known Issues

  • There is a known convergence issue with the deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B base model used in the DeepScaleR recipe when cuda graphs are enabled in vLLM. This is fixed on main with #857. This divergence has not been observed in other models.
  • Qwen 32B needs minimum 32 nodes on DTensor backend #656
    • We have observed increased memory pressure on Qwen 32B requiring a much higher node count than expected.
    • This bug does not seem to affect the model on the Megatron backend.
  • Qwen/Qwen3-30B-A3B needs 8 nodes to run causing overheads due to extra parallelisms
  • Llama 70B performance on long context (>=32k) is slower than expected on the Megatron-Core backend.
  • DeepSeekV3 and Qwen3-253B E2E performance is still WIP, with improvements targed for v0.4
  • On the Megatron backend, training with MoE models using EP>1 and DP>1 can
    hang if sequence packing is enabled #718.
    • Our recommendation if you are using EP>1 and DP>1 is to disable sequence packing.
  • DPO does not support sequence packing or dynamic-batching #719.

📊 Release Runs

We have provided Tensorboard logs to release runs to give you a head start on what to expect from our recipes.

To view these Tensorboard logs easily, we've provided a Google Collab to download and serve the Tensorboard logs.

What's Changed

  • fix: update the comment about why we init in fp32 by @parthchadha in #354
  • feat: add and log a very rough entropy approximation by @SahilJain314 in #342
  • fix: fix issues preventing running grpo on volta by @parthchadha in #294
  • docs: remove license that was erroneously copy-pasted by @terrykong in #357
  • fix: recipes missing args by @terrykong in #365
  • test: make dpo functional test threshold higher until flakiness resolved by @terrykong in #371
  • ci: Migrate ...
Read more

Contributors

pcmoritz, yfw, and 26 other contributors
Loading

Release v0.2.1

15 May 22:03
@terrykong terrykong
v0.2.1
81d421f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Release v0.2.1

🚀 Release v0.2.1

🎉 Official Open Source Release!

We are thrilled to announce that NeMo RL is now officially open source! We welcome the community to use and contribute to it to help shape the future of reinforcement learning.

✨ Highlights

🎯 DeepScaleR Reproducer in NeMo RL

This release features a reproducer for the DeepScaleR work by Agentica AI, where a 1.5B parameter model surpassed O1-Preview on the AIME benchmark (Pass@1). Our implementation replicates this by iteratively scaling DeepSeek's GRPO algorithm from 8K → 16K → 24K context lengths.

You can start the first stage of training (8K context window) using the following command:

uv run examples/run_grpo_math.py --config=examples/configs/grpo-deepscaler-1.5b-8K.yaml

For the complete 3-stage iterative training instructions and more details, please see our GRPO on DeepScaleR guide.

📐 OpenMathInstruct-2 SFT in NeMo RL

This release includes a Supervised Fine-Tuning (SFT) recipe that follows the OpenMathInstruct-2 paper. Using this recipe, training a Llama-3.1-8B model on the train_1M split of the nvidia/OpenMathInstruct-2 dataset achieves a score of 0.5020 on the MATH-500 benchmark, matching the reference implementation in NeMo-Skills.

You can run the OpenMathInstruct-2 recipe using the following command:

uv run examples/run_sft.py --config=examples/configs/sft_openmathinstruct2.yaml

For more details on dataset splits, training times, and evaluation, please see our SFT on OpenMathInstruct-2 guide.

⚡ Faster GRPO with Dynamic Batching

GRPO E2E performance has been significantly improved with the introduction of dynamic batching. This feature optimizes GPU utilization by sorting variable-length responses by sequence length and bucketing them into microbatches. These microbatches aim to have a total number of tokens close to train_mb_tokens and logprob_mb_tokens for the training and logprob stages, respectively.

Important: Dynamic batching requires dtensor to be enabled.

You can enable dynamic batching and dtensor in your YAML configuration like so:

policy:
  # Enable DTensor (required for dynamic batching)
  dtensor_cfg:
    enabled: True
    # Other dtensor settings like tensor_parallel_size, sequence_parallel, etc.
    # tensor_parallel_size: 1
    # sequence_parallel: False
    # activation_checkpointing: True

  # Dynamic batching settings
  dynamic_batching:
    enabled: True
    # Target number of tokens for training microbatches
    train_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.train_micro_batch_size}}
    # Target number of tokens for logprob microbatches
    logprob_mb_tokens: ${mul:${policy.max_total_sequence_length}, ${policy.logprob_batch_size}}
    # Round sequence lengths to the nearest multiple of this value for bucketing
    sequence_length_round: 64
  # Other policy settings like max_total_sequence_length, train_micro_batch_size, etc.
  # max_total_sequence_length: 4096
  # train_micro_batch_size: 4
  # logprob_batch_size: 8

Alternatively, you can enable these features and configure them via command-line overrides when running a script (e.g., run_grpo_math.py):

uv run examples/run_grpo_math.py \
  --config=<your_base_config.yaml> \
  policy.dtensor_cfg.enabled=True \
  policy.dynamic_batching.enabled=True \
  # Optionally override other dynamic batching or dtensor parameters:
  # policy.dynamic_batching.train_mb_tokens=16384 \
  # policy.dynamic_batching.logprob_mb_tokens=32768 \
  # policy.dtensor_cfg.tensor_parallel_size=2

Make sure to adjust train_mb_tokens, logprob_mb_tokens, and other parameters according to your sequence length and batch size configuration.

💎 Broad Model Support (including Gemma3)

NeMo RL enables users to leverage powerful open models from families such as Qwen, Llama, and Gemma for reinforcement learning. For this v0.2.1 release, we've enhanced support, particularly for Gemma3 models, addressing their unique characteristics like tied weights across all model sizes (which require special handling for tensor parallelism) and specific vLLM initialization needs. NeMo RL automatically handles these model quirks to ensure seamless training and inference. For more details on this, please see our Model Quirks guide.

🛠️ Bug Fixes

  • Gradient Accumulation: Resolved a common issue where naive averaging of losses during gradient accumulation, especially with varying sequence lengths, led to inaccurate loss calculations; this fix (see #266) ensures training runs and loss calculations are performed accurately.

📊 Release Runs

We have provided Tensorboard logs to release runs to give you a head start on what to expect from our recipes.

To view these Tensorboard logs easily, we've provided a Google Collab to download and serve the Tensorboard logs.

What's Changed

Read more

Contributors

yfw, terrykong, and 12 other contributors
Loading

Release v0.2.0

24 Apr 16:38
@terrykong terrykong
v0.2.0
26f651e

Choose a tag to compare

View all tags
Release v0.2.0

🚀 Release v0.2.0

⚙️ Advanced Parallelism — FSDP 2, TP & SP for Efficient Training

The headline feature of v0.2 is the new DTensorPolicyWorker.

It enables advanced parallelisms—FSDP 2, Tensor Parallelism, and Sequence Parallelism—letting us scale to 32 B-parameter models.

Enable it via YAML or CLI overrides:

policy.dtensor_cfg.enabled=True \
policy.dtensor_cfg.tensor_parallel_size=8 \
policy.dtensor_cfg.sequence_parallel=True \
policy.dtensor_cfg.activation_checkpointing=True

🧠 Learning Algorithms — DPO (Direct Preference Optimization)

Our algorithm suite now includes DPO, compatible with both FSDP 1 and DTensor.

uv run examples/run_dpo.py

More examples live in the docs.

🔄 Multi-Turn RL — Tool Use, Games & Beyond

We now support multi-turn generation and training with GRPO.

An E2E example of training to play a sliding puzzle game will be available in the next release, but you can try it by cherry-picking this PR: #242

# 8x80GB GPUs recommended
uv run python examples/run_grpo_sliding_puzzle.py

🏋️‍♂️ Large-Model Support — Native PyTorch up to 32 B @ 16k sequence length

FSDP 2 + TP + SP make RL and SFT on 32 B models possible:

uv run ./examples/run_grpo_math.py \
  --config examples/configs/grpo_math_8B.yaml \
  policy.model_name='Qwen/Qwen2.5-32B' \
  policy.generation.vllm_cfg.tensor_parallel_size=4 \
  policy.max_total_sequence_length=16384 \
  cluster.num_nodes=16 \
  policy.dtensor_cfg.enabled=True \
  policy.dtensor_cfg.tensor_parallel_size=8 \
  policy.dtensor_cfg.sequence_parallel=True \
  policy.dtensor_cfg.activation_checkpointing=True

Full multi-node walkthrough in the docs.

🛡️ Environment Isolation — Per-Worker Deps with uv

In NeMo RL, workers can now launch cached, isolated uv virtual environments with their own Python dependencies—a setup we’ve found to be significantly faster than Ray’s builtin conda/pip/uv flow. Details here.

🐞 Known Issues

  • FSDP 1 gradient-clipping bug — see #251
  • Qwen 32 B perf tweaks coming in the next patch
  • Gemma3 convergence: #236
  • SFT/DPO configs default to FSDP1, which is not recommended for 1B models with tied embeddings. #256. Enabling DTensor manually will resolve the error.
  • V100 configuration: #259
  • The default SFT and DPO configs in examples/configs set policy.dtensor_cfg.enabled=False, but dtensor must be enabled to run with the default 1B models. Please make sure to set policy.dtensor_cfg.enabled=True when running with the default SFT and DPO configs.

📊 Release Runs

We have provided tensorboard logs to release runs to give you a head start on what to expect from our recipes.

You may download them here and serve them with tensorboard:

mkdir v0.2.0
tar -xzf release_runs.tar.gz -C v0.2.0/
tensorboard serve --logdir v0.2.0/

🚧 Coming soon… : In future releases, we will share a tensorboard viewer to make it easier to view and compare release runs.

What's Changed

  • fix: ray.sub race condition when overlapping srun commands on same node by @terrykong in #39
  • feat: add gpu mem and util logging to wandb/tensorboard by @terrykong in #37
  • ci: tests now run with HF_DATASETS_CACHE to speed up e2e time by @terrykong in #41
  • fix: update the instructions for multi-node setup; change the title f… by @parthchadha in #78
  • fix: Mixed Prec memory improvements and better default configs (converge-able) by @SahilJain314 in #32
  • fix: Remove reference of tokenizer from generation backend (#75) by @parthchadha in #82
  • feat: unit test metric tracking by @terrykong in #40
  • fix: unit test error when coverage wasn't specified by @terrykong in #88
  • ci: temporarily disable CI on main since PRs must be up to date before merge by @terrykong in #91
  • fix: error out early if ray cluster does not have resources by @parthchadha in #89
  • ci: skip functional until more capacity available and/or tests speed up by @terrykong in #94
  • feat: evaluation implement by @yuki-666 in #16
  • fix: gradient should be averaged instead of summed across mbs by @parthchadha in #86
  • fix: Use separate step_metric for GPU Monitoring by @yfw in #92
  • feat: Update sft config to use single GPU by @ashors1 in #90
  • fix: Grammar nit by @SahilJain314 in #98
  • feat: add capability to set min/max eps separately as proposed in the… by @parthchadha in #95
  • fix: change format messages to out of place by @KiddoZhu in #77
  • fix: correct version and use setuptools.dynamic metadata for version/readme by @terrykong in #104
  • fix: remove usage of vllm to get device uuid and instead use nvidia-m… by @parthchadha in #105
  • fix: Change optional-dependencies to dependency-groups by @hemildesai in #81
  • feat: Add support for hydra style overrides by @hemildesai in #80
  • fix: Do not initialize reference model for sft by @ashors1 in #71
  • fix: change grpo default to use 64 prompts per step and 32 generation… by @parthchadha in #111
  • feat: use cuda_graph by default for vllm by @parthchadha in #116
  • fix: ensure that we check for pad_token and not assume pad_token==eos… by @parthchadha in #120
  • ci: Consolidate tests by @chtruong814 in #27
  • feat: support local venvs for dependency isolation by @terrykong in #102
  • fix: make message formatting compatible with tokenizers with no bos/eos token by @ashors1 in #118
  • fix: reset prefix cache when sleep is called to ensure prefix cache i… by @parthchadha in #112
  • ci: Fix unit test summary by @chtruong814 in #128
  • fix: fix error padding by @yuki-666 in #87
  • feat: Distributed checkpointing by @ashors1 in #99
  • ci: Add DCO placeholder check for merge queue by @chtruong814 in #147
  • ci: Clarify DCO check in merge_group by @chtruong814 in #154
  • fix: host ip resolution uses ray vs socket by @terrykong in #153
  • test: Add grpo/reinforce/ppo loss tests (prep for incoming vocab parallel changes) by @SahilJain314 in #162
  • fix: always test vllm by @parthchadha in #167
  • docs: Fix doc build warnings and add external CI config by @mckimn in #157
  • fix: allow configuring ray ports in ray.sub in case conflict on cluster by @terrykong in #173
  • feat: support arbitrary end_strings by @yuki-666 in #96
  • ci: labels for docs/L0/L1/L2 and run even if only doc test by @terrykong in #181
  • fix: don't use cuda-graphs for vllm generation by @parthchadha in #187
  • ci: Update to include public/ folder for pages deployment by @mckimn in #182
  • docs: run tests with --group test to avoid missing test deps by @terrykong in #188
  • fix: default to less verbose logging + uv-venv log once per worker by @terrykong in #141
  • docs: Correcting file names by @aschilling-nv in #161
  • fix: convert DCP to HF script works without ray cluster by @terrykong in #185
  • docs: remove backticks from uv.md title by @terrykong in #179
  • feat: add a unique seed for each vllm llm engine by @parthchadha in #171
  • fix: unit test script halts on first failure by @terrykong in #189
  • feat: Upgrade to vllm v1 runtime by @parthchadha in #170
  • ci: R...
Read more

Contributors

yfw, mckimn, and 11 other contributors
Loading

Release v0.1.1

25 Mar 16:28
@terrykong terrykong
v0.1.1
fbe85c5
This commit was signed with the committer’s verified signature.
terrykong Terry Kong
SSH Key Fingerprint: Yh2/WWVxWJfxIsmdBbjYY6iAsM2vMJngjvex6lS65IU
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Release v0.1.1

Release v0.1.1

Patch release on top of v0.1.0

🛠️ More stable mixed precision configurations and resolves OOMs observed in llama 8b
🛠️ Fixes race condition in ray.sub where pyxis can fail if subsequent srun commands are run too early (with --overlap)

What's Changed

  • fix: ray.sub race condition when overlapping srun commands on same node by @terrykong in #39
  • feat: add gpu mem and util logging to wandb/tensorboard by @terrykong in #37
  • ci: tests now run with HF_DATASETS_CACHE to speed up e2e time by @terrykong in #41
  • fix: update the instructions for multi-node setup; change the title f… by @parthchadha in #78
  • fix: Mixed Prec memory improvements and better default configs (converge-able) by @SahilJain314 in #32

Known Issues

  • gpu memory and utilization in wandb/tensorboard has a bug when enabled. This is tracked in #83

Full Changelog: v0.1.0...v0.1.1

Contributors

terrykong, parthchadha, and SahilJain314
Loading

Release v0.1.0

22 Mar 00:07
@terrykong terrykong
v0.1.0
9ee564e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Release v0.1.0

Release v0.1.0

  • ✅ Fast Generation - vLLM backend for optimized inference
  • ✅ HuggingFace Integration - Works with 1-8B models (Qwen1.5, Llama)
  • ✅ Distributed Training - FSDP support and Ray-based infrastructure
  • ✅ Environment Support - Support for multi-environment training.
  • ✅ Learning Algorithms - GRPO (Group Relative Policy Optimization) and SFT (Supervised Fine-Tuning)
  • ✅ Worker Isolation - Process isolation between RL Actors (no worries about global state)

What's Changed

New Contributors

Known Issues

  • There is a known bug with SFT checkpointing that requires the full model to be gathered on GPU before saving a checkpoint. This causes OOM for larger model sizes. If you run into OOM when checkpointing, disable checkpointing by adding checkpointing.enabled=False to your run command.

Full Changelog: https://github.com/NVIDIA/NeMo-RL/commits/v0.1.0

Contributors

yfw, terrykong, and 4 other contributors
Loading