NVIDIA Megatron Core 0.13.0rc4

Prerelease: NVIDIA Megatron Core 0.13.0rc4 (2025-07-22)

NVIDIA Megatron Core 0.13.0rc3

Prerelease: NVIDIA Megatron Core 0.13.0rc3 (2025-07-17)

NVIDIA Megatron Core 0.14.0rc1

Prerelease: NVIDIA Megatron Core 0.14.0rc1 (2025-07-14)

NVIDIA Megatron Core 0.14.0rc0

Prerelease: NVIDIA Megatron Core 0.14.0rc0 (2025-07-07)

NVIDIA Megatron Core 0.13.0rc2

Prerelease: NVIDIA Megatron Core 0.13.0rc2 (2025-07-02)

NVIDIA Megatron Core 0.13.0rc1

Prerelease: NVIDIA Megatron Core 0.13.0rc1 (2025-07-03)

NVIDIA Megatron Core 0.13.0rc0

Prerelease: NVIDIA Megatron Core 0.13.0rc0 (2025-06-27)

NVIDIA Megatron Core 0.12.2

Merge branch 'completions_unit_test_fix' into 'core_r0.12.0'

Fixes for completions endpoint unit test

See merge request ADLR/megatron-lm!3445

NVIDIA Megatron Core 0.12.1

Merge branch 'gaod/llama4/te_fix' into 'core_r0.12.0'

Fix the TE assertion for release

See merge request ADLR/megatron-lm!3340

NVIDIA Megatron Core 0.12.0

• Add FP8 recipe selection to arguments (--fp8-recipe, --first-last-layers-bf16, --num-layers-at-start-in-bf16, --num-layers-at-end-in-bf16)
• Context parallel: fix loss scaling when calculate_per_token_loss=True
• Make the number of data parallel communication buckets configurable (--ddp-num-buckets, --ddp-pad-buckets-for-high-nccl-busbw)
• Inference
  • Support in-flight batching and chunked KV cache
  • Reduce memory usage,
    • by not materializing full attention mask
    • by only materializing logits for the last token during decode
    • by removing an obsolete tensor reference
• Hybrid Model
  • Inference
    • Add CUDA graph support
    • Change tools/run_mamba_text_generation_server.py to use megatron.core.inference
    • Fix a shape issue when materializing logits for Mamba model
  • Improve initialization of Mamba layers
  • Add configuration switches (--mamba-state-dim, --mamba-head-dim, --mamba-num-groups, --is-hybrid-model)
  • Make num_floating_point_operations work with hybrid model
  • Make hybrid_conversion.py work with mixer that uses TE linear
  • Add FP8 support
  • Fix Mamba dt_bias tensor parallelism
  • Support multimodal tokenizer
  • Improve data parallelism scaling
• MoE
  • Features:
    • DeepEP support, compatible with all the parallelisms and token drop / dropless
    • Important precision improvement: Enable FP32/FP64 routing and unpermutation using –moe-router-dtype. FP32 is recommended for all fine-grained MoE training
    • CUDA Graph support for MoE
    • Multi-Token Prediction (MTP) Support
    • Fused indices_to_multihot kernel for DeepEP dispatcher
  • Bug fixes:
    • Fix Hang Issue with MoE+Dense Hybrid models
    • Update theoretical memory and tflops estimation for MoE and MLA
    • Fix MoE Aux loss scaling for per token loss
    • Fixes for group limited routing and expert bias. We verified these fixes through dsv3 e2e verifications
  • Known issues:
    • The ckpt trained with Custom FSDP for MoE may not be compatible with 3D parallel training.