NVIDIA Megatron Core 0.18.0

Changelog Details

• fix(ci): replace actions/setup-python with apt-get to avoid 429 rate limits by @ko3n1g :: PR: #4072
• ci: Fix package name for code-freeze workflow by @ko3n1g :: PR: #4077
• chore: bump _code_freeze workflow to v0.86.0 by @ko3n1g :: PR: #4078
• Fix checkpoint inspector by @janEbert :: PR: #4079
• Update docs to conform to NVIDIA style guides by @megnvidia :: PR: #4068
• Miscellaneous inference fixes by @santhnm2 :: PR: #4030
• fix fine_grained_callables with fused rmsnorm residual by @CarlosGomes98 :: PR: #4026
• [Main][feat] Support overlapping A2A Combine backprop with wgrad GEMM by @Wohox :: PR: #3795
• Modify mfsdp default data-parallel-sharding-strategy by @wplf :: PR: #3691
• Fix fsdp_dtensor conversion for pretrained-only checkpoints by @DAISY-gh :: PR: #3912
• Guard NVshmem issues by @wdykas :: PR: #4093
• m-fsdp: wire use_precision_aware_optimizer from ddp_config to ParamAn… by @rapatel :: PR: #4024
• Megatron-FSDP: Add MXFP8 transpose helper buffer for Hybrid FSDP by @shjwudp :: PR: #3918
• feat(fsdp): use TE general_gemm for mixed-precision wgrad in FSDP path by @Victarry :: PR: #3822
• Megatron-FSDP: Fix insufficient double buffers during gradient reduce by @shjwudp :: PR: #4054
• Fix M-FSDP MXFP8 related BUGs by @shjwudp :: PR: #3991
• Megatron-FSDP: Make _pre_forward_param_unshard and _register_post_backward_hook formal by @shjwudp :: PR: #4029
• FIX: Use decoupled gradients for precision-aware M-FSDP grad norm by @XueSongTap :: PR: #3746
• Align chat completions endpoint with vLLM by @santhnm2 :: PR: #4063
• [Megatron-FSDP] Fix compatibility with frozen parameters and add unit tests by @shjwudp :: PR: #3287
• [M-FSDP] Refactor uneven dtensor to full tensor and add UT by @shjwudp :: PR: #3190
• Add agent instruction files by @Phlip79 :: PR: #4102
• Bump eopt version by @skyw :: PR: #4100
• Refactor emerging optimizer integration by @skyw :: PR: #4113
• Fix over provisioning of Mamba state memory when max_requests is set by @santhnm2 :: PR: #4114
• base strategy simplification by @dimapihtar :: PR: #4001
• add support for DCP and FSDP async save by @dimapihtar :: PR: #4027
• Add more emerging optimizers (#3907) by @skyw :: PR: #4119
• Fix FSDP checkpoint conversion and loading for Qwen3.5-VL by @DAISY-gh :: PR: #3936
• docs: update mcore optimizer docstrings to google style by @Akshat8510 :: PR: #2799
• Set tensor-parallel attributes irrespective of perform_initialization by @ilml :: PR: #4084
• docs: add developer-guide skill with CI/CD and failure navigation guidance by @ko3n1g :: PR: #4035
• chore: Move skills by @ko3n1g :: PR: #4136
• ci: Let Claude react to comment by @ko3n1g :: PR: #4135
• Nemotron3 Super GB200 release config by @maanug-nv :: PR: #4118
• Enable CUDA graph for ADAM optimizer by @vasunvidia :: PR: #3429
• Claude review should recommend testing by @Phlip79 :: PR: #4137
• cleanup: remove unused scatter_gather_tensors_in_pipeline argument by @Phlip79 :: PR: #4140
• fix: Remove fail-fast (-x) and guard distributed teardown against deadlock by @ko3n1g :: PR: #4139
• Claude: add respond-to-issue skill by @Phlip79 :: PR: #4141
• Fix muon getter backward compatability by @skyw :: PR: #4157
• Audit of user guide by @megnvidia :: PR: #4098
• Fix RerunStateMachine crash (TypeError: 'NoneType' object is not subscriptable) by not saving a checkpoint after a transient NaN / Inf by @yezhengmao1 :: PR: #3981
• Preserve type of decorated methods/classes by @nschank :: PR: #4062
• update muon test case to use new interface by @skyw :: PR: #4163
• [M-FSDP] Fix Tensor Parallel mode detection by @shjwudp :: PR: #3191
• fix: remove weights_only=False for multimodal example by @faradawn :: PR: #4104
• Cudagraphs: Fix sequence packing segfault more generally by @mathemakitten :: PR: #4162
• Make MTP work with materialize_only_last_token_logits by @santhnm2 :: PR: #4166
• Add unit test for Mamba EP inference (eager fallback with mixed CUDA graphs) by @santhnm2 :: PR: #4085
• update docs in respect to async changes by @dimapihtar :: PR: #4177
• update checkpointing docs in respect to async changes by @dimapihtar :: PR: #4208
• chore: improve build-and-test skill with trigger rules and dependency workflow by @ko3n1g :: PR: #4199
• Fix layerwise optimizer with expt_dp_size=1 and contention with element-wise distributed optimizer by @skyw :: PR: #4138
• ci: add --cluster-a100/h100/gb200 args to trigger_internal_ci.py by @ko3n1g :: PR: #4195
• ci: Update golden values for nightly tests by @chtruong814 :: PR: #4215
• rename async_allgather to overlap_param_gather by @skyw :: PR: #4217
• Fix Slack sync for users with GitHub email privacy enabled by @Phlip79 :: PR: #4220
• Miscellaneous MTP inference fixes by @santhnm2 :: PR: #4191
• Move inference guards out of arguments.py by @mathemakitten :: PR: #4210
• Fix: enable fine-grained activation offloading for Mamba model. by @fanshiqing :: PR: #4173
• bump NVRx by @dimapihtar :: PR: #4178
• Update tokenizer args for Nemotron3 release config by @maanug-nv :: PR: #4239
• build: add dynamic git-versioning and drop rc0 pre-release tag by @ko3n1g :: PR: #4212
• Fix unnecessary permute padding for non-quantized MoE dispatch by @xiaoxi-wangfj :: PR: #4038
• Fix split state dict main by @kunlunl :: PR: #3676
• Add /split-pr Claude Code command for splitting PRs by CODEOWNERS by @Phlip79 :: PR: #4160
• Enable FP8 DPA for MXFP8 recipe by @vasunvidia :: PR: #4066
• Enable AG/RS overlap with explicit process group passing by @jeffnvidia :: PR: #3249
• Enable cpu_offloading with Full iteration CUDA graph by @vasunvidia :: PR: #3969
• Fix TransformerConfig validation for mixed dense/MoE upcycling by @rkteddy :: PR: #3647
• Remove cross-rank synchronization during checkpoint load & deprecate torch.distributed.checkpoint.state_dict_loader.load_state_dict by @asolergi-nv :: PR: #2864
• Fix incorrectly set decoupled_grad and DistOpt mechanics for MFSDP. by @cspades :: PR: #4133
• Refit Miscelaneous by @wdykas :: PR: #3973
• Add conditions_embeddings argument to TransformerBlock, TransformerLayer for DiT (diffusion transformer) by @huvunvidia :: PR: #4134
• Fix build_sequences_per_dataset output path arg usage by @DhineshPonnarasan :: PR: #4144
• ci: Flush pending CUDA work before the barrier in destroy_model_parallel by @chtruong814 :: PR: #4259
• Update oncall schedule by @Phlip79 :: PR: #4257
• docs(moe): Update MoE README by @sbhavani :: PR: #3664
• Revert "Add conditions_embeddings argument to TransformerBlock, Trans… by @ko3n1g :: PR: #4270
• reduce the number of shared expert streams by @yangbofun :: PR: #3752
• remove legacy Bert code by @dimapihtar :: PR: #4204
• [Main] Feat(moe): Gated delta net context parallel (CP) by @yuzhongw-nvidia :: PR: #2642
• remove t5 legacy code by @dimapihtar :: PR: #4203
• fix: handle list-typed process groups in ProcessGroupCollection.repr by @cluster2600 :: PR: #3753
• Fix Context Parallelism documentation link by @liangxs :: PR: #4149
• [MLA] fix: Pad V when Q/V head dims differ for THD by @HollowMan6 :: PR: #3003
• fix(megatron-fsdp): build expt_device_mesh only for MoE models by @xuwchen :: PR: #3831
• Allow the evaluation batch size to differ from the training batch size by @michal2409 :: PR: #4014
• Add @NVIDIA/transformer review group to megatron/core/transformer/ by @Phlip79 :: PR: #4281
• Reset AG_pipeline bucket status after validation step. by @vasunvidia :: PR: #3155
• Enhance and fix NVTX for training by @yaox12 :: PR: #3642
• NVFP4 native weights for DDP by @WanZzzzzz :: PR: #4005
• Remove unnecessary arguments for layerwise distributed optimizer by @FDecaYed :: PR: #4272
• reuse grad buffer for layer-wise param allgather by @FDecaYed :: PR: #3751
• feat(ci): add strict review mode to Claude review workflow by @Victarry :: PR: #4197
• Fix stale approvals by @Phlip79 :: PR: #4280
• [MoE] Add a new score function to the router by @yaox12 :: PR: #3673
• [MoE] Improvement of shared expert overlap, support shared expert overlap for FlexDispatcher by @Victarry :: PR: #2207
• build: bump DeepEP to 34152ae by @ko3n1g :: PR: #4228
• ci: mark test_fused_indexer_loss_gradient_tp_consistency as flaky_in_dev by @ko3n1g :: PR: #4299
• Fix typo in PR4133. by @cspades :: PR: #4277
• ci: add retry loop to apt-get update to handle transient mirror sync failures by @ko3n1g :: PR: #4209
• fix: enforce correct pass thresholds for deterministic and approximate tests by @ko3n1g :: PR: #4238
• remove legacy biencoder and realm models by @dimapihtar :: PR: #4205
• ci: add configurable launcher support for functional tests (ft_launcher / torchrun) by @ko3n1g :: PR: #4298
• chore: document --target main for local Docker builds by @ko3n1g :: PR: #4307
• Extract args init to launch scripts by @maanug-nv :: PR: #4225
• [Main] Fix TE version check for retain_pinned_cpu_buffers in cpu offload by @BestJuly :: PR: #4267
• Fix documented shape by @janEbert :: PR: #3486
• ci: add sync-skills workflow, rename CLAUDE.md → AGENTS.md, move .claude/skills → skills/ by @ko3n1g :: PR: #4303
• Get device correctly when module returns a dict instead of individual tensor by @shifangx :: PR: #4265
• remove vision legacy code by @dimapihtar :: PR: #4202
• feat: long convergence resiliency for release tests by @ko3n1g :: PR: #4335
• ci(action): improve GitHub Actions output UX by @ko3n1g :: PR: #4337
• build: bump TransformerEngine to release_v2.14 by @ko3n1g :: PR: #4331
• feat: add create-issue skill by @ko3n1g :: PR: #4338
• M4 leftover for TE cuda graph by @shifangx :: PR: #3137
• fix: wait for async P2P send before deallocating output tensor by @ZhiyuLi-Nvidia :: PR: #4047
• ci(gb200): add 1-node mr-github functional test variants by @ko3n1g :: PR: #4334
• Fix potential coredump issue that occurs when saving a checkpoint by @ezioliao :: PR: #1871
• docs: bump versions1.json to 0.17.0 (latest) by @ko3n1g :: PR: #4360
• Port DeepSeek Sparse Attention to MambaModel by @janEbert :: PR: #3553
• Add ...

NVIDIA Megatron Core 0.17.1

Changelog Details

• beep boop 🤖: Bumping versions by @svcnvidia-nemo-ci :: PR: #4349
• cp: NVFP4 native weights for DDP (4005) into core_r0.17.0 by @ko3n1g :: PR: #4290
• docs: bump project.json and versions1.json to 0.17.0 by @ko3n1g :: PR: #4361
• [docs] ci: fix version picker in 0.17.0 docs by @ko3n1g :: PR: #4363
• [docs] ci: use parent-relative json_url for version picker by @ko3n1g :: PR: #4366
• Backport NVRx async checkpoint compatibility to core_r0.17.0 by @sbak5 :: PR: #4453
• cp: add permute fusion into hybrid ep (4089) into core_r0.17.0 by @ko3n1g :: PR: #4488
• cp: get rid of weights_only=False (4434) into core_r0.17.0 by @ko3n1g :: PR: #4554
• cp: SafeUnpickler class for safe pickle usage (4319) into core_r0.17.0 by @ko3n1g :: PR: #4555
• cp: checkpoint integrity verification (4305) into core_r0.17.0 by @ko3n1g :: PR: #4556
• fix(async_ckpt): import inspect in async_utils on core_r0.17.0 by @ko3n1g :: PR: #4597
• chore(beep boop 🤖): Bump uv.lock (core_r0.17.0) (2026-05-04) by @svcnvidia-nemo-ci :: PR: #4598
• cp: fix: Replace polynomial rolling hash with SHA-256 for prefix caching (#4158) by @chtruong814 :: PR: #4612
• build: relax transformers cap to <=5.3.0 on core_r0.17.0 by @ko3n1g :: PR: #4701
• chore: Bump TE to latest 2.14 by @chtruong814 :: PR: #4772
• cp: additional tests for nvrx (#4522) by @chtruong814 :: PR: #4826
• Release 0.17.0 by @ko3n1g
• Bump mfsdp to 0.4.0 by @ko3n1g
• cp: NVFP4 native weights for DDP (4005) into core_r0.17.0 (#4290) by @ko3n1g
• docs: bump project.json and versions1.json to 0.17.0 (#4361) by @ko3n1g
• [docs] ci: fix version picker in 0.17.0 docs (#4363) by @ko3n1g
• [docs] ci: use parent-relative json_url for version picker (#4366) by @ko3n1g
• chore(beep boop 🤖): Bump (core_r0.17.0) (2026-04-20) by @github-actions[bot]
• Backport NVRx async checkpoint compatibility to core_r0.17.0 (#4453) by @sbak5
• add permute fusion into hybrid ep (#4089) by @Autumn1998
• Merge pull request #4488 from NVIDIA/cherry-pick-4089-core_r0.17.0 by @ko3n1g
• get rid of weights_only=False (#4434) by @dimapihtar
• SafeUnpickler class for safe pickle usage (#4319) by @dimapihtar
• checkpoint integrity verification (#4305) by @dimapihtar
• Merge pull request #4554 from NVIDIA/cherry-pick-4434-core_r0.17.0 by @ko3n1g
• Merge pull request #4555 from NVIDIA/cherry-pick-4319-core_r0.17.0 by @ko3n1g
• Merge pull request #4556 from NVIDIA/cherry-pick-4305-core_r0.17.0 by @ko3n1g
• fix(async_ckpt): import inspect in async_utils on core_r0.17.0 (#4597) by @ko3n1g
• chore(beep boop 🤖): Bump uv.lock (core_r0.17.0) (2026-05-04) (#4598) by @svcnvidia-nemo-ci
• cp: fix: Replace polynomial rolling hash with SHA-256 for prefix caching (#4158) (#4612) by @chtruong814
• build: relax transformers cap to <=5.3.0 on core_r0.17.0 (#4701) by @ko3n1g
• chore(beep boop 🤖): Bump (core_r0.17.0) (2026-05-11) by @github-actions[bot]
• chore: Bump TE to latest 2.14 (#4772) by @chtruong814
• cp: additional tests for nvrx (#4522) (#4826) by @chtruong814
• chore(beep boop 🤖): Bump (core_r0.17.0) (2026-05-18) by @github-actions[bot]
• chore(beep boop 🤖): Bump (core_r0.17.0) (2026-05-25) by @github-actions[bot]

26.04-alpha.rc2

[MXFP8 param gather]Post processing after synced param AG in eval (#4…

26.04-alpha.rc1

[Dev] Add high-priority a2a comm stream option and hybridep preproces…

NVIDIA Megatron Core 0.17.0

Changelog Details

• Fix two minor bugs in MTP implementation for hybrid models by @deepakn94 :: PR: #3194
• Update README.md by @mvirts :: PR: #2111
• mRoPE for MTP by @BestJuly :: PR: #3114
• Fix bug in SFTDataset by @duncanriach :: PR: #3185
• Fix several syntax error by @HollowMan6 :: PR: #3004
• Fix for RL Test by @wdykas :: PR: #3148
• Fix latent moe flops and backward_dw by @buptzyb :: PR: #2977
• Use global user buffer when the bucket size does not fit FixedPoolAllocator by @shengf-nv :: PR: #2857
• ci: Checkpoint retention by @ko3n1g :: PR: #3205
• Add unit test for LatentMoE by @venmugil :: PR: #2892
• ci: Enable unit tests on merge-queue by @ko3n1g :: PR: #3186
• Fix seq pack flag in get_logprobs by @mathemakitten :: PR: #3206
• ci(fix): Parse unit tests in merge-queue by @ko3n1g :: PR: #3224
• Fix TE 2.12 AllGather CI failure by @BestJuly :: PR: #3101
• ci(hotfix): Pin uv by @ko3n1g :: PR: #3233
• Add a unit test to check that RL get_logprobs will reuse training cudagraphed forward pass by @mathemakitten :: PR: #3209
• Do not offload grad buffers when training graphs are enabled by @mathemakitten :: PR: #3231
• Fix missing PackedSeqParams import by @parthmannan :: PR: #3214
• Synchronize the request counts for EP inference with strict matching by @santhnm2 :: PR: #3033
• Fix coordinator address collision check in flask by @tdene :: PR: #3208
• Do not let requests fail silently inside inference engine by @tdene :: PR: #3228
• torch saver inference model offload by @wdykas :: PR: #3170
• enable cuda graph ut by @Autumn1998 :: PR: #3197
• Support EP with HSDP by @wplf :: PR: #2840
• [Main] Add the missing part to support 1F1B overlap for Qwen3-Next by @BestJuly :: PR: #2997
• Missing import fix by @parthmannan :: PR: #3241
• Miscellaneous inference cleanup (Replay of !2955) by @santhnm2 :: PR: #3232
• Add DistributedInitConfig by @maanug-nv :: PR: #3173
• Fix checkpoint converter missing parallel group initialization by @yashaswikarnati :: PR: #3217
• Skip empty sequences and chunks in MTP tensor roll by @BestJuly :: PR: #3035
• Implement get_parameters for ChainedOptimizer by @nschank :: PR: #3201
• ci(fix): Create main/dev image tags by @ko3n1g :: PR: #3252
• Reapply "Add MTP support for hybrid models (#2363)" by @sancha :: PR: #3207
• Fix uv install for GH actions by @Phlip79 :: PR: #3259
• Update the project structure in README by @janEbert :: PR: #3251
• Cherry-pick: Fix mtp_num_layers and clip_qk issues (#2581, #2776) by @BestJuly :: PR: #3075
• RL: training cudagraphs functional test by @mathemakitten :: PR: #3235
• [Main] fix cg missing wgrad hook by @Wohox :: PR: #3074
• Avoid .cuda call on meta device in LanguageModel by @nschank :: PR: #3202
• fix checkpointing error message by @dimapihtar :: PR: #3203
• Nano QAT/D fix with sft tokenizer and datasets by @ChenhanYu :: PR: #3254
• Revert "fix checkpointing error message (#3203)" by @ko3n1g :: PR: #3283
• Reapply "fix checkpointing error message (#3203)" (#3283) by @ko3n1g :: PR: #3285
• docs: Add changelog for 0.15.3 by @ko3n1g :: PR: #3286
• ci: Set throughput tests as flaky by @chtruong814 :: PR: #3301
• chore: Move GB200 tests to nightly by @ko3n1g :: PR: #3302
• Ensure type-checker understands use of Submodules in bert_model by @nschank :: PR: #3256
• Override extra_repr instead of repr by @nschank :: PR: #3200
• Replace ModuleSpec with Protocols for LayerNorm submodules by @nschank :: PR: #3090
• Non colocated refit by @wdykas :: PR: #3213
• Fuse permute+pad and unpermute+unpad ops for FP8/FP4 training by @xiaoxi-wangfj :: PR: #2763
• Add check to prevent MFSDP from numeric issue in gradient accumulate fusion by @shjwudp :: PR: #2904
• update get_embedding_ranks and get_position_embedding_ranks docstrings by @c1lovez1 :: PR: #3223
• Param offset in _ParamAndGradBucket should be aligned by @skydoorkai :: PR: #3007
• ci: Add secrets detector by @chtruong814 :: PR: #3180
• Ensure type-checker understands use of Submodules in llava_model by @nschank :: PR: #3257
• updates to support modelopt EAGLE training with CP by @yeyu-nvidia :: PR: #3147
• fully remove legacy tokenizer system by @dimapihtar :: PR: #2946
• M-FSDP: Remove redundant stream waits in HSDP to prevent CG fail by @shjwudp :: PR: #2941
• General README and pyproject fixes by @ahmadki :: PR: #2907
• chore: More aggressive checkpointing by @ko3n1g :: PR: #3315
• ci: Pin down setuptools to lt 82 by @ko3n1g :: PR: #3313
• fix: numpy overflow by @ko3n1g :: PR: #3306
• fix: T5 dataset by @ko3n1g :: PR: #3307
• ci: Revert "ci: Add secrets detector (#3180)" by @chtruong814 :: PR: #3330
• ci: Add more tests, run on merge-queue by @ko3n1g :: PR: #3317
• ci: Remove merge-gate environment check by @chtruong814 :: PR: #3331
• Use FP4 context for mamba by @kwyss-nvidia :: PR: #2604
• ci: Ensure we run all functional tests in merge group by @chtruong814 :: PR: #3332
• Replace ModuleSpec with Protocols for inputs to MLP by @nschank :: PR: #3084
• ci: Fix merge queue functional tests by @chtruong814 :: PR: #3337
• ci: skip queue in merge-gate by @ko3n1g :: PR: #3343
• ci: Timeout for functional tests by @ko3n1g :: PR: #3346
• update checkpointing documentation by @dimapihtar :: PR: #3347
• Update golden values to reflect improvements by @tdene :: PR: #3350
• BUGFIX: gpt vs hybrid model mtp naming mismatch by @sancha :: PR: #3334
• Disable flaky test by @tdene :: PR: #3354
• re-enable gpt grpo tests by @jon-barker :: PR: #3348
• Fix SFT Pipeline when TP>1 by @asolergi-nv :: PR: #3268
• Fixes for KD mode by @AAnoosheh :: PR: #3342
• chore: Update codeowners file by @ko3n1g :: PR: #3365
• Siddharth/fix inference functional tests by @sidsingh-nvidia :: PR: #3357
• Switch oncall by @janEbert :: PR: #3360
• Add missing RMSNorm to llama train script by @AAnoosheh :: PR: #3314
• Fix inference for MTP models by @tdene :: PR: #3297
• Add a logprobs test with real gpt model. by @yobibyte :: PR: #2870
• Add simple GRPO functional test by @tdene :: PR: #3323
• ci: Concurrency control for merge-queue by @ko3n1g :: PR: #3353
• ci: Update golden value download script to work with Github by @chtruong814 :: PR: #3335
• fix: correct typos 'seperated' and 'recieved' by @thecaptain789 :: PR: #3305
• Improved PyTorch profiler and added PyTorch execution trace by @shengf-nv :: PR: #3273
• Removing etc from main index page, shifted name of discussions by @megnvidia :: PR: #3271
• build: Bump TE on 2.12 by @ko3n1g :: PR: #3371
• ci(hotfix): job conditions by @ko3n1g :: PR: #3376
• Record moe routing decisions during inference. by @sidsingh-nvidia :: PR: #3034
• [Main] Fix EP Overlap Bugs for Full-Iter CG by @Wohox :: PR: #3164
• Avoid direct pickle import by @maanug-nv :: PR: #3375
• Delete old pretrain_* files by @Phlip79 :: PR: #3359
• Add Qwen3-VL support with Megatron-FSDP by @xuwchen :: PR: #2841
• Refactor Mamba chunked prefill by @santhnm2 :: PR: #3265
• Improved parallel logging of learning rate by @jstjohn :: PR: #3319
• Add enhanced event tracking with TTFT measurement and compact serialization. by @lmcafee-nvidia :: PR: #3253
• Add assertion that max_requests is divisible by tp_size by @santhnm2 :: PR: #3304
• Move to using the Inference OpenAI API server by @ArEsKay3 :: PR: #3107
• Update moe github test cases. by @Victarry :: PR: #3077
• Split layer_specs to return Submodules instead of ModuleSpecs by @nschank :: PR: #3255
• ci: Remove gpu sanity check by @chtruong814 :: PR: #3420
• [Critical-Bug] Fix Uneven PP for Mamba models (Nemotron3-nano) by @kevalmorabia97 :: PR: #3399
• Fix for rl by @shanmugamr1992 :: PR: #3390
• Add check for full_iteration scope before instantiating CudaGraphManager by @vasunvidia :: PR: #3362
• Fix broken links throughout by @megnvidia :: PR: #3230
• Decouple topk and loss from DSA Indexer by @kunlunl :: PR: #3248
• Extract intermediate embeddings of transformer block by @sajadn :: PR: #3060
• Move to using the Inference OpenAI API server (bis) by @tdene :: PR: #3395
• Make Mamba inference state memory ratio configurable by @santhnm2 :: PR: #3322
• Fix configs for RL model environments by @tdene :: PR: #3441
• Replace pickle with json in rl_utils by @tdene :: PR: #3351
• fix: correct typo in demo training example by @dndnda :: PR: #3428
• Clean up logging inside inference flask server by @tdene :: PR: #3437
• ci: Update release-docs workflow to use FW-CI-templates v0.72.0 by @chtruong814 :: PR: #3438
• Fix --tokenizer-hf-include-special-tokens by @jon-barker :: PR: #3422
• Update num_tokens_to_generate default for Gym by @tdene :: PR: #3453
• Fix slowdown in inference flask server by @tdene :: PR: #3445
• Add a normalized scale for MTP per token loss by @BestJuly :: PR: #3159
• [Bugfix] Fix nan loss caused by zero token in MTP by @BestJuly :: PR: #3396
• Log RL metrics per environment by @yobibyte :: PR: #3446
• Move tensor offload/onload out of RL code by @tdene :: PR: #3029
• Fix another inference flask / Gym interaction by @tdene :: PR: #3467
• Add Engine event to the follow up requests after checkpointing by @ArEsKay3 :: PR: #3473
• adding in copyright blurb at the top of md file by @megnvidia :: PR: #3394
• [Megatron-FSDP] Add fsdp_all_gather_in_start_param_sync option in DDP Config by @shjwudp :: PR: #3095
• ci: Update release workflow to include changelog and publish docs by @chtruong814 :: PR: #3472
• ci(fix): Weekly GPT tests by @ko3n1g :: PR: #3443
• ci: Remove environments by @ko3n1g :: PR: #3462
• update HF tokenizer defaults by @dimapihtar :: PR: #3440
• ci: Bump preflight to detect our svc by @ko3n1g :: PR: #3494
• build: Drop Python 3.10 support and pip install one-logger by @ko3n1g :: PR: #3485
• PTQ changes for upcoming QAD by @AAnoosheh :: PR: #3124
• ci: Bump pre-flight for Bot SSO by @ko3n1g :: PR: #3497
• Revert "build: Drop Python 3.10 support and pip install one-logger (#...

NVIDIA Megatron Core 0.16.1

Changelog Details

• cp: ci: Skip cleanup-taint-node jobs during deployments (3612) into core_r0.16.0 by @ko3n1g :: PR: #3613
• beep boop 🤖: Bumping versions by @svcnvidia-nemo-ci :: PR: #3616
• cp: docs: Fix version picker urls (3621) into core_r0.16.0 by @ko3n1g :: PR: #3622
• cp: ci: Increase changelog generation max PRs fetched (3620) into core_r0.16.0 by @ko3n1g :: PR: #3623
• Cherry-pick #3399 for Mamba Uneven PP fix by @kevalmorabia97 :: PR: #3544
• cp: fix: async_utils: explicit GC in persistent checkpoint worker loop (3591) into core_r0.16.0 by @ko3n1g :: PR: #3628

NVIDIA Megatron Core 0.16.0

Changelog Details

• ci: Fix copyright checker by @ko3n1g :: PR: #1893
• chore: Add codeowners by @ko3n1g :: PR: #1897
• ci: Extend queue-manager for dev branch by @ko3n1g :: PR: #1906
• ci: Move test optimizer into its own bucket by @ko3n1g :: PR: #1909
• ci: Configure cherrypick bot by @ko3n1g :: PR: #1925
• Ci approve dev by @ko3n1g :: PR: #1933
• ci: Update nightly schedule by @ko3n1g :: PR: #1934
• ci: Bump pre-flight for runs on main/dev by @ko3n1g :: PR: #1935
• ci: Allow skipping on main by @ko3n1g :: PR: #1936
• Ko3n1g/ci/pr template community bot by @ko3n1g :: PR: #1937
• ci: More granular unit tests buckets by @ko3n1g :: PR: #1932
• Add sequence packing to RL by @tdene :: PR: #1911
• chore: Update template by @ko3n1g :: PR: #1939
• chore: Add description about who can merge by @ko3n1g :: PR: #1940
• Ko3n1g/ci/fix main on eos by @ko3n1g :: PR: #1938
• Ko3n1g/ci/internal mrs by @ko3n1g :: PR: #1942
• ci: Fix branch of approval bot by @ko3n1g :: PR: #1944
• ci: Approvalbot for other branches by @ko3n1g :: PR: #1947
• ci(fix): Approval bot by @ko3n1g :: PR: #1949
• Ko3n1g/ci/sync branches by @ko3n1g :: PR: #1956
• Ko3n1g/ci/add milestone by @ko3n1g :: PR: #1951
• Remove M-FSDP testing under LTS environment by @shjwudp :: PR: #1959
• ci: Run on push to release branch by @ko3n1g :: PR: #1960
• Fix typo in rl section of CODEOWNERS by @tdene :: PR: #1968
• ci: Update copyright checker by @ko3n1g :: PR: #1973
• Ko3n1g/ci/auto reminder GitHub by @ko3n1g :: PR: #1955
• ci(fix): Run tests label by @ko3n1g :: PR: #1970
• Make get_asyncio_loop safe to use repeatedly by @tdene :: PR: #1990
• chore: Update codeowners by @ko3n1g :: PR: #2012
• zarr soft deprecation by @dimapihtar :: PR: #2004
• Deduplicate dynamic engine + coordinator. by @lmcafee-nvidia :: PR: #1981
• Update symmetric registration interface to sync-up with upstream pytorch change by @youngeunkwon0405 :: PR: #1924
• Safely access state dict args in load ckpt by @maanug-nv :: PR: #1957
• Allow mixed-batch sampling in dynamic inference by @tdene :: PR: #1927
• Stop Nemo_CICD_Test from failing in forks by @tdene :: PR: #2024
• Clean up dynamic inference step by @tdene :: PR: #1992
• ci: Auto-update copy-pr-bot vetters by @ko3n1g :: PR: #1850
• ci: Fix build-push-wheel workflow by @ko3n1g :: PR: #2022
• ci: Enable integration tests by @ko3n1g :: PR: #2023
• chore: Update tooling for interactive jobs by @ko3n1g :: PR: #2032
• Have datasets account for tokenizers which incorrectly define PAD by @tdene :: PR: #2017
• revert(hotfix): ci: trustees_override by @ko3n1g :: PR: #2041
• add missing warnings import in model parallel config by @yashaswikarnati :: PR: #2039
• Reduce-scatter implementation with FP32 accumulation by @deepakn94 :: PR: #1967
• ci(fix): Workflows on main by @ko3n1g :: PR: #2045
• build: Bump modelopt by @ko3n1g :: PR: #2046
• Remove TestCaptureFreezeGC unit test. by @lmcafee-nvidia :: PR: #1978
• ci: Add multi-approval action by @ko3n1g :: PR: #2051
• Ko3n1g/ci/test iteration time by @ko3n1g :: PR: #2067
• Allow inference test throughput to vary by 10% by @mathemakitten :: PR: #2070
• chore: Fix autoformatter by @ko3n1g :: PR: #2073
• ci(hotfix): Bypass approvalbot in merge-queue by @ko3n1g :: PR: #2082
• chore: Update local tooling by @ko3n1g :: PR: #2066
• Add extra RL files by @tdene :: PR: #2077
• Prevent summary jobs from running in forks by @tdene :: PR: #2083
• ci: Fix test scope by @ko3n1g :: PR: #2091
• Refactor the attention metadata into separate classes by @kanz-nv :: PR: #2001
• Guard against incorrectly using MoE prefill graphs by @tdene :: PR: #2030
• Run mr-slim tests in lightweight-mode by @chtruong814 :: PR: #2106
• Inference | Lazy compile UVM allocator. by @lmcafee-nvidia :: PR: #1977
• chore: Reenable trustees by @ko3n1g :: PR: #2108
• Ko3n1g/chore/update release settings by @ko3n1g :: PR: #2097
• ci(fix): Changeset of copyright checker by @ko3n1g :: PR: #2110
• Remove unnecessary check on rotary_pos_cos by @santhnm2 :: PR: #2003
• (Reverted) Inference | Lazy compile UVM allocator. by @lmcafee-nvidia :: PR: #2125
• Refactor Attention Metadata to Separate Classes by @kanz-nv :: PR: #2112
• Refactor model_provider to model_builder format for ModelOpt examples by @AAnoosheh :: PR: #2107
• wandb Inference stats logging by @wdykas :: PR: #2026
• Make PipelineParallelLayout always return str from __repr__ by @ananthsub :: PR: #2055
• Add flash_attn_3 as first option for FA3 import by @santhnm2 :: PR: #2010
• Add debugging hint for case when cudagraphs are created but no matching runner is found by @mathemakitten :: PR: #2129
• ci: LTS container by @ko3n1g :: PR: #2133
• Fix param init by @cuichenx :: PR: #2033
• Hotfix to unit tests on hopper FA3 by @tdene :: PR: #2143
• Add BytesIO to safe_globals by @tdene :: PR: #2074
• add deprecation warning for legacy tokenizer system by @dimapihtar :: PR: #2145
• replay: ci: Bump LTS container by @ko3n1g :: PR: #2157
• Hotfix to unit tests on hopper FA3 (bis) by @tdene :: PR: #2179
• Fix has_modelopt_state() for native Torch checkpoint format by @AAnoosheh :: PR: #2160
• chore: Remove codeowners by @ko3n1g :: PR: #2175
• Fix FP8 inference with sequence parallelism by @santhnm2 :: PR: #2009
• Replace ModelOpt generation server by @AAnoosheh :: PR: #2147
• Add hybrid model support for dynamic inference engine by @santhnm2 :: PR: #1907
• Async task and event loop safety in Megatron Core by @tdene :: PR: #2025
• Rename skip_prompt_log_probs by @tdene :: PR: #2181
• Dynamic inference context | UVM only. by @lmcafee-nvidia :: PR: #1983
• ci: Run auto-update-copy-pr-bot only on forks by @ko3n1g :: PR: #2191
• Inference throughput tests: refactor goldens to be in list format by @mathemakitten :: PR: #2072
• Enable TE custom quantization recipe by @negvet :: PR: #2005
• Add MoE parameters to ModelOpt pruning example + conf fixes by @kevalmorabia97 :: PR: #2205
• Add repr to pg collection class by @yashaswikarnati :: PR: #2089
• Move data_samplers.py from legacy to training.datasets & add DistributedSignalHandler to DataLoader workers by @asolergi-nv :: PR: #2068
• Fix Megatron-FSDP checkpoint save failure by @shjwudp :: PR: #2138
• Fix moe CODEOWNERS. by @jaredcasper :: PR: #2200
• chore: Update LICENSE by @ko3n1g :: PR: #2219
• remove megatron.training dependency from megatron.core for FSDP checkpoint with EP by @ananthsub :: PR: #2113
• Tensorize dynamic inference mixed sampling by @tdene :: PR: #2105
• Add unit test for inference DP coordinator by @tdene :: PR: #2187
• Inference linear layer by @sidsingh-nvidia :: PR: #1908
• chore: Prefer Nvidia email addresses for reminder bot by @ko3n1g :: PR: #2221
• [Megatron-FSDP] Fix hang caused by non-deterministic reduce-scatter by @shjwudp :: PR: #2218
• Remove qwen symlink to fix for case-insensitive FS by @kevalmorabia97 :: PR: #2235
• Optimizer refactor: clean up public get_megatron_optimizer interface and provide a more general API to support passing in different hyperparameters to subsets of parameters by @deepakn94 :: PR: #2047
• Fix CI for PR#1983 by @lmcafee-nvidia :: PR: #2245
• Fix aux-loss logging for hybrid models by @deepakn94 :: PR: #2197
• Update flops calculation (for throughput) for hybrid MoEs by @deepakn94 :: PR: #2198
• Enable kv cache in training for eagle by @yeyu-nvidia :: PR: #1895
• Tensorize dynamic inference mixed sampling (bis) by @tdene :: PR: #2231
• chore: Fix codeowners by @ko3n1g :: PR: #2264
• Allow loading checkpoint from iteration 0 by @ananthsub :: PR: #2199
• ci: Skip install test in merge queue by @chtruong814 :: PR: #2281
• Add MoE layer type to hybrid models by @deepakn94 :: PR: #2259
• Add the Hybrid-EP backend to the Flex Dispatcher by @Autumn1998 :: PR: #2176
• [MAIN][NVFP4] Support NVFP4 MOE with Proper Padding by @zhongbozhu :: PR: #1985
• Update ModelOpt example readmes and advanced usage by @kevalmorabia97 :: PR: #2273
• Fix UVM compatibility with CUDA 13. by @lmcafee-nvidia :: PR: #2243
• ci: Add flaky marker to LTS tests by @ko3n1g :: PR: #2290
• Dynamic engine suspend/resume via prefill. by @lmcafee-nvidia :: PR: #1982
• fix: Pass the timeout argument for the EP group by @yanring :: PR: #2268
• JIT for MoE router and preprocess by @yaox12 :: PR: #1919
• Hotfix to CI, until the fix gets reviewed by @tdene :: PR: #2298
• Add functional test for DP coordinator throughput by @tdene :: PR: #2189
• Add asyncio Queue like in Python 3.13 by @tdene :: PR: #2224
• Fixes for PR#1982 by @lmcafee-nvidia :: PR: #2303
• Fix PP KV cache allocation and enable multi-node PP inference by @santhnm2 :: PR: #2182
• Revert active-buffer-size-gb arg name. by @lmcafee-nvidia :: PR: #2257
• feat: check: api backwards compatibility by @pablo-garay :: PR: #2251
• Add MambaInferenceStateConfig dataclass by @santhnm2 :: PR: #2265
• Fix typo in inference example by @santhnm2 :: PR: #2311
• feat: initialization of API backward compatibility verification by @pablo-garay :: PR: #2310
• Fix Mamba TP and remove confusing legacy initialization by @jaredcasper :: PR: #2202
• Refactor KD to use ModelOpt plugins file by @AAnoosheh :: PR: #2305
• Fix dynamic context syntax and remove redundant tensors by @kanz-nv :: PR: #2336
• Improve asyncio exception handling by @tdene :: PR: #2300
• ci: Upload to testpypi only on main by @ko3n1g :: PR: #2342
• implement graph config by @kanz-nv :: PR: #2203
• feat: required check adjustment by @pablo-garay :: PR: #2350
• fix: load iteration 0 for release checkpoints by @ananthsub :: PR: #2351
• Explicitly zero out padding token activations for dynamic inference by @santhnm2 :: PR: #2008
• Bugfix for Mamba with Chunked-Prefill by @sidsingh-nvidia :: PR: #2293
• Break apart dynamic inference step into 2 methods by @tdene :: PR: #2192
• Prevent unnecessarily overwriting the default Hugging Face chat te...

NVIDIA Megatron Core 0.15.3

This release addresses known security issues. For the latest NVIDIA Vulnerability Disclosure Information visit https://www.nvidia.com/en-us/security/, for acknowledgement please reach out to the NVIDIA PSIRT team at PSIRT@nvidia.com

NVIDIA Megatron Core 0.15.2

core_v0.15.2

Megatron-Core v0.15.2

NVIDIA Megatron Core 0.15.1

core_v0.15.1

Core v0.15.1