NVIDIA Megatron Core 0.4.0

Key Features and Enhancements

Models

• BERT
• RETRO
• T5

Parallelism

• Mixture of Experts support for GPT
• Model parallel efficient Distributed Data Parallel (DDP)
• Context Parallel (2D Tensor Parallel) support

Datasets

• GPT Dataset
• Blended Dataset

23.04

Merge branch 'pip_package' into 'main'

Add pip package for megatron.core

See merge request ADLR/megatron-lm!598

v2.5

Merge branch 'sc21' into 'main'

scripts for sc21

See merge request ADLR/megatron-lm!298