A type of artificial intelligence model designed to understand and generate human-like text by leveraging vast amounts of data and computational power.
The process of evaluating the performance of a system, model, or algorithm by comparing it against a set of predefined standards or metrics.
The process of breaking down text into smaller units, such as words or subwords, which are used as input for language models.
The process of using a trained model to make predictions or generate outputs based on new input data.
The process of adapting a pre-trained model to a specific task or dataset by training it further on task-specific data.
Configurable parameters that are set before training a model, such as learning rate, batch size, or number of layers.
An optimization algorithm used to minimize the loss function by iteratively updating model parameters.
A training technique where gradients are accumulated over multiple microbatches before performing a single optimizer step
The total number of samples processed across all GPUs before a parameter update in a distributed training setup.
The number of samples processed in a single step(forward+backward only) on a single GPU.
A parallelization technique where each DP worker works on different input data and the model parameters are updated with an average of the gradients across all the DP workers. For MOE models, DP here referrers to DP on the Attention block.
A model parallelization technique that distributes the parameter tensor of an individual layer across GPUs.
A parallelization technique that divides consecutive layers of the model across multiple GPUs, with each GPU processing different layers or segments of the network sequentially.
A technique that is used to reduce the pipeline bubbles or a measure of the time the GPUS spend idle when pipeline parallelism is enabled. See more information here
A parallelism technique that enables GPU memory savings by sharding the model parameters, gradients, and optimizer states across GPUs
A metric that measures the efficiency of GPU usage in AI computations by comparing the observed throughput to the theoretical maximum performance of the hardware.
The model parallelism strategy used in Mixture of Experts (MoE) architectures, that distributes experts of an MoE model across GPUs.
The model parallelism strategy for doing Tensor Parallelism for MoE layers.