SFT & GRPO (RLVR) & DPO (RLHF alt) template

sft_iteration_1 is simple SFT on token outputs of LLM, initial interp

sft_iteration_2 uses dataloader, multimodal inputs, custom collater (custom padding) and Dataloader, also has a cosine scheduler for optimizer and grad clipping

sft_iteration_3 uses +DDP

Use LoRA, k bit quantization, gradient accumulation

grpo_iteration_1 is my initial formulation after reading about grpo with some optimizations mentioned above, raw impl

grpo_iteration_2 is a cleaner implementation, instead of duplicating the model for ref model, cache the logits, advantage scores and masks

grpo_iteration_3 is an implementation for multimodal inputs

Just for reference

GRPO function

DPO function DPO proof / derivation is longish but cool, find it here: https://cameronrwolfe.substack.com/p/direct-preference-optimization