Gradient accumulation is an important parameter to tune to find the best compromise between throughput and convergence speed:
- small batch: bad throughput because of internodes latency, but "faster" SGD
- large batch: high throughput but slower SGD
Llama2 uses 4M tokens for pretraining == 4096 x 1000
GPT3 uses 250k tokens
My script uses initially 1024x256 = 260k tokens, or 2048x256 = 500k tokens
See figure 1 from https://arxiv.org/pdf/1812.06162.pdf
See tuning playbook from Google: https://github.com/google-research/tuning_playbook
==> batch size shall be tuned wrt speed, not performance, so they recommend using the largest possible batch size on the hardware! ==> this conclusion is true for the raw batch size, but when using gradient accumulation, it's more about finding the best compromise between convergence speed and throughput.
Global batchsize may be "ramp up", from 192 to 2048 (+16 every 9_765_625 samples): https://github.com/huggingface/llm_training_handbook/tree/main/hparams
Gradient accumulation is an important parameter to tune to find the best compromise between throughput and convergence speed:
Llama2 uses 4M tokens for pretraining == 4096 x 1000
GPT3 uses 250k tokens
My script uses initially 1024x256 = 260k tokens, or 2048x256 = 500k tokens
See figure 1 from https://arxiv.org/pdf/1812.06162.pdf
See tuning playbook from Google: https://github.com/google-research/tuning_playbook
==> batch size shall be tuned wrt speed, not performance, so they recommend using the largest possible batch size on the hardware! ==> this conclusion is true for the raw batch size, but when using gradient accumulation, it's more about finding the best compromise between convergence speed and throughput.
Global batchsize may be "ramp up", from 192 to 2048 (+16 every 9_765_625 samples): https://github.com/huggingface/llm_training_handbook/tree/main/hparams