FP8 QAT / FP8 block-wise quantization

[cassanof]

Having QAT for FP8 would be a great addition, and FP8-blockwise quantization in general.

[danielvegamyhre]

We have an issue tracking fp8 quantization with block-wise scaling here #1594

[supriyar]

@cassanof for QAT - do you mean quantized fine-tuning with FP8 or QAT (which simulates the quantization but doesn't actually quantize during training). Also cc @andrewor14 for QAT

[cassanof]

the latter :)

[andrewor14]

Hi @cassanof, thanks for raising the issue! Do you mind sharing the use case for FP8 QAT? Most use cases we've seen are directly doing FP8 (in lower precision) pretraining or finetuning. Is your goal to do FP8 QAT (fake quantize in high precision, e.g. bfloat16), and then actually quantize the model to FP8 after training? Is there more context you can share regarding this workflow?

[cassanof]

hey @andrewor14. my goal is to do most of my training in bf16, and then right at the end do blockwise QAT to improve the performance of my model, which will get block-wise quantized for inference.