is nemo export-deploy going to support exporting fp8 checkpoints (HF)?

[snowmanwwg]

is nemo export-deploy going to support exporting fp8 checkpoints (HF)? Or are they suppossed to directly use ModelOpt?

[oyilmaz-nvidia]

Is the fp8 ckpts quantized with ModelOpt? If yes, we'll use ModelOpt's APIs to export it.

[snowmanwwg]

No not yet via ModelOpt. We're using recipe like bf16_with_fp8_current_scaling_mixed during my training, is there a native way to export the model weights as fp8 (with the scaling factors) instead of currently bf16? What is the recommended process?

[oyilmaz-nvidia]

One more question, is this for exporting a fp8 HF ckpt to TensorRT-LLM or a fp8 MBridge ckpt to HF?

We currently supporting exporting fp8 NeMo 2.0 checkpoints to TensorRT-LLM. These ckpts are not PTQed with Modelopt. They come with scaling factors.

[verdimrc]

Hi, @oyilmaz-nvidia, this is for exporting bf16 Mbridge ckpt to fp8 HF.

The bf16_with_fp8_current_scaling_mixed() recipe in MBridge keeps the param dtype as bf16.

[oyilmaz-nvidia]

Oh I see. Currently the code to export MBridge bf16 to HF bf16 is handled by the MBridge team and the repo. So, I am guessing this should go into the MBridge repo. Then, the ED should use that functionality to deploy the fp8 mbridge ckpts.

@ananthsub What do you think?