[Feature] FP8 Quantization

[1145284121]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/mit-han-lab/nunchaku/discussions/new/choose. Otherwise, it will be closed.
• 2. I will do my best to describe the issue in English.

Motivation

Has your team tried using FP8 quantization? The accuracy will be better compared to NF4/INT4? Also, are there any plans to support it in the future?

Related resources

No response

[lmxyy]

Yes, it's on our roadmap, but currently a lower priority.

[1145284121]

Yes, it's on our roadmap, but currently a lower priority.

Thank you for your reply. Could I ask why FP8 quantization is considered a relatively low priority? Is it mainly for efficiency reasons or because of accuracy?

[lmxyy]

Yes, it's on our roadmap, but currently a lower priority.

Thank you for your reply. Could I ask why FP8 quantization is considered a relatively low priority? Is it mainly for efficiency reasons or because of accuracy?

Just because we have a lot of TODOs and do not have enough bandwidth.