Model quantization

[jrdimale]

Hi ! Congrats for the work

I was wondering if you’ve experimented with quantizing the model to reduce GPU memory usage, particularly during inference. Have you run any tests or benchmarks in this direction?

Also, do you think quantization could be feasible and effective for this model without causing a significant performance drop?

[destrooooyer]

Hi, I conducted some experiments using TensorRT for model conversion and acceleration.
My tests showed that when running in fp32 mode, the converted model performed very well. However, when I attempted to convert it to fp16, the inference results became completely incorrect.
Given that the model seems highly sensitive even to fp16, I suspect that further quantization might be quite challenging.