Can we borrow information from aimet-onnx when doing Mix-QAT

[iamweiweishi]

I used aimet-onnx to do mix-precision quantization, and found that 15% percentage of int16 activation could meet my requirements.

Then, I tried to do QAT to improve the accuracy. But I found that the results from aimet-onnx cannot be used when doing QAT.

Is there any way to achieve this? Or, is there any way to config the activation of specific layers to be int16-activation, like in config_file ?

Thank you.

[ShobithaShivakumar]

Hi @iamweiweishi,
Unfortunately, ONNX does not support QAT in AIMET - this is a fundamental limitation. QAT requires backpropagation through the quantization operations, which is only supported in PyTorch.

However, you have a few options:

1. Re-run mixed-precision search in PyTorch: Use aimet_torch.mixed_precision (Automatic Mixed Precision) to find which layers need int16 activation directly on your PyTorch model. This is the recommended approach.
2. Manually transfer the configuration: If you know which layers need int16 from your ONNX experiments, you can manually configure those layers in your PyTorch QuantizationSimModel by accessing the individual quantizers and setting their bitwidth to 16 and then run QAT.

[iamweiweishi]

Hi @ShobithaShivakumar ,
I found that some nodes, like conv, could be set using 'mp_configurator.set_precision' with int16 activation.
But, some nodes like, add and matmul, could be be recognized and cannot be set with int16 activation.

Thank you.