Quantization on NPUs

[mpaulazamin]

Hello!

I have a question about your software stack. Can I perform quantization of LLMs on NPU using AMD Quark? If yes, how? I was reading some tutorials and only found resources related to quantization on GPU.

Thanks in advance!

[fxmarty-amd]

@mpaulazamin I am not very familiar with this part of the stack, but you may want to have a look at https://quark.docs.amd.com/latest/supported_accelerators/ryzenai/index.html#quark-for-ryzen-ai-npu and/or search NPU in the doc.

My understanding is that ONNX Runtime GenAI is eventually used to execute quantized models for NPU.

[rwfsmith]

This is probably a better doc for it.

https://ryzenai.docs.amd.com/en/latest/oga_model_prepare.html

I've been able to successfully run this for Phi-3.5 and Llama 3.1 models, but haven't been successful with Qwen 2.5 yet.