export AWQ、GPTQ 、GGUF format

[artificialzjy]

My deployment pipeline requires AWQ、GPTQ format，Can convert "compress-tensors" quantization format to AWQ、GPTQ format etc.

[HDCharles]

Can you clarify the question. What do you mean by AWQ format?

The way we and most of the literature looks at quantization has 3 parts

1. What type of quantization are you doing, fp8, W4A16, int8_dynamic...etc. this also includes things like packing and storage format. Often called quantization format/scheme. This is what compressed tensor handles (here) in a way that integrates with vllm nicely.
2. How do you calculate your quantized weight and quantization parameters given the type of quantization you want to do
   2a) preprocessing transforms like smoothquant/spinquant
   2b) quantized weight calculation, things like round to nearest, autoquant, GPTQ, AWQ
   This is often called quantization technique
3. What kernel is going to take your packed quantization format and apply the actual inference op. (handled by vllm)

1 and 3 largely depend on one another while 2 is orthogonal to both.

This is why i'm confused AWQ and GPTQ are not quantization formats. They are techniques, if you have a model that is deployable after applying AWQ then it should also be deployable if you did round to nearest assuming the quantization format is identical. As a comparison, the quantization technique is like the optimizer when training a model. If you train a model with an SGD optimizer and deploy it, you would expect that the same model trained with an Adam optimizer would work with the exact same deployment setup.