Possible PrefixQuant issues on Smaller LLMs?

[sasha-hailo]

Dear @ChenMnZ,
Thank you for the paper and sharing the code.
I'm very interested about the idea. Especially encouraging is the prospect of reaching great accuracy using static per-channel / tensor quantization!

I evaluated PrefixQuant with a few HF models, and was able to reproduce the good results of your paper with Llama-2-7B, Llama-3-8B, Mistral-7B (and a couple other models).
[I'm writing this to indicate that my settings are probably OK, and are not the cause for the issues below]

However, with smaller models (of sizes 0.5B - 3B), the results I'm getting are catastrophic (multi-digit perplexity numbers on WikiText2).
The models I tested that resulted in PPL disasters are:

• Qwen2-0.5B
• Llama-3.2-1B
• Qwen2-1.5B-Instruct
• Llama-3.2-3B-Instruct

A couple remarks:

• Llama-3.2-xx models are not supported by the older transformers library version required by your repo (4.40.1).
  I encountered minor compatibility issues when trying to run your repo with a more recent transformers version, but solved them by explicitly configuring use_cache=True. But even after solving them, the models would not properly quantize.
• For Qwen2-0.5B model, there was an issue with creating the online Hadamard matrix for down_proj input (function get_hadK() does not support the intermediate feature size). I overcame it by disabling the configuration option down_online_had, but it didn't help reach reasonable quantization accuracy.
• As a sanity check, I tried to quantize one of the problematic models listed above with additional fine-tuning - this did not help, either.

Do you have any inputs on what can help SLM models quantize well?

Thanks in advance!

[sasha-hailo]

Update: I nailed down the root cause of the quantization disaster for the models mentioned above, and it's when pre_rotate is enabled.

Still have no clue where the actual bug is that breaks the smaller models (while working fine for the larger ones).

[sasha-hailo]

Nailed it!
After comparing with another quantization repo implementing rotation (LLMC),
I found the code that makes it work.
Please see lines 23-33 here:
https://github.com/ModelTC/llmc/blob/9943e792f5f49341351dbd0c242ce8e0ba50b9a9/llmc/compression/quantization/quarot.py#L26

[ChenMnZ]

@sasha-hailo Thanks for reporting this issue and offer a solution!

[sasha-hailo]

@ChenMnZ , thank you for your response.
Now that the technical issue seems solved, we can focus on the really important issues - actual accuracy of PrefixQuant on SLMs.
After fixing the rotation issue, most of the affected models I mentioned above (all but Llama-3.2-3B) exhibit quantization quality issues:

• WT2 perplexity results without fine-tuning show a significant degradation (from native FP16 model).
• Even with fine-tuning, PPL results of some models are significantly worse than other quantization methods (e.g., GPTQ).
• Most notably: though the "Prefix trick" (via set_prefixed_tokens=True) improves PPL results without fine-tuning, with fine-tuning it degrades them for most SLMs I profiled!
• Llama-3.2-1B model is the most prominent example, where setting set_prefixed_tokens=True dramatically degraded perplexity (experiments carried out with fine-tuning).

Do you have any insights on why PrefixQuant looses its competitive edge for the smaller LMs,
or ideas on how to improve it?

Thanks in advance!

[ChenMnZ]

@sasha-hailo
If you suffer performance degeneration after fine-tuning, you can try to add --loss_type skip_mse for more stable training.

Additionally, I also observed that prefixed token may significantly increase the perplexity of some small model even without quantization. In this case, you should test on more other tasks to analusis weather prefixed tokens are useable.