[Question] How to address the sharp decline in output length

[blue-whale00]

Hi team,

I'm trying to quantize a heavily fine-tuned Qwen3 4B (for Agent tool-use) with AutoRound (W4A16), but I'm getting severe performance degradationl.

The Problem:
FP16 model works great.
W4A16 model fails (1:4 win-rate vs FP16) and output is ~25% shorter (e.g., avg. 120 tokens -> 90 tokens), indicating premature stopping.

My Diagnosis:
The model has systemic activation outliers (e.g., Max Abs Value > 5000 in several mlp.down_proj layers), which are caused by the heavy SFT.
The Prefill (Prompt) stage is critical; using "Pure Answer" data to calibrate makes performance even worse.
Crucially, my main fix failed: I tried "selective quantization" (using quant_config to set bits=16 for the Top 4 worst outlier layers). This did not help and performance was still just as bad (1:4 win-rate).

My Questions:

1. How to address the sharp decline in output length
2. Does this failure (even after skipping layers) prove that W4A16 (AutoRound/AWQ) is fundamentally unsuited for models with systemic SFT-induced activation outliers?
   Thanks!

[wenhuach21]

Since we’re unable to debug the model directly and it’s relatively small, identifying the root cause is not straightforward. Based on the information you provided, I’ll check the following:

1. Calibration dataset: Did you pad your calibration data? If so, please switch to the main branch, as there are some known issues related to padding. If you don’t have enough data, you can try our default datasets.
2. Maximum activations: The maximum value of FP16 is around 65k. You might want to try using the BF16 data type instead.
3. AutoScheme: We recently introduced AutoScheme, which automatically assigns higher bit widths to more sensitive layers. You may give it a try.
4. AutoRound Best Recipe:

   auto-round-best --model xxx

It’s difficult to draw a fully conclusive negative result at this point, as multiple factors could contribute to the issue. For example, larger models tend to be more robust to quantization, while some models such as LLaMA3 are more sensitive to quantization compared to Qwen.