[Bug]: Evaluate AWQ and GPTQ for Gemma

[Jeevi10]

⚙️ Your current environment

Environment Information

Operating System: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
Python Version: 3.12.11 | packaged by Anaconda, Inc. | (main, Jun 5 2025, 13:09:17) [GCC 11.2.0]
llm-compressor Version: 0.10.0.1
compressed-tensors Version: 0.14.0.1
transformers Version: 4.57.6
torch Version: 2.10.0
CUDA Devices: ['NVIDIA A100-SXM4-80GB', 'NVIDIA A100-SXM4-80GB']
AMD Devices: None
NPU Devices: None

🐛 Describe the bug

AWQ W4A16 quantization via llm-compressor produces catastrophically degraded output quality on Gemma 3-based models (tested on google/medgemma-27b-it). Accuracy drops from 85.6% (FP16 baseline) to 69.2% on the CareQA medical benchmark — a ~16 point degradation. In contrast, GPTQ W4A16 on the same model retains 83.6%, demonstrating that the issue is specific to AWQ's scaling mechanism, not a general quantization sensitivity.

#2500 (comment)

🛠️ Steps to reproduce

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier, AWQMapping
from llmcompressor.modifiers.quantization import QuantizationModifier

# Select calibration dataset
DATASET_ID = "FreedomIntelligence/medical-o1-reasoning-SFT"
DATASET_SPLIT = "train"
language = "en"



# Select model and load it.
MODEL_ID = "google/medgemma-27b-text-it"

model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

# Increasing the number of calib samples to 256 or higher can improve accuracy.
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, language, split=f"{DATASET_SPLIT}[:{NUM_CALIBRATION_SAMPLES}]")
ds = ds.shuffle(seed=42)
print(f"Dataset size: {len(ds)} samples")


# Preprocess: apply chat template and tokenize properly
def preprocess(example):
    text = tokenizer.apply_chat_template(
        [{"role": "user", "content": example["Question"]}],
        tokenize=False,
        #add_generation_prompt=True,
    )
    return tokenizer(
        text,
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )


ds = ds.map(preprocess, remove_columns=ds.column_names)

# Gemma 3 specific mappings (GQA-aware: skip v_proj -> o_proj due to dimension mismatch)
gemma3_mappings = [
    # 1. Smooth Input Norm against Q, K, V
    AWQMapping(
        smooth_layer="re:.*input_layernorm$",
        balance_layers=["re:.*q_proj$", "re:.*k_proj$", "re:.*v_proj$"]
    ),


    AWQMapping("re:.*v_proj$", ["re:.*o_proj$"]),
    # 2. Smooth the MLP using the pre-feedforward norm
    AWQMapping(
        smooth_layer="re:.*pre_feedforward_layernorm$",
        balance_layers=["re:.*gate_proj$", "re:.*up_proj$"]
    ),
    # 3. Smooth Up-proj against Down-proj
    AWQMapping(
        smooth_layer="re:.*up_proj$",
        balance_layers=["re:.*down_proj$"]
    ),
]

# Initialize the modifier with these mappings
recipe = [
    AWQModifier(
        targets=["Linear"],
        scheme="W4A16_ASYM",
        ignore=["lm_head"],
        #mappings=gemma3_mappings,
        duo_scaling="both"
    )
]

"""
 recipe = QuantizationModifier(
     targets="Linear",
     scheme="FP8_DYNAMIC",
     ignore=["lm_head"],
     duo_scaling="both"
 )
"""


#Run quantization algorithm
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

[brian-dellabetta]

Hi @Jeevi10 , are you trying on latest main? The fix you linked in #2500 is on main but not on the llm-compressor version (0.10.0.1) you listed you are using

[Jeevi10]

@brian-dellabetta Thanks for the swift reply, let me check. I might have missed something let me test it.

[Jeevi10]

@brian-dellabetta I installed from the source. is this correct?

Environment Information

Operating System: Linux-6.8.0-100-generic-x86_64-with-glibc2.39
Python Version: 3.12.11 | packaged by Anaconda, Inc. | (main, Jun 5 2025, 13:09:17) [GCC 11.2.0]
llm-compressor Version: 0.10.1.dev34+gcf3bd646
compressed-tensors Version: 0.14.1a20260325
transformers Version: 4.57.6
torch Version: 2.10.0
CUDA Devices: ['NVIDIA A100-SXM4-80GB', 'NVIDIA A100-SXM4-80GB']
AMD Devices: None
NPU Devices: None

[brian-dellabetta]

Hard to confirm just from the version number but that looks better. does it work for you now? You should see a line like this in the logs

2026-03-25T21:24:11.1687 | norm_calibration_context | INFO - Found 101 offset-norm modules to convert

[Jeevi10]

2026-03-26T15:38:04.139058+0000 | norm_calibration_context | INFO - Found 373 offset-norm modules to convert
is this correct? I am running it on 27b model btw. I am running the script again.

[Jeevi10]

This is the results, still the accuracy drop is significant compare to GPTQ or
FP8

[brian-dellabetta]

I don't see any issue with the mappings. You'll have to compare against the same scheme, W4A16 can perform very differently than FP8.
Your results are the following?

• Baseline fp16 -> 85.6%
• GPTQ W4A16 -> 83.6%
• AWQ W4A16 -> 71.2%
• RTN (using QuantizationModifier) W4A16 -> (?)

If RTN performs better than AWQ, there may be an issue. If AWQ performs better than RTN, but worse than GPTQ, that can happen. You may also want to experiment with W4A16_ASYM, which can improve recovery without much additional overhead

[dsikka]

Tracked in JIRA: https://issues.redhat.com/browse/INFERENG-5600

JIRA Details:

• Issue Type: Bug
• Priority: Critical
• Component: LLM Compressor
• Team: INFERENG Model Optimizations
• Status: Backlog

[Jeevi10]

@dsikka Thanks for sharing this, I was not able to check the issues in jira due to access restriction. is there anything i could do at this moment ?
@brian-dellabetta Thanks for sharing this insight, I will do the testing with RTN (using QuantizationModifier) W4A16 and post the results.

[Jeevi10]

Hi @brian-dellabetta,

Sharing the W4A16 quantization results (rounded to nearest), which point to an issue with AWQ:

• Baseline (FP16): 85.6%
• RTN (Quantization Modifier)-W4A16: 84.0%
• GPTQ-W4A16: 83.6%
• AWQ-W4A16: 71.2%

AWQ shows a significant accuracy drop (~14.4 pp) compared to baseline, while both RTN and GPTQ remain within ~2 pp. I also tried W4A16_ASYM with AWQ, but it didn't yield any meaningful improvement over the symmetric results.

This confirms there's likely a problem on the AWQ side. Happy to share further details or run additional experiments if needed.

[brian-dellabetta]

Hi @Jeevi10 , thanks for sharing the results. it's weird that RTN outperforms GPTQ as well, but yeah the AWQ regression is very strange. Can you try autoround and see if that performs any better? If none of the calibrated flows outperforms RTN, the calibrated dataset is not helpful in steering the quantized version to improve on the eval dataset.

You can ignore the JIRA ticket&link. That is something new we've added for internal processes.

[Yatimai]

Hi @brian-dellabetta,

I looked into this as a contributor to #2500. I believe the root cause is a missing module2inspect equivalent in llm-compressor's AWQ implementation.

In the reference MIT implementation (llm-awq/awq/quantize/auto_scale.py), each mapping can specify a module2inspect, the module on which the grid search loss is computed. For the post_attention_layernorm → gate,up mapping, MIT sets module2inspect=module (the full DecoderLayer), not just mlp. This ensures the loss sees everything downstream of the balance layers, including any post-norms.

llm-compressor computes the loss at the LCA of the balance layers (self_attn or mlp), with no way to override it. On Gemma3, post_attention_layernorm and post_feedforward_layernorm sit outside that scope, so the grid search optimizes scales without seeing how the post-norms transform the quantization error.