ignore= is honored at calibration but NOT at save_pretrained_wrapper — modules excluded from quantization still get RTN-packed at save

[pasta-paul]

Summary

compressed_tensors (or llmcompressor's save_pretrained_wrapper that wraps it) treats the recipe's ignore= list inconsistently between calibration phase and save phase:

• At calibration time: ignore= correctly excludes matched modules from the GPTQModifier / QuantizationMixin pipeline. The matched modules have no Hessian built, no quantize_weight call, no observer attached. This is verified — under our recipe, the MTP block's compress_module_list subgraph was empty (0 modules processed for MTP) during calibration.

• At save time: the saved artifact has matched modules in W4A16 packed form anyway. They were re-quantized at save via what looks like an RTN-style pass that consults targets= but NOT ignore=. The Compressing model: 0/N progress bar in save_pretrained_wrapper (llmcompressor/transformers/compression/compressed_tensors_utils.py:79) compresses any module matched by targets=, regardless of whether it's also in ignore=.

Reproduction

Recipe used in our DSv4-Flash W4A16+BF16-MTP run:

GPTQModifier(
    config_groups={
        "experts": QuantizationScheme(
            targets=[r"re:.*mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"],
            **W4A16,
        ),
        "attention": QuantizationScheme(
            targets=[r"re:.*self_attn\.(q_a_proj|q_b_proj|kv_proj|o_a_proj|o_b_proj)$"],
            **FP8_BLOCK,
        ),
    },
    ignore=[
        "lm_head",
        r"re:.*mtp\..*",   # ← INTENDED: exclude MTP draft head from quantization
    ],
)

targets= matches both model.layers.X.mlp.experts.* (intended) and model.mtp.0.mlp.experts.* (unintended — the .* prefix doesn't anchor). ignore=r"re:.*mtp\..*" is supposed to exclude all MTP paths.

Result at calibration: subgraph for MTP block had 0 modules — ignore worked.

Result at save:

import json
with open("model.safetensors.index.json") as f:
    wm = json.load(f)["weight_map"]
print([k for k in wm if "mtp" in k and "weight_packed" in k][:3])
# → ['model.mtp.0.mlp.experts.0.down_proj.weight_packed',
#    'model.mtp.0.mlp.experts.0.gate_proj.weight_packed',
#    'model.mtp.0.mlp.experts.0.up_proj.weight_packed']

768 MTP expert linears in weight_packed (int4) format. The MTP block was not supposed to be quantized.

Workaround

Anchor targets= patterns more specifically so MTP paths don't match in the first place:

targets=[r"re:^model\.layers\.\d+\.mlp\.experts\.\d+\.(gate_proj|up_proj|down_proj)$"]

With anchoring, save-time RTN doesn't match MTP paths → MTP modules ship in their loaded BF16 form. We confirmed this fixes the artifact on a subsequent smoke run.

Proposed fix

Either:

1. Make save-time compression consult ignore= — the cleaner fix. save_pretrained_wrapper should filter the compression target set through the recipe's ignore= list the same way GPTQModifier does at calibration. This restores symmetric semantics between calibration and save.

2. Document the asymmetry loudly — if (1) is intentional, the docs should say so. The current behavior is surprising: users naturally assume ignore= means "ignore everywhere," not "ignore at GPTQ but quantize at save."

(1) is what most users would expect. Mixed-precision recipes (e.g., DSv4 with BF16 MTP draft head + W4A16 main experts — see our model card draft for the design rationale) only work cleanly if ignore= is symmetric.

Why this matters

Speculative-decoding architectures (DSv4 MTP, Eagle, Medusa, etc.) need the draft head at higher precision than the main model — quantization noise in the draft directly kills acceptance rate. The natural way to express this in a recipe is:

targets=[main-model-MoE-pattern],
ignore=["draft-head-pattern"],

Without ignore= being honored at save, this pattern silently fails — the draft head gets quantized despite the user's intent, and the speedup the architecture is supposed to provide is silently degraded. The failure is invisible without inspecting weight_map of the saved artifact.

Related

• vllm-project/llm-compressor#2735 — DSv4 example drops MTP layer (the root issue this PR's fix enables)
• vllm-project/llm-compressor#2739 — extend ARCH_TO_2D_MAPPINGS for MTP paths (makes MTP visible to the pipeline so ignore= becomes meaningful)
• huggingface/transformers#46127 — DeepseekV4NextNPredictor class so MTP loads at all

[pasta-paul]

Production validation — workaround shipped

Filing back as cross-link: the targets= anchor workaround discussed in this thread has been validated end-to-end in a shipping artifact.

Setup

• Repo: https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp
• Artifact: https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (public)
• Recipe: scripts/quantize_v4_w4a16_mtp.py — anchors targets= at ^model.layers.\d+\. so MTP paths (mtp.0.*) cannot match
• ignore=[r"re:.*mtp\..*"] also present as a defense-in-depth gate

Evidence the workaround works at scale

• Phase 2 GPTQ calibration: 768 calibration samples, 8× H200, ~15h wall
• Output artifact: 0 mtp.0.* keys carry quantization suffixes (.weight_scale, .weight_packed, etc.). Verified post-save via scripts/verify_option_y.py + direct safetensors inspection.
• Serve verify with --speculative-config method=mtp num_speculative_tokens=1 passes the MTP draft head as BF16 → 69.94% acceptance over 200 random prompts (raw data in benchmarks/phase2/acc_*.json).

So why this issue still matters

The workaround relies on a regex-anchored targets= to defeat the bug. That's fragile — users who don't know to anchor will silently end up with MTP weights quantized at save (with no error from the recipe). The real fix is to honor ignore= consistently at both calibration AND save stages of save_pretrained_wrapper. Until that lands, every downstream MTP-preserving quant has to discover the anchor workaround independently. Two have so far: our W4A16 (canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP) and the sibling NVFP4 (canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP). A third (the upstream NeMo/RedHat path for similar MoE+MTP architectures) is at risk.

Happy to provide a minimal repro from the working artifact if helpful. Thanks for keeping the issue open.

[ishrith-gowda]

Tried to reproduce this on current main and it no longer occurs. save_pretrained_wrapper now compresses via ModelCompressor.from_pretrained_model + compress_model, both keyed off the per-module quantization_scheme (is_module_quantized) rather than re-matching targets=. Since apply_quantization_config already excludes ignore= when attaching schemes, an ignored module has no scheme at save and is never packed. The targets-re-matching pass at the old compressed_tensors_utils.py:79 is gone; that line is now the #2798 weight retie.

Repro (Qwen2.5-0.5B, GPTQ W4A16, ignore=["lm_head", "re:.*layers\.0\.mlp\.down_proj$"]): the ignored Linear ships dense and lands in the config ignore list, while the control layer packs.

Your case used unanchored targets= regexes that also matched MTP paths, so anchoring targets= is still the robust fix, but the save-time asymmetry itself looks resolved. Could you confirm on your DSv4 recipe on current main? If it's clean, this can probably be closed.

[ishrith-gowda]

Good numbers, glad the workaround holds in production. To be clear on scope: I agree the anchored targets= shouldn't be required. My point is that the save path your report cited was rewritten since May (save_pretrained_wrapper now compresses via ModelCompressor.from_pretrained_model + compress_model, keyed off attached schemes rather than re-matching targets=), and on current main my repro ships an ignored module dense even when targets= matches it. A minimal repro in your recipe shape (config_groups with regex targets plus regex ignore) would settle it: if it reproduces on main I'll dig in, if not this can close as fixed, with the anchor workaround documented for older releases.