Skip to content

Commit 9d6826a

Browse files
committed
[Bugfix] Fix HYV3 shared_mlp prefix for compressed-tensors ignore matching
HYV3MoEFused built its shared expert (shared_mlp) as an HYV3FeedForward with `prefix=f"{prefix}"`, omitting the `.shared_mlp` segment. The Linear layers inside it (gate_up_proj, down_proj) register their parameters under `...mlp.shared_mlp.<proj>.*` (the attribute name is appended by nn.Module regardless of the prefix argument), but CompressedTensorsConfig ignores Linears by matching the *prefix* against its `ignore` list. With the missing segment the ignore check saw `...mlp.down_proj` instead of `...mlp.shared_mlp.down_proj`, so a `re:.*shared_mlp.*` ignore entry failed to match. Consequence: the shared expert was built as a quantized Linear expecting weight_packed/scale/shape, while real W4A16 hy_v3 checkpoints (produced by llm-compressor) correctly keep shared_mlp in BF16 as `.weight`. This raised `KeyError: '...mlp.shared_mlp.down_proj.weight'` at load time. Fix: pass `prefix=f"{prefix}.shared_mlp"` so the ignore check matches the real parameter registration path and the shared expert is built unquantized (UnquantizedLinearMethod), aligning with the BF16 checkpoint. Parameter registration names are unchanged. Co-authored-by: xu_xiaoyi@hotmail.com Signed-off-by: Xiaoyi Xu <xu_xiaoyi@hotmail.com>
1 parent 31be872 commit 9d6826a

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

vllm/model_executor/models/hy_v3.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ def __init__(
171171
intermediate_size=config.expert_hidden_dim * config.num_shared_experts,
172172
hidden_act=config.hidden_act,
173173
quant_config=quant_config,
174-
prefix=f"{prefix}",
174+
prefix=f"{prefix}.shared_mlp",
175175
reduce_results=False,
176176
)
177177
else:

0 commit comments

Comments
 (0)